Two of the most serious AI governance gaps in modern enterprises are not about the AI systems the organisation built. They are about AI the organisation is using without having built it, and often without knowing it is using it.
The first is third-party AI: foundation models, API-based services, embedded AI features in enterprise software, and AI capabilities in the supply chain. Most enterprise AI is not built by the enterprise. It is bought, licensed, or accessed through APIs. The governance challenge is that responsibility does not travel with the transaction. A foundation model provider’s terms of service do not make them accountable for the outputs a customer produces. A vendor’s SOC 2 report does not tell you whether their new AI feature has been validated for your use case.
The second is shadow AI: the AI systems employees are using without organisational authorisation, oversight, or governance. Every enterprise I know has this. Employees use consumer AI tools with organisational data. Business units procure AI services outside the technology approval process. Individual contributors adopt AI-assisted development tools without security review. The scale is much larger than most CIOs and CROs realise.
APRA’s 30 April 2026 letter identified third-party AI as one of its four areas of concern, with specific reference to concentration risk and gaps in contractual arrangements. The MAS Guidelines reference third-party AI risk management as a specific expectation. The Australian Voluntary AI Safety Standard, the NIST AI RMF, and ISO/IEC 42001 all include third-party governance requirements.
Shadow AI is less directly addressed in regulation, but it appears in APRA’s discussion of AI literacy and vendor oversight, and it is now widely acknowledged by regulators as a material risk. Once a regulator asks whether you have an inventory of AI systems and your answer excludes the systems your employees are actually using, the inventory does not do what it needs to do.
This article covers both.
Third-party AI: the specific problem
Third-party AI risk is a specific variant of vendor risk that traditional vendor management was not designed for.
Traditional vendor risk management focused on operational risk (will the vendor be reliable), security risk (will the vendor protect our data), and financial risk (will the vendor stay in business). All three of these still apply to AI vendors. But AI vendors introduce three additional risk categories.
Model behaviour risk. The AI vendor’s model produces outputs. Those outputs may be biased, may hallucinate, may be manipulated by adversaries, and may change over time as the vendor updates the model. The deployer’s use case depends on model behaviour that the deployer cannot fully verify and does not fully control.
Concentration risk. Foundation models are provided by a very small number of firms. If most of your AI depends on one provider, that provider’s decisions (pricing, deprecation, terms of service, availability) become concentration risk. This is what APRA specifically called out. It applies to smaller providers too: any dependency without a credible substitute is concentration risk.
Value chain risk. The vendor’s AI depends on their vendors: cloud infrastructure, training data providers, tooling suppliers. The value chain has multiple layers, and disruption anywhere can affect the deployer. The EU AI Act calls this out with specific provisions on value chain participants (Article 25 for providers, and various obligations flowing through the chain).
Traditional vendor risk instruments (SOC 2 reports, penetration tests, financial reviews) do not fully address these AI-specific risks. New instruments are emerging (ISO/IEC 42001 certification, model cards, AI-specific attestations) but they are not yet universal, and their use in vendor selection is still developing.
What a defensible third-party AI process looks like
Three phases: assessment, contracting, and ongoing management.
Assessment before selection. Before committing to a vendor, you assess their AI-specific risk. This includes: their governance maturity (is there an AI management system, ISO 42001 certification, published responsible AI framework), their technical posture (model documentation, evaluation practices, safety measures, incident history), their commercial posture (financial stability, market position, roadmap credibility), and their fit for your specific use case (does their model do what you need it to do at the quality level you need).
The assessment should be proportionate. A foundation model provider being used for a low-risk internal productivity tool warrants less scrutiny than the same provider being used for a high-risk customer-facing decision system. The tiering from article 2 applies.
Contracting. For material AI vendors (as defined by your policy, typically for high-tier use cases), contracts should address AI-specific concerns. Notification of material model updates. Model behaviour warranties where meaningful ones can be obtained. Data handling and confidentiality of prompts and outputs. Audit rights (for regulated entities in particular). Incident notification with specified timeframes. Data residency and processing location. Termination and exit provisions with defined timelines. Allocation of responsibility for compliance with legal and regulatory obligations that continue to apply to the deployer.
For foundation model providers, you often cannot negotiate individually. You are accepting standard terms. Read them. Understand what representations the provider is making, what they are excluding, and what your fallback plan is if the standard terms change. Some providers offer enterprise agreements with negotiable terms above certain commit levels; if your use case warrants, engage on this.
For Australian regulated entities, CPS 230 requires contract updates for material service providers by the earlier of renewal or 1 July 2026. AI vendors are typically material service providers.
Ongoing management. The vendor relationship is not managed at contracting time; it is managed continuously. Regular reviews of vendor governance changes, model updates, incident history, and roadmap alignment. Monitoring of vendor stability and market position. Testing of exit and substitution plans. Escalation processes for material issues.
Ongoing management is where most vendor risk programs are weakest. The initial assessment gets attention. The ongoing management gets forgotten. Then something changes at the vendor, and the deployer is caught unaware.
The concentration risk question
APRA’s specific call-out on concentration risk deserves its own attention.
Foundation model concentration is real. A small number of providers dominate the market. Enterprises building on foundation models often build on one, or two at most, with limited practical ability to substitute. The technical work to migrate from one foundation model to another is non-trivial: prompts perform differently, outputs vary, behavioural characteristics diverge, and cost structures change.
The governance response to concentration risk has several elements.
Portfolio approach where feasible. For enterprises with many AI use cases, spreading foundation model dependence across providers reduces concentration. This has costs (more integrations, more relationships to manage) but reduces vendor-specific exposure.
Substitutability analysis. For each material use case, assess how substitutable the current provider is. If a substitute exists in principle, what is the time and cost to actually migrate? What is the quality gap between primary and substitute? Regulators are increasingly asking for this analysis specifically.
Exit plans that are tested. A theoretical exit plan is not a real one. For material AI dependencies, exit plans should be documented in detail and tested at least periodically. APRA’s CPS 230 expectations on business continuity apply.
Contractual protections. Contracts should address the concentration risk explicitly where possible: minimum notice for deprecation or material changes, transition assistance obligations, defined pricing frameworks that limit unilateral change.
Diversification of the value chain. Even where the foundation model is concentrated, other layers of the AI stack (infrastructure, tooling, integration) may allow diversification.
The uncomfortable reality is that concentration risk cannot be fully mitigated in the current market structure. What can be done is documented, monitored, and prepared for.
Figure 1: The third-party AI risk stack
Figure 1 shows the third-party AI risk stack as it appears to a typical enterprise. At the base are foundation model providers, whose models underpin most enterprise generative AI. Above them are AI platform providers (orchestration, tooling, evaluation) who wrap and extend foundation models. Above them are AI feature providers (SaaS applications with embedded AI capabilities) who deliver specific AI-powered functionality. At the top are the deployer’s own systems and processes that consume AI capabilities. Each layer introduces its own governance considerations, and each is opaque to the layers above it. Effective third-party governance addresses all four layers, not just the visible one.
Shadow AI: what is actually happening
Shadow AI is the systematic use of AI in an organisation outside the sanctioned governance perimeter.
The scale of shadow AI in most enterprises is materially larger than what is officially tracked. Studies conducted through 2025 and 2026 have consistently found that individual employee use of AI tools substantially exceeds the AI use recognised in organisational inventories. The tools involved include public generative AI services accessed through personal accounts or free tiers, AI-enabled features embedded in productivity software the organisation may not have separately approved, AI coding assistants installed by developers without security review, and business unit procurement of AI services outside the technology approval process.
The consequences of shadow AI fall into three categories.
Data leakage. Employees paste organisational data into external AI services. Some of that data may be confidential, regulated, or subject to specific contractual restrictions. Once pasted, it is outside the organisation’s control, and depending on the service’s terms, may be used to train future models.
Compliance gaps. AI use that is not tracked cannot be assessed against regulatory obligations. The EU AI Act’s requirements, GDPR’s provisions on automated decision-making, sector-specific rules on model risk, and organisational policy on AI use all become unenforceable for shadow use.
Quality and reliability risk. Shadow AI use often does not go through the validation, monitoring, or documentation that sanctioned use does. The outputs feed into work products without the safeguards the governance program was designed to provide.
Why shadow AI persists
Shadow AI is not primarily a discipline problem. It is a governance design problem.
Employees adopt shadow AI because it is useful for their work and the sanctioned alternatives are inadequate. The tools available in the sanctioned catalogue may be less capable, may not be available for the specific use, may require lengthy approval processes, or may be perceived to be more constrained than the freely-available alternatives.
Treating shadow AI as a compliance violation to be enforced against does not work. It drives the use further underground and damages the trust relationship between employees and the governance program. The pattern I see working is different: reduce the reasons for shadow AI by making sanctioned use fit for purpose.
Provide credible sanctioned tools. If employees have access to enterprise-grade AI that meets their productivity needs, the pull toward shadow AI drops substantially. This means investing in the tools your employees actually want to use, in configurations that meet security and data protection standards.
Streamline approval processes. If the sanctioned path is fast, employees will use it. If it takes three weeks to get access to a tool, employees will find another way. Investment in the operational efficiency of AI approval processes has direct governance value.
Segment by risk. Not every use case needs the same level of governance. Low-risk uses can be self-service; high-risk uses require review. Applying the same friction to everything drives shadow AI for the low-risk cases.
Educate on real risks. Employees are generally reasonable when the risks are explained clearly. “Do not use consumer AI with customer data because it may be used for training” is a specific, credible, actionable message. “AI is dangerous” is not.
Monitor and act. Some shadow AI will always exist. Detection through network monitoring, DLP tools, and behavioural analytics identifies specific concerns. Response should be proportionate: education for lower-risk violations, formal action for higher-risk ones.
Building the AI inventory
The foundation for both third-party and shadow AI governance is a comprehensive AI inventory. This is what APRA identified as a specific gap in Australian financial services.
A comprehensive AI inventory covers:
Sanctioned systems. AI systems approved through governance processes, whether built internally or procured externally. This is the inventory most organisations have some version of.
Embedded AI. AI features embedded in enterprise applications, whether or not the AI feature was separately approved. If your CRM has an AI-powered content generator, that is an AI system in scope even if you never separately approved it.
Third-party AI dependencies. AI capabilities that your organisation depends on through vendors, whether the AI is prominent or incidental to the vendor’s offering.
Shadow AI, where identifiable. AI use through consumer services, unsanctioned enterprise services, or informal adoption. Complete visibility is unrealistic, but detection through network monitoring, expense analysis, and periodic surveys catches material patterns.
The inventory should be live, not point-in-time. Systems come and go, features change, vendors update their offerings. A quarterly review cycle is typical for large organisations; monthly for those with more dynamic environments.
Figure 2: The AI risk visibility landscape
Figure 2 shows the AI risk visibility landscape as it appears in most enterprises. In the visible zone are sanctioned AI systems that the governance program tracks and manages. In the partially visible zone are embedded AI features and third-party AI dependencies, where the governance program has partial visibility but often lacks direct control. In the shadow zone are unsanctioned AI uses, largely invisible to the governance program. The size of each zone varies by organisation, but almost every organisation has substantial partially visible and shadow zones. Governance maturity is measured largely by how much of the visibility landscape is brought into the visible zone over time.
What the regulators expect
The regulatory picture on third-party and shadow AI can be summarised quickly.
APRA expects mapping of the full AI supply chain, including material third and fourth-party dependencies, with contractual and governance arrangements that provide transparency, auditability, and assurance. Concentration risk must be assessed and managed.
MAS expects third-party AI risk management as a specific practice, with enhanced procurement, vendor assessment, and contracting.
The EU AI Act allocates specific obligations to providers, deployers, importers, and distributors of AI systems, with value chain provisions ensuring that responsibility flows through the chain rather than dissolving.
ISO/IEC 42001 control A.10 addresses third-party and customer relationships.
NIST AI RMF category GV-6 addresses third-party software and data.
None of these regulators names shadow AI specifically as an enforcement priority, but all of them expect AI inventories that reflect actual use. The inventory that excludes shadow use is not the inventory that will satisfy a regulator.
Where to start
If your organisation is building the third-party and shadow AI capability now, here is the sequence I would suggest.
Complete the inventory. Sanctioned systems first, then embedded AI, then third-party dependencies, then identifiable shadow AI. This work is unglamorous and takes months for large organisations. It is also the substrate on which everything else rests.
Assess the top-tier third-party dependencies. For the material AI vendors, run the full assessment: governance, technical, commercial, fit for purpose. Document concentration exposure. Update contracts where needed.
Address the visible shadow AI patterns. Where you have detected specific patterns of shadow AI, respond with a combination of education, provision of credible sanctioned alternatives, and process streamlining. Blanket enforcement rarely works.
Build monitoring capability. Network monitoring, expense analytics, and periodic surveys give you visibility into what is actually happening. This is a continuous discipline, not a one-time project.
Iterate. The third-party and shadow AI landscapes are changing quickly. What was true six months ago is not true now. The governance capability needs to keep pace.
The next and final article of this track pulls everything together: AI incident response, red-teaming, and audit readiness. It is the operational closer, and it includes the practical governance checklist that we have been building toward across the whole track.