AI Governance, Risk, and Compliance
Every earlier chapter in this module assumed the AI system in question was known, approved, and intentionally deployed. That assumption breaks down constantly in real organizations. This chapter covers the governance layer that sits above the technical controls: why ungoverned AI adoption is itself a security problem, the two frameworks security teams are most likely to hear referenced in this space, the practice of red teaming AI systems before and during deployment, and how to fold all of it into a working program rather than a one-time compliance exercise.
Why AI governance is now a security concern, not just a legal one
Governance used to be something security teams handed off to legal or compliance and checked in on once a quarter. AI has collapsed that separation: organizations adopt AI tools faster than formal review can keep pace, and every review gap becomes a security gap almost immediately, unknown data handling, an unreviewed supply-chain dependency, an incident with no clear owner.
Shadow AI is the clearest entry point into this problem, borrowed directly from an older one. Shadow IT was employees standing up their own SaaS tools or devices outside IT's visibility because the sanctioned process was too slow; shadow AI is the same behavior, applied to AI.
- An employee pastes a chunk of a customer contract into a public chatbot to get a faster summary.
- A team installs a browser extension that quietly routes page content through a third-party model.
- A business unit turns on an AI feature bundled into a SaaS product without anyone from security ever reviewing what data it sends where.
None of this is malicious. All of it is invisible to the organization until something goes wrong.
- Sensitive data (source code, customer records, unreleased financials, credentials) can leave the organization's boundary with no logging, no DLP visibility, and no way to claw it back.
- The organization inherits the data retention, training-use, and security posture of whatever vendor is on the other end, without ever having evaluated it.
- There is no inventory, so there is no way to know the actual blast radius when a vendor discloses a breach or a model provider changes its terms of service.
- Accountability is undefined. If the AI tool contributes to a bad outcome, no one signed off on it being in use, so no one is positioned to answer for it.
Security has the visibility tooling (network egress monitoring, SaaS discovery, endpoint telemetry) to actually find shadow AI. Governance has the framework for deciding what to do once it's found. Neither function alone closes the gap.
The NIST AI Risk Management Framework (AI RMF)
In January 2023, NIST published the AI Risk Management Framework (AI RMF) 1.0, a voluntary framework for identifying, assessing, and managing AI risk across a system's lifecycle, from design through deployment and retirement. Voluntary matters: nothing in it compels an organization to do anything. It's guidance, not law (the next section covers a framework where that's reversed).
The AI RMF organizes its guidance around four core functions. They aren't a sequence completed once, they operate together, continuously, feeding back into each other as an AI system and its risk profile evolve.
Govern sits first but doesn't finish before Map begins, and Manage doesn't close the loop, it feeds back into ongoing monitoring that reshapes what gets mapped and measured next.
The EU AI Act
Where the AI RMF is voluntary guidance, the EU AI Act is binding law. Formally Regulation (EU) 2024/1689, it entered into force on 1 August 2024 and applies to providers and users of AI systems operating in or serving the EU market, regardless of where the provider is headquartered. It does not take effect all at once. It rolls out on a phased, risk-tiered timeline, with different obligations activating on different dates over several years.
| Date | What takes effect |
|---|---|
| 2 February 2025 | Prohibited AI practices (such as social scoring) become unlawful; AI literacy obligations begin applying. |
| 2 August 2025 | Rules for general-purpose AI (GPAI) models, governance structures, and penalty provisions start applying. |
| 2 August 2026 | General application of the Act, plus Article 50 transparency duties (chatbot disclosure, synthetic content labeling). |
| 2 December 2027 | High-risk systems under Annex III must comply. Deferred from the original August 2026 target. |
| 2 August 2028 | High-risk, product-embedded systems under Annex I must comply. Deferred from the original August 2027 target. |
Underneath that timeline sits the Act's core structure: AI systems are categorized by risk level, and obligations scale with it. Systems that pose unacceptable risk are prohibited outright. High-risk systems face the heaviest compliance burden: documentation, human oversight, testing, and conformity requirements. Limited-risk systems carry transparency obligations, such as disclosing that a user is interacting with an AI system. Minimal-risk systems face essentially no obligations under the Act. Penalties scale with severity too; the maximum tier, for violations involving prohibited practices, is up to €35 million or 7% of global annual turnover, whichever is higher.
AI red teaming
AI red teaming is adversarial testing of an AI system, done before it ships and repeated after, to find where it breaks. It's penetration testing extended into a domain with its own failure modes, the exact ones covered in chapters 2, 3, and 6:
- Prompt injection payloads that steer the model off its instructions
- Jailbreaks that bypass safety guardrails
- Leakage of training data or context that shouldn't be disclosed
- Biased or harmful outputs under adversarial or ordinary prompting
Governance frameworks increasingly point to red teaming as evidence of due diligence, not an optional extra, under the NIST AI RMF it's a natural way to satisfy Map and Measure. The mistake to avoid: treating it as a pre-launch gate that never gets revisited. Models get fine-tuned, guardrails get updated, new jailbreak techniques get published constantly, a system that passed six months ago hasn't necessarily passed now. Ongoing testing, not a one-time exercise, is what the practice requires to mean anything.
Bringing governance and security together in practice
None of the frameworks above do anything on their own. What actually closes the gap between "we have a policy document" and "we know what AI is running in our environment" is a small set of practical habits, most of which a security team is already positioned to own.
| Habit | Why it matters |
|---|---|
| Maintain an inventory | The direct countermeasure to shadow AI: a maintained list of AI systems in active use, including ones embedded inside other SaaS products, not just standalone chatbots. You can't govern, measure, or defend what you don't know exists. |
| Risk-tier new use cases | A marketing copywriting assistant shouldn't get the same review burden as a system with access to customer financial data. Scaling effort to actual risk keeps the process from collapsing under its own weight or being bypassed as too slow. |
| Security checkpoint before production | Any system touching sensitive data or taking a consequential action (approving a transaction, modifying access, generating externally-shipped content) gets reviewed before it reaches production, not after an incident surfaces it. |
| Treat it as a lifecycle, not a checklist | Maps directly onto the AI RMF: Govern the policy and culture, Map each system's specific risk, Measure it against real evidence, Manage it with monitoring that continues for as long as it's in use. A program that stops at initial approval only checks things once. |
Chapter 8 covers what it takes to build all of this into a functioning, ongoing program rather than a set of good intentions.
Key Takeaways
- Shadow AI, unsanctioned AI tool use outside any governed process, turns governance gaps into security gaps: unknown data handling, unreviewed vendor dependencies, and undefined accountability.
- The NIST AI RMF (published January 2023) is voluntary guidance built around four functions working together: Govern, Map, Measure, Manage.
- The EU AI Act, Regulation (EU) 2024/1689, is binding law with a phased timeline: prohibited practices and AI literacy (Feb 2025), GPAI/governance/penalties (Aug 2025), general application and Article 50 transparency (Aug 2026), high-risk Annex III (Dec 2027), high-risk Annex I product-embedded (Aug 2028).
- AI red teaming adversarially tests for prompt injection, jailbreaks, data leakage, and biased or harmful output, and needs to be ongoing, not a one-time pre-launch gate.
- An AI inventory, risk-tiered review, a security checkpoint before production, and treating governance as a continuing lifecycle are what actually make these frameworks operational.
Knowledge Check
Click an answer to reveal the explanation.
What does "shadow AI" refer to?
Under the NIST AI RMF, which function is primarily responsible for identifying the context and risks specific to a given AI system's use case?
Under the EU AI Act's phased timeline, which milestone corresponds to 2 February 2025?