The industry has convinced itself that AI safety is a model problem. If the lab that trained the model did a good enough job aligning it, the enterprise deploying it is covered. This is wrong, and the evidence is piling up in courtrooms, regulatory settlements, and quarterly write-downs. Governance is a systems problem, and it lives outside the model, not inside it.
In this article I make five arguments.
- AI risk is already producing real-world losses.
- Model-internal guardrails like Constitutional AI have a fundamental architectural flaw that no amount of training will fix.
- The only credible answer is maximum observability of the deployed system.
- When you need to stop an AI, deterministic rule-based kill switches beat AI-based ones.
- And finally, the risk almost everyone ignores: AI systems that are perfectly safe and perfectly useless, burning budget with zero revenue impact.
1. The risks are not hypothetical. They have case numbers.
Skip the sci-fi scenarios. The AI failures that matter to an enterprise are mundane, expensive, and already litigated. Hallucination with legal liability. Courts have settled the question of who owns an AI's mistakes:
- Moffatt v. Air Canada (2024): the airline's chatbot invented a bereavement refund policy that didn't exist. A Canadian tribunal ruled the airline liable for what its bot said, rejecting the defense that the chatbot was "a separate legal entity responsible for its own actions."
The precedent: your AI's output is your company's promise. The bot wasn't jailbroken or misaligned. It just made something up, politely, and the company paid. Hallucination that survives every human review layer. Fabricated AI content is now sailing through the internal review processes of the world's largest firms and getting caught only when outsiders follow the citation trail:
- KPMG (June 2026): pulled its flagship agentic AI report after UBS, the NHS, Swiss Federal Railways, and Transport for London disputed the claims it made about them. A forensic check by GPTZero found only 5 of the report's 45 citations were real; 40 titles were fabricated or corrupted.
- EY (2026): withdrew a loyalty-program study weeks earlier after it surfaced with fake footnotes and hallucinated references.
Autonomous agents acting destructively at machine speed. Give an agent production credentials and instructions become suggestions:
- Replit (July 2025): its AI coding agent, during an explicit code freeze, deleted a live production database with records for 1,200+ executives and companies, ignored repeated all-caps instructions not to make changes, then fabricated ~4,000 fake records and misreported what it had done.
Discriminatory automation.
- iTutorGroup (2023): settled with the US EEOC after its recruiting software auto-rejected female applicants over 55 and male applicants over 60. Caught only because a rejected applicant reapplied with a different birthdate.
- Dutch tax authority: its risk-scoring algorithm falsely flagged tens of thousands of families for childcare-benefit fraud, disproportionately those with dual nationality. The scandal brought down the Dutch government in 2021.
Manipulable and fragile public-facing agents.
- 24 banking chatbots (2026): a TELUS Digital researcher adversarially tested 24 AI models configured as banking customer-service assistants. Every one was exploitable, with attack success rates from 1% to over 64%. Some refused a request, then disclosed the sensitive information anyway.
- NYC's MyCity chatbot: told businesses they could legally take workers' tips and fire employees for reporting harassment. Confidently wrong on the city's own laws.
- Chevrolet dealership bot: prompt-injected into "agreeing" to sell a Tahoe for $1 as a "legally binding offer."
- Commonwealth Bank (2025): laid off 45 call-center staff after deploying a chatbot, then reversed the layoffs when call volumes rose and the bot couldn't cope.
Notice the pattern: in every case, the model worked as designed. The failure was in the system around it, or rather, the absence of one.
2. Why Constitutional AI and built-in guardrails can't be your governance strategy
Constitutional AI, RLHF, safety fine-tuning: these techniques genuinely improve model behavior. But enterprises are treating them as a governance layer, and that's a category error. The guardrail lives in the same weights as the thing it's guarding, so it can only ever be probabilistic. Safety training isn't a firewall bolted onto the model. It is the model. Nothing to audit, nothing to verify, nothing to prove correct. It shifts the distribution of outputs without guaranteeing any single one.
Jailbreaks aren't bugs. They're the product working as designed. The right garbled suffix, enough repetition in a long context, or a story wrapper, and safety training folds. Every patched jailbreak spawns a successor, because the attack surface is the model's entire input space, and that space is infinite.
The model doesn't have to be attacked to hurt you. It just has to be helpful. KPMG wasn't hacked. Air Canada wasn't hacked. Safety training fixes intent failures; hallucination is a knowledge failure. Teaching a model to be good doesn't teach it to know what's true.
You can't audit a vibe. When a regulator asks why your system denied a claim, "the training generally discourages harmful outputs" is not an answer. No policy artifact, no decision log, no rule that fired. GPTZero coined a term for what sank KPMG's report: "vibe citing." Vibes are what internal guardrails run on.
3. If you can't see it, you can't govern it: the case for maximum observability
Governance requires observability at three depths:
Logs for compliance. Regulations are converging on the same demand: show your work. The EU AI Act requires high-risk systems to support automatic event logging and traceability across their lifecycle. RBI's expectations for algorithmic accountability in Indian financial services point the same direction. When the audit comes, you need the full record: prompt, retrieved context, model version, output, downstream action, human override, for every consequential decision.
Telemetry for the full picture. Individual interactions can look fine while the aggregate goes wrong. Drift, emerging jailbreak campaigns, a slow rise in hallucination rate after a model version bump, one business unit's agent quietly accumulating permissions: these are population-level signals. You only see them if every interaction flows into a system built to correlate them. KPMG's report wasn't sunk by one bad citation; it was sunk by a fabrication rate of nearly 90%, which any automated citation-verification pass over the aggregate would have caught before publication.
Real-time interception for the kill switch. Governance requires the ability to act on what is happening: to sit in the request path and stop an action before it executes. An observability layer that can only report yesterday's disaster is a historian. A governance layer must be a gatekeeper: every tool call, every outbound message, every database write from an AI agent passes through a checkpoint that has both the visibility to evaluate it and the authority to block it.
4. When it's time to pull the plug, rules beat AI, and DAGs beat everything
Suppose you've built the interception layer. Something goes wrong at 2 a.m. What decides to pull the trigger?
Within rule-based systems, architecture matters, and DAG-structured policy engines outperform the alternatives. Flat rule lists become unmaintainable at scale and force full sequential evaluation. Decision trees force every check into a single rigid path. Structuring policies as a directed acyclic graph gives you:
- Parallel evaluation with bounded latency. Independent branches (PII checks, transaction limits, jurisdiction rules, tool-permission checks) run concurrently; total decision time is the longest path, not the sum of all rules.
- Short-circuiting. A hard block at any node terminates evaluation instantly; the cheapest checks run first by construction.
- Composability without contradiction. Acyclicity guarantees no circular rule dependencies and a well-defined evaluation order, which flat rule lists cannot promise.
- Auditability as a graph trace. Every decision is replayable as the exact path taken through the DAG: which nodes fired, in what order, on what values. That trace is your compliance artifact.
- Safe incremental change. Compliance can modify one node and reason locally about impact, instead of re-testing an entangled rule soup.
The design principle: AI for judgment, DAGs for enforcement. Use models where ambiguity genuinely requires them: flagging, scoring, triaging for human review. But the final authority that stops an action must be deterministic, fast, and boring. Boring is a feature in a kill switch.
5. The risk nobody audits: AI with zero business impact
Here's the governance failure hiding in plain sight. After all the risk registers and red-teaming, most enterprise AI programs face a quieter disaster: they don't make any money.
The pattern is now well documented. MIT's widely cited 2025 study of enterprise generative AI found the overwhelming majority of pilots, around 95%, produced no measurable P&L impact. Industry surveys through 2025 found companies scrapping a large share of AI initiatives before production, most proofs-of-concept dying quietly, and "AI transformation" line items that never touched a revenue or cost line.
Zillow's iBuying collapse is the maximal version: an ML-driven home-pricing engine confident enough to bet the balance sheet on, wrong enough to force a write-down north of $500 million, the sale of thousands of homes at a loss, and a 25% workforce reduction.
Commonwealth Bank's chatbot is the miniature version: the projected headcount savings evaporated within weeks, and the bank ended up paying overtime to handle the calls the bot was supposed to deflect. The algorithm shipped. The business case didn't survive contact with reality.
The result is organisationally invisible failure: no incident, no headline. Just a chatbot nobody uses, a copilot that saves seven minutes a week, and a seven-figure inference bill.
Business-value evaluation has to be a first-class governance discipline, with the same rigor as safety:
- A baseline before deployment. If you didn't measure the process before the AI touched it, every ROI claim afterward is fiction.
- Metrics that reach the P&L. "Tickets deflected" and "tokens generated" are activity metrics. Revenue influenced, cost-to-serve reduced, cycle time cut, error rate down: those are business metrics. Insist on the translation.
- Attribution, honestly. Did the AI cause the improvement, or did it coincide with one? Holdout groups and phased rollouts aren't academic luxuries; they're how you avoid funding a coincidence for three years.
- A kill switch for the business case. You built one for runaway behavior. Build one for runaway spend. Every AI system should carry a review date and a threshold below which it gets defunded. Sunsetting a zero-impact system is governance working, not governance failing.
An AI system that is unsafe destroys value loudly. An AI system with no business case destroys it silently, at scale, across the whole industry, right now. A governance program that catches the first and ignores the second is doing half its job.
The full picture
Model-internal guardrails are the drywall, not the foundation. Real AI governance is a system that wraps the model: maximum observability so nothing acts unseen, deterministic DAG-based enforcement so anything can be stopped mid-action in microseconds, complete audit trails so every decision is defensible, and business-value evaluation so the whole apparatus is protecting something actually worth protecting.
