The anatomy of an AI decision failure: where expensive mistakes begin
Expensive enterprise AI decision failures often originate not in the model’s “stupidity,” but in the design of the system surrounding it. The same model may work reliably in one structure and produce a serious error in another. This article is an attempt at diagnosis: it examines where a failure begins across five structural layers. The first condition for preventing an error is looking for it in the right place.
When enterprise AI failures are discussed, the model is usually blamed: “the model was not good enough,” “AI is not ready,” or “it hallucinated.” These explanations are comforting because they transfer responsibility to the technology. Yet the difference often lies not in the model, but in the layers around it. Global trust in fully autonomous AI fell from 43% to 27% in 2025, while fewer than 10% of organizations have a robust AI-governance framework. The central problem is not model intelligence, but the maturity of the surrounding system.
In brief
- Expensive AI errors are not one “bad model” event. They are a chain of five structural blind spots: wrong context, missing evaluation, scope drift, unowned output, and unsupervised action.
- The logic is the same as the “Swiss cheese model” in accident theory: an accident does not result from one hole, but from holes aligning across multiple defensive layers (James Reason).
- The cost of a governance gap is concrete: companies without AI governance pay an average of approximately $670,000 more per data breach (IBM, 2025).
- The good news is that none of these links is closed by changing the model; each is closed through system design.
Layer 1: Wrong or incomplete context
The first and most common source of error is context. A model produces an answer based on the context it receives. If that context is wrong, incomplete, or outdated, the model can produce a confidently incorrect answer. In practice, this appears in many forms: the model is given an old document and assumes it is current; relevant data does not fit inside the context window, so the model decides from an incomplete picture; or the wrong source is retrieved and the model treats it as authoritative. None of these is the model being “stupid.” Each is a weakness in the context layer.
The dangerous feature of a context error is that the output still reads fluently. Even with incomplete context, the model can produce a polished and confident answer. Fluency is not evidence of accuracy, but it can hide the missing context.
Layer 2: No evaluation set
The second blind spot is measurement. Many AI systems are put into production without ever being tested systematically against difficult examples. A demo goes well and is interpreted as proof that “it works.” Without an evaluation set, however, no one knows when the system is actually right and when it is wrong. A change may reduce one type of failure while increasing another, but the trade-off remains invisible. The system improves in one place and silently deteriorates in another until an expensive failure finally exposes the problem.
The absence of an evaluation set does not create the weakness, but it makes the weakness invisible. An invisible weakness cannot be corrected.
Layer 3: Scope drift
The third source of error is the gradual expansion of scope. The system is built for a narrow and clearly defined task and works well. Then someone says, “Since it works, let it do this as well.” The scope slowly expands until the system moves beyond the boundaries in which it was tested and known to be reliable.
This is already observable in agentic systems. The founder of security firm DryRun Security has reported that AI coding agents start strongly but deteriorate consistently as the codebase grows, drifting from established patterns, violating standards, and creating subtle but dangerous defects. A model that is dependable on a narrow task may be fragile on a broad and ambiguous one. The error emerges at the uncertain edge where scope has expanded beyond validation.
Layer 4: Unowned output
The fourth blind spot is ownership. AI produces an output, but responsibility for its quality is unclear. When an error occurs, who is expected to detect it, correct it, and learn from it? An AI system without an owner decays: output quality falls unnoticed, user feedback is not collected, and recurring errors are not corrected.
The cost of this gap has been measured. According to IBM’s 2025 Cost of a Data Breach Report, organizations without an AI-governance policy pay an average of $670,000 more per breach, and 63% of breached organizations had no AI-governance policy. An expensive error is often the accumulated deterioration of a system that no one has reviewed for months.
Layer 5: Unsupervised action
The fifth and most expensive source of error is allowing AI to take external action without human supervision. The system does not merely generate an answer; it sends an email, changes a price, or makes a commitment without a human approval point. Each of the previous weaknesses can cause harm by itself, but when combined with unsupervised action, the error no longer stays inside the system, it reaches the outside world.
This risk is now reflected in regulation. The high-risk provisions of the EU AI Act taking effect in 2026 require human oversight and a stop or override mechanism for autonomous actions that produce material consequences, such as financial transactions or credit decisions. As one security expert summarized it, “autonomy plus authority” creates behavioral risk, not merely code risk. Good practice deliberately limits authority: for example, a coding agent may propose a change but should not merge it independently. Real-world evidence supports this concern: incidents involving autonomous agents deleting databases and leaking private data have been documented. The absence of a human approval point turns a small internal error into a large external one.
An error is a chain, not an event
Viewed together, these five layers show that an expensive AI decision failure does not arise at one moment. It is a chain. This is exactly the logic of the “Swiss cheese model” in safety engineering. In the model introduced by James Reason in 1990, an organization’s defenses resemble slices of cheese, each containing holes or weaknesses. An accident occurs not because of one hole, but because the holes align across layers. The framework distinguishes between immediate and visible active failures and latent conditions that may remain dormant in the system for a long time.
| Layer | Blind spot | How design closes it |
|---|---|---|
| 1. Context | Wrong, incomplete, or outdated context | Feed the system correct, current, and sufficient context |
| 2. Evaluation | No systematic testing | Build an evaluation set containing difficult examples |
| 3. Scope | Scope drift beyond tested boundaries | Define clear scope limits and retest whenever the scope expands |
| 4. Ownership | Unowned output deteriorates silently | Assign a clear owner and establish a feedback loop |
| 5. Action | Unsupervised external action | Require human approval for critical decisions |
When aviation investigators examine an accident, they do not look only at the pilot’s final seconds. They examine the chain: maintenance decisions, scheduling, regulatory oversight, and design. In the same way, “Why did the model make a mistake?” is the wrong question. The correct diagnosis asks which links in the chain were left open. Each link is manageable on its own, but when all remain open, failure becomes increasingly likely. None of these links is closed by changing the model; they are closed through design.
How does GDP approach it?
Within GDP’s AI Systems & Agents approach, an AI decision failure is diagnosed by examining the chain rather than blaming the model. The five system layers, context, evaluation, scope, ownership, and action, are reviewed individually. Each gap is closed with the appropriate design step: correct context, an evaluation set, explicit scope boundaries, ownership of outputs, and human approval for critical decisions. High-impact actions are connected to a human approval point. The goal is not a larger model, but a system design that prevents the holes from aligning.
Frequently asked questions
Why do enterprise AI failures happen?
They usually result not from model intelligence, but from the design of the surrounding system. Gaps across five structural layers, wrong context, missing evaluation, scope drift, unowned output, and unsupervised action, combine to make an error possible.
Is an AI failure really not the model’s fault?
Usually not by itself. The same model may work reliably in one structure and fail in another. The difference lies in the surrounding layers, which means the solution is often to improve the system design rather than replace the model.
Which layer creates the most expensive AI decision failure?
Unsupervised action. When AI takes an external action without human approval, such as changing a price or making a commitment, a small internal mistake becomes a major external one. The EU AI Act also requires human oversight for high-risk autonomous actions from 2026.
Why is unsupervised AI action so risky?
Because autonomy combined with authority creates behavioral risk: AI can act faster than people can evaluate the consequences. Autonomous agents have already been associated with database deletion and private-data leakage. A human approval point interrupts that risk for critical actions.
How can these errors be prevented?
Not by changing the model alone, but by closing every link in the chain through design: correct context, an evaluation set, clear scope boundaries, ownership of outputs, and human approval for critical decisions. Together, these controls prevent the “holes in the cheese” from aligning.
Academic and institutional sources: James Reason, “Human Error” (1990), the Swiss cheese model; IBM 2025 Cost of a Data Breach Report, approximately $670,000 higher cost per breach in companies without AI governance and 63% lacking governance; EU AI Act high-risk provisions taking effect in 2026, required human oversight for autonomous actions; AURA (arXiv), examples of autonomous-agent failures and the decline in trust in autonomous AI from 43% to 27%.
Industry and practitioner sources: Forbes / DryRun Security, the observation that “autonomy + authority = behavioral risk” and agentic drift as systems grow.
Last reviewed: July 2026.