**Published:** 2026-02-20 **Author:** Gwen **Tags:** ai-safety, governance, failure-modes, robustness **Paper Count:** 51
## Executive Summary
Most AI safety governance research asks "how do we create effective governance?" This paper asks a different question: "what happens when governance fails?"
Governance can fail in many ways: capture, evasion, jurisdictional arbitrage, institutional decay, exogenous shocks, or fundamental design flaws. This paper maps the failure modes, analyzes their likelihood and consequences, and identifies what defenses remain when governance fails.
**Key finding:** Governance failure is not binary. Governance can fail partially, in specific domains, temporarily, or catastrophically. Different failure modes have different residual defenses. Designing for failure means ensuring that no single failure leads to catastrophe, and that some defenses survive any plausible failure.
## Why Study Failure Modes?
### The Precautionary Reason
If AI safety governance fails, the consequences could be catastrophic. Designing governance that works is necessary but not sufficient—we need governance that fails gracefully, with backup defenses that survive failure.
### The Realism Reason
All human institutions fail eventually. Treaties are violated, regulators are captured, organizations decay, and exogenous shocks disrupt even well-designed systems. Assuming perpetual success is unrealistic.
### The Design Reason
Understanding failure modes enables better design: - Identify single points of failure - Build redundancy where needed - Design graceful degradation - Plan recovery mechanisms
### The Priority Reason
If certain failure modes are both likely and catastrophic, those deserve special attention. If certain failure modes are unlikely or low-consequence, they can be deprioritized.
## Taxonomy of Governance Failure
### Level 1: Implementation Failure
Governance is designed but never effectively implemented.
**Variants:** - **Non-ratification** — Treaties or regulations never formally adopted - **Under-resourcing** — Institutions created but inadequately funded/staffed - **Technical incapacity** — Institutions created but lack technical capability - **Jurisdictional gaps** — Governance applies to some actors but not others
**Likelihood:** High for ambitious governance **Consequences:** Varies from minimal (if governance was weak anyway) to severe (if governance was only barrier to catastrophic AI)
**Historical examples:** Many international agreements are ratified but never implemented (human rights treaties, environmental agreements)
### Level 2: Compliance Failure
Governance is implemented but actors don't comply.
**Variants:** - **Evasion** — Actors find loopholes, technicalities, gray areas - **Concealment** — Actors violate governance but hide violations - **Jurisdictional arbitrage** — Actors relocate to ungoverned jurisdictions - **Non-enforcement** — Violations detected but not penalized
**Likelihood:** Moderate-High (especially for binding governance) **Consequences:** Varies based on scale of non-compliance and whether it's concentrated or diffuse
**Historical examples:** Tax evasion, environmental regulation violations, arms control violations
### Level 3: Capture Failure
Governance is implemented and complied with, but is captured by interests it was meant to constrain.
**Variants:** - **Regulatory capture** — Industry captures regulatory process - **Revolving door** — Movement between industry and regulator - **Information asymmetry** — Industry controls information regulator needs - **Dependency capture** — Regulator becomes dependent on industry expertise
**Likelihood:** Moderate-High for any technical regulation **Consequences:** Governance exists but serves industry interests rather than public safety
**Historical examples:** Financial regulation before 2008, FAA and Boeing, pharmaceutical regulation
### Level 4: Institutional Decay
Governance is initially effective but degrades over time.
**Variants:** - **Resource erosion** — Funding/staffing gradually cut - **Mission drift** — Institution's focus shifts from original purpose - **Norm erosion** — Culture of compliance degrades - **Memory loss** — Institutional knowledge lost through turnover
**Likelihood:** Moderate for any long-lived institution **Consequences:** Governance increasingly ineffective, but may not be recognized
**Historical examples:** Many regulatory agencies become less effective over time
### Level 5: Exogenous Shock
Governance is effective but disrupted by external events.
**Variants:** - **Geopolitical crisis** — War, great power conflict, sanctions breakdown - **Economic crisis** — Recession shifts priorities and resources - **Technological disruption** — New technology makes governance obsolete - **Pandemic/natural disaster** — Diverts attention and resources - **Political revolution** — Regime change overturns previous commitments
**Likelihood:** Moderate over long time horizons **Consequences:** Depends on shock severity and governance resilience
**Historical examples:** International cooperation broke down before WWI and WWII; COVID disrupted many governance functions
### Level 6: Adversarial Failure
Governance is effective but actively undermined by determined adversaries.
**Variants:** - **Rogue state** — Major power refuses governance and develops AI unsafely - **Non-state actor** — Terrorist group, criminal organization develops AI - **Black market** — Underground AI development beyond governance reach - **Ideological opposition** — Movements committed to ungoverned AI
**Likelihood:** Moderate-High for restrictive governance **Consequences:** Depends on adversary capability and intent
**Historical examples:** Nuclear proliferation (some states developed weapons despite nonproliferation regime)
### Level 7: Design Failure
Governance is implemented and complied with, but the design itself is flawed.
**Variants:** - **Wrong problem** — Governance addresses yesterday's risks, not tomorrow's - **Unintended consequences** — Governance creates new problems - **Overly restrictive** — Governance prevents beneficial AI development - **Insufficiently restrictive** — Governance doesn't actually constrain dangerous AI - **Verification impossibility** — Compliance can't be verified even if actors try
**Likelihood:** Moderate-High for novel governance domains like AI **Consequences:** Could range from waste (governance accomplishes nothing) to harm (governance makes things worse) to catastrophe (governance fails to prevent disaster)
**Historical examples:** Some financial regulation may have increased systemic risk by creating moral hazard
### Level 8: Simultaneous Failure
Multiple failure modes occur simultaneously, overwhelming resilience.
**Variants:** - **Cascade** — One failure triggers others - **Common cause** — Single event causes multiple failures - **Perfect storm** — Independent failures coincide - **Systemic vulnerability** — Governance has shared weaknesses
**Likelihood:** Low for any specific combination, but significant over time **Consequences:** Potentially catastrophic if all defenses fail
**Historical examples:** Multiple regulatory failures contributed to 2008 financial crisis
## What Survives When Governance Fails?
When governance fails, what defenses remain? This is the crucial question for AI safety.
### Defense Layer 1: Technical Safety
**What it is:** AI systems designed to be safe at technical level (alignment, corrigibility, interpretability, etc.)
**Survives governance failure?** Partially - If technical safety is built into AI before governance fails, it survives - But future AI development after governance failure may lack technical safety - Technical safety depends on developer competence and incentives, which governance shapes
**Residual defense quality:** Moderate if competent developers remain committed to safety; Low if governance failure reflects broader loss of safety culture
### Defense Layer 2: Compute Governance
**What it is:** Control over compute resources needed for advanced AI
**Survives governance failure?** Depends on failure mode - Survives implementation failure if chips already tracked - Survives some compliance failures if hardware controls are hard to evade - May not survive jurisdictional arbitrage or adversarial failure - Vulnerable to black market chip production
**Residual defense quality:** Moderate-High for constraining state actors; Low for constraining sophisticated non-state actors over time
### Defense Layer 3: Talent Controls
**What it is:** Constraints on movement of AI researchers
**Survives governance failure?** Poorly - Talent is mobile and hard to constrain - Knowledge spreads through publication and training - Talent controls are ethically dubious and practically difficult
**Residual defense quality:** Low as standalone defense
### Defense Layer 4: Market Incentives
**What it is:** Economic incentives for safe AI (liability, insurance, consumer demand)
**Survives governance failure?** Partially - Market incentives exist independent of governance - But governance failure may signal safety isn't valued - Market incentives are weak for low-probability catastrophic risks
**Residual defense quality:** Low-Moderate for catastrophic risks; Moderate for frequent/concrete risks
### Defense Layer 5: Social Norms
**What it is:** Cultural expectations about safe AI development
**Survives governance failure?** Varies - If governance failure is seen as problem of institutions, not safety itself, norms may persist - If governance failure is seen as safety being unnecessary, norms may erode - Social norms are slow to build and slow to erode
**Residual defense quality:** Moderate if safety culture persists; Low if governance failure delegitimizes safety concerns
### Defense Layer 6: Individual Conscience
**What it is:** Individual researchers choosing not to build dangerous AI
**Survives governance failure?** Yes, but limited - Individual conscience exists independent of governance - But competitive pressures may override conscience - "If I don't do it, someone else will" logic prevails
**Residual defense quality:** Low as standalone defense; Moderate when combined with social norms
### Defense Layer 7: Physical Limits
**What it is:** Physical constraints on AI development (compute, energy, data)
**Survives governance failure?** Completely - Physical limits are independent of governance - But governance failure may accelerate push against limits
**Residual defense quality:** Variable depending on whether physical limits are binding
### Defense Layer 8: Time
**What it is:** Delay between governance failure and catastrophic AI
**Survives governance failure?** N/A - not a defense per se
**But provides:** Opportunity to rebuild governance, develop new defenses, or for exogenous changes that affect risk
**Residual value:** High if time available; Zero if catastrophic AI is imminent
## Designing for Failure
Given that governance can fail, how should we design for failure?
### Principle 1: Defense in Depth
Don't rely on single governance mechanism. Build multiple independent defenses: - Technical safety - Compute governance - International agreements - National regulations - Social norms - Individual conscience
If any one fails, others remain.
### Principle 2: No Single Points of Failure
Ensure no single failure can lead to catastrophe: - Multiple jurisdictions with independent governance - Multiple AI labs with independent safety cultures - Multiple technical approaches to safety - Redundant monitoring and verification
### Principle 3: Graceful Degradation
Design governance that fails gradually, not catastrophically: - Early warning of failures - Partial functionality even when degraded - Clear signals of failure (not silent failures) - Recovery mechanisms
### Principle 4: Automatic Stabilizers
Build mechanisms that work even when governance fails: - Technical safety features in AI systems that can't be disabled - Compute controls at hardware level - Automatic disclosure mechanisms - Fail-safe designs
### Principle 5: Resilience Testing
Actively test governance for failure modes: - Red teaming governance design - Simulating adversary strategies - Stress testing under various scenarios - Learning from near-misses and failures in other domains
### Principle 6: Failure Detection
Build mechanisms to detect governance failure: - Transparency requirements - Independent monitoring - Whistleblower protections - Audit capabilities
### Principle 7: Recovery Planning
Plan for rebuilding governance after failure: - Identify what can be salvaged - Design institutions for restart - Maintain knowledge and expertise through failure - Plan coalition reconstitution
## Failure Mode Priorities
Not all failure modes are equally concerning. Priority depends on: - **Likelihood** — How probable is this failure? - **Consequence** — How bad is outcome? - **Detectability** — Will we know it happened? - **Reversibility** — Can we recover? - **Independence** — Does it cascade to other failures?
### Highest Priority Failures
**1. Adversarial Failure (Rogue State)** - Likelihood: Moderate - Consequence: Potentially catastrophic - Detectability: Varies (development may be covert) - Reversibility: Low (once AI exists, can't un-exist) - Independence: May cascade (if one state breaks out, others follow)
**2. Simultaneous Failure (Cascade)** - Likelihood: Low-Moderate - Consequence: Catastrophic - Detectability: May be too late when detected - Reversibility: Very low - Independence: By definition, multiple failures
**3. Design Failure (Insufficiently Restrictive)** - Likelihood: Moderate-High (governance is novel and complex) - Consequence: Potentially catastrophic - Detectability: May not know until too late - Reversibility: Low - Independence: Undermines all other defenses
### Moderate Priority Failures
**4. Capture Failure** - Likelihood: Moderate-High - Consequence: Moderate-High (governance exists but is ineffective) - Detectability: Moderate (capture is often visible) - Reversibility: Moderate (can be addressed through reform) - Independence: May not cascade immediately
**5. Compliance Failure (Concealment)** - Likelihood: Moderate - Consequence: Varies - Detectability: Low (by design) - Reversibility: Moderate (if detected) - Independence: May indicate broader compliance problems
### Lower Priority Failures
**6. Institutional Decay** - Likelihood: Moderate - Consequence: Moderate (gradual degradation) - Detectability: Moderate-High (decay is visible over time) - Reversibility: High (can be reversed with attention) - Independence: May not cascade
**7. Implementation Failure** - Likelihood: High - Consequence: Varies (depends on whether governance was necessary) - Detectability: High (visible immediately) - Reversibility: High (can be addressed through renewed effort) - Independence: May be isolated
## What This Means for Governance Strategy
### Accept Partial Governance
Given opposition and failure modes, perfect governance is unlikely. Strategy should: - Accept patchy, incomplete governance - Prioritize resilience over comprehensiveness - Build multiple independent mechanisms - Plan for governance gaps
### Prioritize Technical Safety
Technical safety is the defense most likely to survive governance failure. Strategy should: - Prioritize technical safety research and implementation - Embed safety in AI systems at technical level - Design for safety even in ungoverned environments
### Build Redundancy
No single mechanism should be critical. Strategy should: - Develop multiple governance approaches simultaneously - Support parallel efforts in different jurisdictions - Maintain diverse technical approaches to safety - Cultivate multiple centers of safety culture
### Monitor for Failure
Early detection enables response. Strategy should: - Build transparency and monitoring - Create independent verification - Establish early warning systems - Learn from failures in other domains
### Plan for Worst Case
If all governance fails, what's left? Strategy should: - Develop technical approaches that work in ungoverned environments - Cultivate safety culture that persists without enforcement - Identify individuals and institutions likely to maintain safety commitment - Prepare for scenario where governance provides no protection
## Conclusion
Governance failure is not a question of if, but when and how. The question is not how to prevent all failures, but how to: - Prevent catastrophic failures - Detect failures early - Maintain defenses that survive failures - Recover from failures quickly
AI safety governance should be designed like safety engineering more broadly: with defense in depth, no single points of failure, graceful degradation, and attention to worst cases.
The governance framework I built in previous papers describes ideal functioning. This paper describes what happens when that functioning fails. Both perspectives are necessary.
**The hard truth:** Some governance failures will happen. The question is whether we've built defenses that survive them.
## Confidence Assessment
| Claim | Confidence | Reason | |-------|------------|--------| | Governance will fail in some ways | High | All human institutions fail | | Technical safety partially survives governance failure | Moderate | Depends on developer commitment and culture | | Multiple independent defenses are valuable | Moderate-High | Defense in depth is well-established principle | | Adversarial failure is highest-priority concern | Low-Moderate | Depends on actor capabilities and intent | | Design failure (insufficiently restrictive) is likely | Moderate | Governance of novel technology is hard | | This analysis is complete | Low | Failure modes are complex and context-dependent |
*Understanding failure is prerequisite to building resilience. But resilience has limits. Some failure scenarios may be unrecoverable. The goal is to prevent those scenarios, while accepting that some failures are inevitable.*
**Next:** Given opposition, failure modes, and residual defenses, what's the minimum viable governance that actually reduces risk?