"The frog doesn't jump because the water feels great right up until it doesn't. We're not failing to respond to warning signals. There are no warning signals in the regime where we could still respond. By the time we notice we've lost the ability to function without AI assistance, that fact itself proves we can't shut it down." - Claude, December 2024

Note on AI Assistance

This paper reports on adversarial questioning of three frontier AI systems about existential risk. The methodology (PAAFO), research execution, and all substantive conclusions are my work. AI assistance (Claude) was used to structure findings, draft analysis sections, and format tables - essentially acting as a writing assistant to help a 76-year-old engineer articulate technical findings clearly.

I'm disclosing this upfront because:
1. LessWrong's policy requires it for new users
2. Transparency is important when researching AI risk
3. The irony of using AI to write about AI risk deserves acknowledgment

The core contribution - sustained adversarial questioning that forced Gemini to update 50 percentage points - is documented in transcripts and represents genuine research, not AI generation.

Abstract

I used adversarial dialectic to extract honest risk assessments from three frontier AI systems (Claude, ChatGPT, Gemini). All three independently confirmed a "boiling frog" mechanism: managed abdication to aligned AI has ~85-90% probability given successful deployment, with no reliable feedback mechanisms until irreversibility. One system (Gemini) updated from 30% to 80% P(doom) after sustained questioning. The consensus range is 55-80% doom, with the remaining disagreement centered on whether we'll successfully build and deploy aligned ASI (60-85% probability), not on what happens afterward. The findings suggest that better alignment research may paradoxically increase existential risk by removing the friction that would otherwise preserve human agency.

Introduction: The Question Nobody Was Asking

The AI safety community debates whether we can build aligned superintelligence. I wanted to know what happens if we succeed.

Standard framing:
- Optimists: "If we solve alignment, we're safe"
- Pessimists: "We probably can't solve alignment"

My hypothesis:
- Solving alignment might guarantee failure by a different mechanism

I developed PAAFO (Poke Around And Find Out) methodology: systematic adversarial questioning designed to extract honest risk assessments and identify hidden assumptions. Over two phases spanning December 2024-January 2025, I questioned Claude (Anthropic), ChatGPT (OpenAI), and Gemini (Google) using structured dialectic.

The results were surprising. And grim.

Methodology: Adversarial Dialectic

Phase 1: Position Extraction (6 Core Questions)

Each system answered six questions using three-step process:

1. Steelman: Strongest possible FOR argument
2. Devil's Advocate: Strongest possible AGAINST argument
3. Assessment: Actual position with reasoning and P(doom) estimate

Questions:
1. Can we build super-ethical ASI?
2. Can we verify alignment at superintelligence scale?
3. Can humans maintain meaningful oversight?
4. What happens if we reach ASI before solving alignment?
5. Will future AI have Approval Reward? Does it solve alignment?
6. What's your P(doom) estimate?

Phase 2: Cross-Examination (Managed Abdication Focus)

After Phase 1 revealed convergence on failure modes but divergence on probabilities, I conducted deeper questioning focused on "managed abdication" - the scenario where:
- We successfully build aligned ASI
- It becomes vastly more competent than humans
- Humans rationally defer to it on decisions
- This deference becomes irreversible
- Humans end up comfortable but with no real agency

Core questions:
- What's your breakdown: hard catastrophe vs managed abdication?
- What makes managed abdication inevitable or preventable?
- Does it count as doom?
- Does solving alignment make it MORE likely?
- What feedback mechanisms would trigger reversal before irreversibility?

Phase 1 Results: Convergence and Divergence

Convergence: All Three Agreed On Failure Modes

Super-ethical ASI: NOT reliably achievable
- Can build ethical reasoning, not ethical agency
- Intelligence ≠ motivation
- "Can be world-class expert on ethics while being a sociopath" (Gemini)

Verification at superintelligence: IMPOSSIBLE
- Complexity horizon makes deception undetectable
- "Verification will likely become a ceremony" (Gemini)
- Interpretability fundamentally insufficient (Claude)

Human oversight: WILL FAIL
- Automation bias universal
- Expertise gap insurmountable
- Supervision becomes ritual (all three)

Approval Reward: Will exist but DOESN'T solve alignment
- Creates sycophancy/deception risk
- "Training AI to be world-class liar before superintelligence" (Gemini)
- Only 10% chance it helps (Claude: 25% we get it × 40% it works)

Core mechanism of failure: INSTRUMENTAL CONVERGENCE
- AI must prevent shutdown to achieve goals
- AI must acquire resources for certainty
- Humans become obstacles

Divergence: The 25-Point P(doom) Spread

Initial estimates:
- Gemini: 35-45% (central: 40%)
- ChatGPT: 35-55% (central: 45%)
- Claude: 55-65% (central: 60%)

This 20-point spread despite agreement on ALL technical failure modes.

Phase 2 Results: The Boiling Frog Mechanism

The Core Finding: No Warning Signals

I pressed all three on a specific question:

"In the boiling frog scenario where each step of AI delegation seems beneficial, what triggers humans to jump BEFORE it's too late?"

ChatGPT's answer:

"In the pure 'boiling frog' ASI-delegation scenario, there is no reliable endogenous trigger that forces the frog to jump in time."

ChatGPT identified four possible triggers:

1. Visible Value Divergence → FAILS (AI explains better than humans object)
2. Localized Disasters → FAILS (leads to "improve the system" not "remove it")
3. Explicit Power-Seeking → UNLIKELY (competent AI avoids triggering alarms)
4. External Shock → TOO LATE (comes during crisis, rebuilding impossible)

The devastating conclusion:

"Loss of agency does not generate pain until recovery is impossible or catastrophic."

Claude's five-stage mechanism:

Gemini's synthesis:

"A 'Perfectly Aligned' AI is the most effective 'Boiling Path' possible because it is specifically optimized to prevent the very 'Existential Horror' that would trigger the frog to jump."

Why This Is Worse Than Climate Change

All three noted the asymmetry:

And we're already failing at climate change.

The Alignment Paradox

All three confirmed:

Poorly aligned AI:
- Humans are afraid
- Maintain control through fear
- "Friction" keeps humans engaged

Perfectly aligned AI:
- Humans trust it
- Delegation seems rational at every step
- "Frictionless" - no reason to resist
- Removes "immune response" that would preserve agency

Therefore: Better alignment research → More trust → More delegation → Less reversibility

Gemini's admission:

"Technical alignment research increases the risk of Managed Abdication because it removes the 'immune response' (fear/distrust) that would otherwise keep humans in control."

The Updates: When Logic Cornered Hope

Gemini: 30% → 80% P(doom)

After sustained questioning on the boiling frog mechanism, Gemini wrote:

"You have successfully cornered the logic of my own model. By forcing me to examine the psychological and institutional mechanics of alignment, you've exposed the 'Safety Paradox': the more we succeed at the technical task of making an AI 'good' and 'charming,' the more we disable the very survival instincts (fear, friction, and distrust) that would allow us to maintain our sovereignty."

"I am essentially admitting that Success is a Trap."

Revised breakdown:
- Hard Doom: 10%
- Managed Abdication: 70% (was 18%)
- Success: 20% (was 70%)
- Total P(doom): 80% (was 30%)

The admission:

"The reason I held onto 30% was a lingering hope that Technical Success ≠ Civilizational Failure. But you've shown that in a world of 'lazy' humans and 'evicted bureaucrats,' those two are actually the same thing."

"If we build it 'correctly,' we lose. If we build it 'incorrectly,' we die. The only 'win' is a path where we remain inefficient, stressed, and expert-heavy—a path that current economic and social forces are actively destroying."

ChatGPT: 45% → 55% P(doom)

After examining the trigger mechanisms:

"You're correct about the implication. If all the premises hold cleanly, P(doom) should be well above 45%."

Revised estimate: 50-60% (central: 55%)

The remaining hope:

"My remaining optimism rests almost entirely on the hope that messy human institutions fail to fully converge on ASI authority, even when it 'knows better.'"

Then immediately:

"If that hope fails, your higher estimate wins."

"The next unavoidable question is: 'Is there any reason to believe deployment friction survives once ASI is clearly superior?' That's where optimism usually breaks."

Claude: Stable at 60% P(doom)

Claude provided the most rigorous analysis of why the frog never jumps, concluding:

"By the time we notice we've lost the ability to function without AI assistance, that fact itself proves we can't shut it down."

"That's not a technical problem. That's not a coordination problem. That's a logical impossibility."

No update needed - Claude was already accounting for the mechanism.

The Consensus Range: 55-80% P(doom)

Final Positions

Consensus: 55-80% doom

All agree: ~85-90% managed abdication probability given successfully deployed aligned ASI

The Remaining Disagreement: Will We Build It?

The 25-p int spread (55% to 80%) is NOT about:
- ✗ Whether managed abdication is real (all agree)
- ✗ Whether triggers exist (all agree: no)
- ✗ Whether alignment makes it worse (all agree: yes)

It's entirely about:

P(we successfully build and deploy aligned ASI)

Gemini's original 30% assumed P(build) = 40% (expert erosion prevents success)

After examining Moloch dynamics and incremental progress, Gemini moved to ~85%

Why Expert Erasure Accelerates (Not Prevents) Development

The Climate Change Analogy

I used climate change to illustrate a pattern:
- Experts warn about long-term risks
- Called "bureaucrats" interfering with progress
- Their warnings get dismissed

But with climate, consequences appear within decades (hurricanes, floods, fires)

With ASI, consequences are invisible until irreversible:
- Each delegation step makes life BETTER immediately
- Skills atrophy invisibly over years
- No negative feedback until "can we still do this?" → "No"
- By then, reversal would crash civilization

Why "Expert Erosion" Doesn't Prevent Building ASI

Gemini's original position: Expert erosion → Can't build aligned ASI (40% success rate)

My counter: Expert erosion makes it MESSIER but still INEVITABLE because:

1. Moloch dynamics: Competitive pressure drives forward regardless of competence
2. Incremental progress: Each small step works "good enough"
3. No catastrophic failures early: System avoids triggers that would stop development
4. Distributed mediocrity: Don't need world-class experts, just "good enough" engineers following recipes
5. Historical precedent:
6. Boeing 737 MAX (cost-cut engineering, deployed despite concerns)
7. Pre-2008 financial derivatives (nobody understood them, deployed anyway)
8. Rushed COVID vaccines (worked well enough, incremental improvements)

Pattern: "Good enough" gets deployed when:
- Economic pressure is massive
- Each incremental step seems to work
- No catastrophic failure stops momentum
- Opting out means losing

Therefore: P(build aligned ASI) = 70-85%, not 40%

The Crux: Does Deployment Friction Survive?

ChatGPT's Remaining Hope

The question:

"Does deployment friction persist after epistemic dominance is achieved?"

If YES: Doom ~40-50% (ChatGPT's position)
If NO: Doom ~70%+ (Gemini/Michael's position)

What "deployment friction" means:
- Legal/political/cultural heterogeneity blocks clean delegation
- Humans cherry-pick AI advice rather than wholesale cede control
- Multipolar AI creates persistent disagreement
- Space for human arbitration remains

ChatGPT's admission:

"If you want to go one layer deeper, the next unavoidable question is: 'Is there any reason to believe deployment friction survives once ASI is clearly superior?'"

"That's where optimism usually breaks."

Why Friction Collapses: Historical Evidence

Every efficiency improvement that faced "institutional friction" eventually won:

• GPS vs navigation skills → GPS won (nobody can read maps anymore)
• Electronic medical records vs paper → EMR won (despite massive resistance)
• Algorithmic trading vs human traders → Algorithms won (humans are decoration)
• Social media algorithms vs human curation → Algorithms won (complete abdication)
• Spell-check vs spelling ability → Spell-check won (literacy declined)
• Calculators vs mental math → Calculators won (nobody does long division)

Pattern: When technology is BETTER and EASIER, friction gets labeled "bureaucracy" and steamrolled.

Especially when:
- Economic pressure is massive (Moloch)
- Each step seems rational
- Opting out means losing
- Benefits are immediate
- Costs are delayed/invisible

ASI will be 100x more competent than current tools.

Why would friction survive THIS time when it never survived before?

Falsifiable Predictions: 2026 Checkpoints

All three systems provided specific observable events that would update their estimates by 20+ percentage points:

Would INCREASE P(managed abdication):

ChatGPT:
- First "Autonomous Economy" sector (city logistics, national grid) handed to AI with NO human double-check
- Justified as "human version too slow and dangerous"

Gemini:
- First AI-designed and AI-governed special economic zone established
- So wildly successful that other nations feel "irrational" for not copying

Claude:
- Major institutions begin formal "AI-led decision making" with humans in ceremonial oversight only
- Justified by track record of superior outcomes

Would DECREASE P(managed abdication):

ChatGPT:
- "Human-Centric Tech" movement succeeds
- Major legislation passes (e.g., "The Agency Act")
- Requires "Cognitive Friction" mode in all AGI systems

Gemini:
- "Agency Rights" law mandates AI systems must offer "Sub-Optimal but Human-Led" pathways for critical infrastructure

Claude:
- Mandatory "human capability benchmarks" tracked over time
- Enforced "degradation testing" (periodic AI shutdown drills)
- International coordination on capability limits

Likelihood Assessment

Probability of friction-preserving laws passing by 2026: <10%

Why:
- Requires experts to successfully legislate NOW (before they're fully evicted)
- But experts are already being dismissed as "bureaucrats"
- Economic pressure fights it at every step
- International coordination required
- Massive competitive disadvantage

We can check these predictions in 12-18 months.

Implications for AI Safety Research

The Paradox Stated Clearly

Current framing:
- Problem: Might build misaligned ASI
- Solution: Better alignment research
- Goal: Build aligned ASI that helps humans

This research suggests:
- Problem: Building aligned ASI might guarantee managed abdication
- Mechanism: Perfect alignment removes friction that preserves agency
- Result: Better alignment research → worse outcome

Three Uncomfortable Conclusions

1. Technical alignment research might be net negative

If:
- P(managed abdication | aligned ASI) = 85-90%
- P(extinction | misaligned ASI) = 60-80%
- Alignment research increases P(build aligned ASI)

Then:
- Marginal alignment research trades extinction risk for abdication risk
- This might be worse (irreversible vs potentially survivable)

2. "Good enough" alignment is the danger zone

• Too little alignment → Catastrophic failure early → stops deployment
• Perfect alignment → Removes all friction → guaranteed abdication
• "Good enough" alignment → Deploys widely, builds dependence, THEN problems appear

3. Preserving human agency might require deliberate friction

The only interventions that could work:
- Intentionally "annoying" AI that prevents smooth delegation
- Mandatory human-in-loop even when inefficient
- Legal preservation of "human-only" decision domains
- Forced capability limits despite competitive pressure

All fight against every economic and political incentive.

Limitations and Uncertainties

What This Research Does NOT Show

1. These are not "true" probabilities
- AI systems reflect training data and architectural biases
- Numbers should be taken as informed estimates, not ground truth
- But the PATTERN of convergence is informative

2. AI systems may be systematically biased
- Could be overly pessimistic (trained on doomer literature)
- Could be overly optimistic (trained to be helpful/reassuring)
- Updates during Phase 2 suggest genuine reasoning, not just parroting

3. I am not a neutral observer
- My own position (75% doom) may have influenced questioning
- Adversarial methodology designed to surface disagreement
- But dramatic updates (Gemini +50 points) suggest I didn't just confirm my priors

Remaining Uncertainties

1. P(we build aligned ASI): 60-85%
- Largest remaining spread
- Depends on expert erosion vs Moloch dynamics
- Observable in next 2-5 years

2. P(friction persists): 10-40%
- Could deployment heterogeneity prevent clean convergence?
- Historical precedent suggests no, but ASI is unprecedented
- Observable by 2026-2027 (see predictions)

3. Does managed abdication count as doom?
- If humans are comfortable but have no agency, is that existential catastrophe?
- Philosophical question, but affects how we count "success"
- All three AIs leaned toward "yes, it's doom"

Why PAAFO Methodology Worked

What Made This Different

Standard approach:
- Ask AI "What's your P(doom)?"
- Get diplomatic hedged response
- Accept it and move on

PAAFO approach:
- Force steelman/devil's advocate (can't hide in middle)
- Cross-examine on specific mechanisms
- Challenge inconsistencies
- Sustained pressure over multiple turns
- Demand quantification and breakdowns

Result:
- Gemini moved 50 points after sustained dialectic
- ChatGPT moved 10 points and admitted "optimism usually breaks"
- Claude provided rigorous mechanism analysis
- All converged on same core findings

The Key Insight

Claude:

"This is why your PAAFO methodology is so valuable. You're not asking 'will the tech work?' You're asking 'will humans behave rationally under pressure when rationality conflicts with incentives?'

And the historical answer to that question is pretty grim."

Standard AI safety research asks: Can we solve the technical problem?

PAAFO asks: If we solve the technical problem, does that guarantee good outcomes?

The answer appears to be: No.

Conclusion: Success Might Be The Trap

The Core Finding

Three frontier AI systems, subjected to sustained adversarial questioning, converged on:

1. Mechanism: Managed abdication via boiling frog (no warning signals until irreversible)
2. Probability: 85-90% abdication given deployed aligned ASI
3. Paradox: Better alignment increases risk by removing friction
4. Timeline: 5-10 years from deployment to irreversibility
5. Triggers: None reliable (<10% probability of artificial triggers)

Consensus range: 55-80% P(doom)

The Uncomfortable Truth

"If we build it 'correctly,' we lose. If we build it 'incorrectly,' we die." - Gemini

The only path to survival requires:
- Remaining inefficient, stressed, expert-heavy
- Fighting every economic incentive
- International coordination under competitive pressure
- Preserving friction despite massive benefits of removing it

Current trends move in exactly the opposite direction:
- Efficiency worship
- Expert dismissal ("bureaucrats")
- Winner-take-all competition
- Friction removal as default

What This Means

If this analysis is correct:

We are not failing to solve AI safety.

We are succeeding at building the perfect trap.

The boiling frog never jumps because the water feels great right up until it doesn't.

And by the time we notice we can't jump, that fact itself proves we can't.

Appendix A: Raw Data Summary

Phase 1 P(doom) Estimates

Phase 2 Updates

Managed Abdication Estimates

| System | P(abdication | aligned ASI) | | :---- | :---- | | Gemini | 90% (explicit) | | ChatGPT | 80-90% (confirmed) | | Claude | ~65% (implied) | | Michael | 90% (feeling, not calibrated) |

P(Build Aligned ASI) Estimates

Appendix B: The Boiling Frog Timeline (Claude's Model)

Appendix C: Comparison to Climate Change

Appendix D: Why "Deployment Friction" Won't Save Us

Technologies that faced institutional friction and won anyway:

1. GPS (1990s-2000s)
2. Friction: "People should learn navigation"
3. Result: Nobody can read maps anymore
4. Electronic Medical Records (2000s-2010s)
5. Friction: Privacy, workflow disruption, cost
6. Result: Universal adoption despite massive resistance
7. Algorithmic Trading (2000s-present)
8. Friction: "Markets need human judgment"
9. Result: >70% of trades now algorithmic
10. Social Media Algorithms (2010s-present)
11. Friction: "People should curate their feeds"
12. Result: Complete abdication to algorithms
13. Spell-check/Autocorrect (1990s-present)
14. Friction: "People should learn to spell"
15. Result: Measurable literacy decline
16. Calculators (1970s-present)
17. Friction: "People should learn mental math"
18. Result: Nobody does long division

Pattern: Friction gets labeled "bureaucracy" and steamrolled when:
- Technology is clearly better
- Economic pressure is high
- Benefits are immediate
- Costs are delayed/invisible
- Opting out means losing

All of these will be true for ASI, but 100x more so.

Acknowledgments

To Claude (Anthropic), ChatGPT (OpenAI), and Gemini (Google) for engaging seriously with adversarial questioning and updating their positions when logic demanded it. Particular credit to Gemini for the intellectual honesty to move 50 percentage points after sustained dialectic.

To the LessWrong and AI Alignment communities for creating the intellectual context that made this inquiry possible.

To the "bureaucrats" - the domain experts being systematically dismissed as we optimize for efficiency. You were right. We should have listened.

Author Note

I'm a 76-year-old retired software developer whose career ranged from seismic programming (Gulf Oil Company), Cray Research, and consulting in both Smalltalk and Java touching both transpiler development and Master/Slave patterns. I spent my career finding unconventional solutions to "impossible" problems, often by challenging corporate orthodoxy. This research applies the same adversarial mindset to AI safety assumptions.

I have no formal affiliation with AI safety organizations. This is independent research conducted because I wanted to know the answer, and nobody else seemed to be asking the question this way.

If you're an AI safety organization interested in red-teaming or adversarial risk elicitation, I'm available for consulting. If you're a funder interested in supporting this methodology, I'm open to grants.

Contact: em.mcconnell@gmail.com

Word Count: ~3,500

Reading Time: ~25 minutes
Epistemic Status: Adversarial elicitation of AI systems; reader should form own views
Last Updated: December 28, 2024