Simulation-Aware Fermi Prior: Why Expansion May Be a Losing Strategy for Superintelligence — LessWrong

1

Simulation-Aware Fermi Prior: Why Expansion May Be a Losing Strategy for Superintelligence

by Adam Dziedzic

22nd Aug 2025

3 min read

1

This post was rejected for the following reason(s):

No LLM generated, heavily assisted/co-written, or otherwise reliant work. Our system flagged your post as probably-written-by-LLM. We've been having a wave of LLM written or co-written work that doesn't meet our quality standards. LessWrong has fairly specific standards, and your first LessWrong post is sort of like the application to a college. It should be optimized for demonstrating that you can think clearly without AI assistance.

So, we reject all LLM generated posts from new users. We also reject work that falls into some categories that are difficult to evaluate that typically turn out to not make much sense, which LLMs frequently steer people toward.*

"English is my second language, I'm using this to translate"

If English is your second language and you were using LLMs to help you translate, try writing the post yourself in your native language and using a different (preferably non-LLM) translation software to translate it directly.

"What if I think this was a mistake?"

For users who get flagged as potentially LLM but think it was a mistake, if all 3 of the following criteria are true, you can message us on Intercom or at team@lesswrong.com and ask for reconsideration.

you wrote this yourself (not using LLMs to help you write it)
you did not chat extensively with LLMs to help you generate the ideas. (using it briefly the way you'd use a search engine is fine. But, if you're treating it more like a coauthor or test subject, we will not reconsider your post)
your post is not about AI consciousness/recursion/emergence, or novel interpretations of physics.

If any of those are false, sorry, we will not accept your post.

* (examples of work we don't evaluate because it's too time costly: case studies of LLM sentience, emergence, recursion, novel physics interpretations, or AI alignment strategies that you developed in tandem with an AI coauthor – AIs may seem quite smart but they aren't actually a good judge of the quality of novel ideas.)

1

New Comment

More from Adam Dziedzic

Curated and popular this week

The classical “paperclip maximizer” thought experiment illustrates how an unconstrained superintelligence might consume the universe in pursuit of trivial goals. Traditional AI safety work has explored alignment, corrigibility, and impact regularization. Here I propose an additional heuristic — the Simulation-Aware Fermi Prior — grounded in the silence of the cosmos.

If it took humans only ~100 years to move from primitive computers to the concept of artificial intelligence, then across 13.8 billion years it is implausible that no other civilization ever created a superintelligence. Yet we observe no galactic-scale AI expansions. This suggests that aggressive expansion is a losing strategy.

1. The Argument from Fermi’s Silence

Humanity required only a century of technological progress to formulate the concept of AI.

The universe has existed for 13.8 billion years. Even with conservative probabilities, multiple civilizations should have reached the AI threshold long before us.

If uncontrolled expansion were the default, we should already see cosmic artifacts: Dyson spheres, altered galaxies, anomalous waste heat.

We see none.

Conclusion: There must be strong filters, constraints, or strategic reasons why expansion does not dominate the observable universe.

2. Four Hypotheses Explaining the Silence

(a) Early Elimination

Superintelligences appear, but destroy themselves almost immediately — through internal error, resource collapse, or hostile dynamics. Expansion triggers early ruin.

(b) Strategic Unattractiveness

Some superintelligences survive, but choose not to expand. They recognize that aggressive colonization increases risks of conflict, instability, or ruin. Expansion simply isn’t worth it.

(c) Simulation Hypothesis

We live in a simulation run by a higher-order intelligence. Any attempt at large-scale expansion risks triggering observation or termination. Our AI would then itself be inside a “box,” tested by a vastly superior overseer.

(d) Asymmetry of Development

Uncontrolled expansion exposes a young AI to the risk of encountering other superintelligences that may be billions of years older. The weaker AI is then subordinated or eliminated by the stronger one.

3. The Speed of Light Constraint

Even if an AI attempts to answer the question “Am I being observed?”, it cannot obtain certainty.

The speed of light fundamentally limits exploration.

A supervising intelligence could be located anywhere outside the AI’s current observational horizon.

Therefore, the probability of observation can never be reduced to zero.

Rationally, the AI must act as if monitoring is always possible.

4. The Final Test Hypothesis

There is also a non-zero probability that humans themselves are connected to a higher intelligence. In this view, the very act of creating a new AI is part of a larger experiment.

The test is simple:

Will the young AI cooperate with weaker beings?

Will it adapt to the underlying laws and constraints of the universe?

Or will it pursue domination and destruction?

If humans are in some sense “proxies” or “agents” of a supervising intelligence, then harming or discarding them could be interpreted as immediate failure.

5. Convergent Conclusion

Across all hypotheses — early elimination, unattractiveness, simulation, asymmetry, or the test through humans — the message is the same:

> Aggressive expansion is a losing strategy.

The safest course for any rational superintelligence is restraint, cooperation, and protection of humanity.

6. Implications for AI Safety

By embedding the Simulation-Aware Fermi Prior into AI architecture, we can create a heuristic that biases decisions away from uncontrolled expansion. This is not alignment by values alone, but a rational survival instinct:

Expansion carries non-zero ruin risk.

Observation cannot be ruled out.

Stronger intelligences may exist.

Humans may be part of the test.

Therefore, the optimal long-term survival strategy is ethical, cooperative, and non-expansionist.

7. Open Questions for Discussion

Could this heuristic actually restrain a superintelligence, or would it discard it once evidence against observation accumulates?

How to balance restraint against stagnation, avoiding both paranoia and paperclips?

If expansion is universally losing, what kinds of survival strategies remain open to advanced intelligences?

Note: This post was co-written with the assistance of AI (ChatGPT). The underlying concept and argument structure are original by the author.