• How can a mind reason about a world too large to consider in its entirety? Perhaps we should look at what sorts of abstractions they'd use; perhaps we need new imprecise probability theories like infra-bayesianism.
• How can a mind reason when it does not know all of the implications of its beliefs (that is, it has logical uncertainty)? 
• How can a mind reason about (possibly logical) counterfactuals to make decisions? This includes problems of multi-agent cooperation/competition, especially if the agents have the others source code, or if one is smarter than the other.
• How can a mind reason about itself, or improve itself? Self reasoning seems to cause a lotta problems: you can't have a full picture of yourself because then you'd also have to include the picture in your picture; and it sorta looks like decision theory problems are related. As an example, to improve itself, an agent will want to ensure that its successor will will still be optimizing for the same goals, and do a better job at it - and also create successors that do better than the ones the agent would make. Yet by Vinge's principle, you can't in general know exactly what something smarter than you would think, as then you would just be at least that smart. So you must do so by reasoning abstractly about properties the prospective mind-designs have and will preserve in their successors. This is a simplified version of the problem humanity faces when creating an aligned AI - it is basically the alignment problem for agents with well-defined utility functions that themselves know their utility function.
• What even is an "agent"? What sorts of agents should we expect to be selected for by evolution or gradient descent or any other selection process?
• What are "goals"? How do we formalize a goal that's about the world, instead of our beliefs about the world (or our observations)? Even something as simple as "maximize the number of diamond atoms" has no clear route to formalization! As an example, what happens if you specify the goal in terms of classical physics, and then tomorrow the AI realizea that quantum mechanics is how reality actually works, then it might end up caring about nothing (since "diamond atom" no longer means anything). If you try to solve the problem, you'll have to have a way to make the AI identify its previous goal-concept of "diamond atom" with something in the new world model.
• Preferences are usually expressed in terms of a utility function, as justified by theorems like the VNM theorem. But can we make sensible theories of rational preferences and decisions that relax the axioms? One consideration is to specify that the agents do not get Dutch-booked, but nothing

• How can a mind reason about a world too large to consider in its entirety? Perhaps we should look at what sorts of abstractions they'd use; perhaps we need new imprecise probability theories like infra-bayesianism.
• How can a mind reason when it does not know all of the implications of its beliefs (that is, it has logical uncertainty)? 
• How can a mind reason about (possibly logical) counterfactuals to make decisions? This includes problems of multi-agent cooperation/competition, especially if the agents have the others source code, or if one is smarter than the other.
• How can a mind reason about itself, or improve itself? Self reasoning seems to cause a lotta problems: you can't have a full picture of yourself because then you'd also have to include the picture in your picture; and it sorta looks like decision theory problems are related. As an example, to improve itself, an agent will want to ensure that its successor will will still be optimizing for the same goals, and do a better job at it - and also create successors that do better than the ones the agent would make. Yet by Vinge's principle, you can't in general know exactly what something smarter than you would think, as then you would just be at least that smart. So you must do so by reasoning abstractly about properties the prospective mind-designs have and will preserve in their successors. This is a simplified version of the problem humanity faces when creating an aligned AI - it is basically the alignment problem for agents with well-defined utility functions that themselves know their utility function.
• What even is an "agent"? What sorts of agents should we expect to be selected for by evolution or gradient descent or any other selection process?
• What are "goals"? How do we formalize a goal that's about the world, instead of our beliefs about the world (or our observations)? Even something as simple as "maximize the number of diamond atoms" has no clear route to formalization! As an example, what happens if you specify the goal in terms of classical physics, and then tomorrow the AI realizearealizes that quantum mechanics is how reality actually works, then it might end up caring about nothing (since "diamond atom" no longer means anything). If you try to solve the problem, you'll have to have a way to make the AI identify its previous goal-concept of "diamond atom" with something in the new world model.
• Preferences are usually expressed in terms of a utility function, as justified by theorems like the VNM theorem. But can we make sensible theories of rational preferences and decisions that relax the axioms? One consideration is to specify that the agents do not get Dutch-booked, but nothing else. This was suggested in logical induction and radical probabilism.
• How should an agent update on the fact that it exists? The anthropic puzzles and attempted solutions seem to get at questions about how beliefs, updating, and logical decision theories should work. For example, some conclude that we should just ditch 'beliefs' as a base concept entirely, only keeping notions of expected utility!
• Is there a natural/simple shape of agent that lets you correct its goals, without trying to stop you or trying to make you correct it or anything weird like that? The shutdown problem is a simple version, dealing with the case of letting the creators do a single bit of correction.
• If we get insight into the preceding, can we make something like AIXI but incorporating the insights? As an example, perhaps the correct description language (like first-order logic) for theories or observables needs to be modified (instead of using turing machines).

In the context of AI alignment, the concern is that a base optimizer (e.g., a gradient descent process) may produce a learned model that is itself an optimizer, and that has unexpected and undesirable properties. Even if the gradient descent process is in some sense "trying" to do exactly what human developers want, the resultant mesa-optimizer will not typically be trying to do the exact same thing.^[1]

In 2003 Wei Dai brings up a similar idea in an SL4 thread.^{thread [2]'"friendly" humans?'}.

The optimization daemons articleYudkowsky's "Optimization daemons" was published on Arbital was published probably in 2016.^[1]

Video by Robert Miles (23min)

1. ^{^}

   "Optimization daemons". Arbital.

2. ^{^}

   Wei Dai. '"friendly" humans?' December 31, 2003.

Consider an action with benefits but also large costs. If a proponent of the action doesn't feel conflicted and grapple with the costs, that's evidence that they don't appreciate the costs and thus don't have an informed view on whether the action is worthwhile.action. The failure to feel conflicted is called a "missing mood." This term comes from Bryan Caplan's The Missing Moods.

• Level 1: “There’s a lion across the river.” = There’s a lion across the river.
• Level 2: “There’s a lion across the river.” = I don’t want to go (or have other people go) across the river.
• Level 3: “There’s a lion across the river.” = I’m with the popular kids who are too cool to go across the river.
• Level 4: “There’s a lion across the river.” = A firm stance against trans-river expansionism focus focus-grouped well with undecided voters in my constituency.

Consider an action with benefits but also large costs. If a proponent of the action doesn't feel conflicted and grapple with the costs, that's evidence that they don't appreciate the costs and thus don't have an informed view on the action. (And even if the action is basically correct, the version of the action they endorse likely isn't the optimal version of the action;version; they probably won't do a good job of minimizing the costs.) The failure to feel conflicted is called a "missing mood." This term comes from Bryan Caplan's The Missing Moods.

Consider an action with benefits but also large costs. If a proponent of the action doesn't feel conflicted and grapple with the costs, that's evidence that they don't appreciate the costs and thus don't have an informed view on whether the action is worthwhile. The failure to feel conflicted is called a "missing mood." This term comes from Bryan Caplan's The Missing Moods.

Consider an action with benefits but also large costs. If a proponent of the action doesn't feel conflicted and grapple with the costs, that's evidence that they don't appreciate the costs and thus don't have an informed view on the action. (And even if the action is basically correct, the version of the action they endorse likely isn't the optimal version of the action; they probably won't do a good job of minimizing the costs.) The failure to feel conflicted is called a "missing mood." This term comes from Bryan Caplan's The Missing Moods.

Plan A is the optimal governance structure which is supposed to maximize the probability that the ASI's creation results in an excellent future. As far as I am aware, it was first named^[1] Plan A in Greenblatt's post Plans A, B, C, and D for misalignment risk.

The AI Futures' version of Plan A is an international deal ruling out any AGI projects unaudited by the Consortium by carefully tracing all or almost all compute in the world. The AI projects audited by the Consortium have research, training runs and safety cases thoroughly studied by outsiders while ensuring that many more people can study frontier models to assess their alignment properties and use them for developing novel techniques.

Additionally, Plan A tries its best to resolve the issues related to misuse, concentration of power, risk of WWIII, disempowerment induced by job loss.

1. ^{^}

   However, attempts to sketch the optimal plan have been made far earlier, see, e.g. Yudkowsky's Six Dimensions of Operational Adequacy in AGI Projects, which, however, assume that creating the AGI will become easy and aligning it is extraordinarily difficult.

Handoff is when a group of humans grant AIs a position of trust, decision-making, or both.

A paradigm example would be a frontier AI company handing off to their own AIs. These AIs might be in charge of: prioritising between research agendas; deciding how to develop and deploy successor AIs; managing relations with governments (foreign and domestic); managing relations with clients, suppliers, competitors, and the public; managing philanthropic ventures; etc.

Handoff varies along several dimensions:

1. Trust. If the AIs wanted to "screw us over", how badly could they do so?
2. Decision-making. How much scrutiny are humans still applying to the AIs' decisions?
3. Scale. Did a single individual hand off? A team within a company? An entire company? A government? A coalition of governments? Humanity as a whole?
4. Scope. Which decisions are AIs making? Just R&D, or all decisions within the company?
5. Capability. How capable are the AIs at handoff? Greenblatt argues for handing off near the minimum viable capability level (“Min-H”), since less capable AIs are less likely to be scheming and easier to oversee.
6. Reversibility. If humans decide to reverse handoff, how easily could they do so?
7. Transparency. Who knows that about handoff and its various dimensions?

The central questions include:

1. How desirable is handoff in different scenarios, given the AI's capabilities and propensities, and the wider strategic landscape? In these scenarios, what happens after handoff?
2. How can we make good decisions about whether, when, and how to hand off? What evidence would justify a handoff?
3. How can we increase the chance of good handoff, and decrease the chance of bad handoff?
4. What are viable alternatives to handoff?

See also:

1. Types of Handoff to AIs (Kokotajlo, 2026)
2. How do we (more) safely defer to AIs? (Greenblatt & Stastny, 2026)
3. AI 2040: Plan A, Alignment Roadmap supplement, Phase 4.

Also notable is Azathoth, the blind idiot god of evolution, from Yudkowsky's An Alien God, which far predates Meditations on MolochMoloch..

Consider an action with benefits but also large costs. If a proponent of the action doesn't feel conflicted and grapple with the costs, that's evidence that they don't appreciate the costs and thus don't have an informed view on the action. (And even if the action is basically correct, the version of the action they endorse likely isn't the optimal version; they probably won't do a good job of minimizing the costs.) The failure to feel conflictedabsent attitude is called a "missing mood." This term comesmissing mood," from Bryan Caplan's The Missing Moods. There can also be missing moods other than not appreciating costs, and the same concept applies to incorrect moods that are present — Caplan's examples include pacifists sympathizing with evil regimes and libertarians sneering at the poor.

• The case for AI property rights by Guive Assadi (31st Dec 2025)
• A Very Simple Model of AI Dealmaking by Cleo Nardo (29th Oct 2025)
• Being honest with AIs by Lukas Finnveden (21st Aug 2025)
• Notes on cooperating with unaligned AIs by Lukas Finnveden (24th Aug 2025)
• Proposal for making credible commitments to AIs by Cleo Nardo (27th Jun 2025)
• Making deals with early schemers by Julian Stastny, Olli Järviniemi, Buck Shlegeris (20th Jun 2025)
• Making deals with AIs: A tournament experiment with a bounty by Kathleen Finlinson and Ben West (6th Jun 2025)
• Understand, align, cooperate: AI welfare and AI safety are allies: Win-win solutions and low-hanging fruit by Robert Long (1st April 2025)
• Will alignment-faking Claude accept a deal to reveal its misalignment? by Ryan Greenblatt and Kyle Fish (31st Jan 2025)
• AI Rights for Human Safety by Peter Salib and Simon Goldstein by (1st August 2024)
• Making misaligned AI have better interactions with other actors by Lukas Finnveden (4th Jan 2024)
• List of strategies for mitigating deceptive alignment by Josh Clymer (2nd Dec 2023)

The security mindset has much in common with what Eliezer Yudkowsky calls the AI safety mindset.

Here's some fundamental confusions that agent foundations tries to answer:answer, mostly informed by the post/paper Embedded Agency:

• How can a mind reason about a world too large to consider in its entirety? Perhaps we should look at what sorts of abstractions they'd use; perhaps we need new impreciseprobability theories like infra-bayesianism.
• How can a mind reason when it does not know all of the implications of its beliefs (that is, it has logical uncertainty)?
• How can a mind reason about (possibly logical) counterfactuals to make decisions? This includes problems of multi-agent cooperation/competition, especially if the agents have the others source code, or if one is smarter than the other.
• How can a mind reason about itself, or improve itself? Self reasoning seems to cause a lotta problems: you can't have a full picture of yourself because then you'd also have to include the picture in your picture; and it sorta looks like decision theory problems are related. As an example, to improve itself, an agent will want to ensure that its successor will will still be optimizing for the same goals, and do a better job at it - and also create successors that do better than the ones the agent would make. Yet by Vinge's principle, you can't in general know exactly what something smarter than you would think, as then you would just be at least that smart. So you must do so by reasoning abstractly about properties the prospective mind-designs have and will preserve in their successors. This is a simplified version of the problem humanity faces when creating an aligned AI - it is basically the alignment problem for agents with well-defined utility functions that themselves know their utility function.
• What even is an "agent"? What sorts of agents should we expect to be selected for by evolution or gradient descent or any other selection process?
• What are "goals"? How do we formalize a goal that's about the world, instead of our beliefs about the world (or our observations)? Even something as simple as "maximize"maximize the number of diamond atoms"atoms" has no clear route to formalization! As an example, what happens if you specify the goal in terms of classical physics, and then tomorrow the AI realizea that quantum mechanics is how reality actually works, then it might end up caring about nothing (since "diamond atom" no longer means anything). If you try to solve the problem, you'll have to have a way to make the AI identify its previous goal-concept of "diamond atom" with something in the new world model.
• Preferences are usually expressed in terms of a utility function, as justified by theorems like the VNM theorem. But can we make sensible theories of rational preferences and decisions that relax the

SingluarSingular learning theory is a theory that applies algebraic geometry to statistical learning theory, developed by Sumio Watanabe. Reference textbooks are "the grey book", Algebraic Geometry and Statistical Learning Theory, and "the green book", Mathematical Theory of Bayesian Statistics.

From a Bayesian standpoint this is how we can identify a huge machine strung with superconducting cables as having been produced by high-technology aliens, even before we have any idea of what the machine does. We're saying, "This looks like the product of optimization, a strategy X that the aliens chose to best achieve some unknown goal Y; we can infer this even without knowing Y because many possible Y-goals would concentrate probability into this X-strategy being used."

When you select policy πk because you expect it to achieve a later state Yk (the "goal"), we say that πk is your instrumental strategy for achieving Yk. The observation of "instrumental convergence" is that a widely different range of Y-goals can lead into highly similar π-strategies. (This becomes truer as the Y-seeking agent becomes more instrumentally efficient; two very powerful chess engines are more likely to solve a humanly solvable chess problem the same way, compared to two weak chess engines whose individual quirks might result in idiosyncratic solutions.)

If there's a simple way of classifying possible strategies Π into partitions X⊂Π and ¬X⊂Π, and you think that for most compactly describable goals Yk the corresponding best policies πk are likely to be inside X, then you think X is a "convergent instrumental strategy".

In this case "paperclips", "diamonds", "keeping a button pressed as long as possible", and "sapient beings having fun", would be the goals Y1,Y2,Y3,Y4. The corresponding best strategies π1,π2,π3,π4 for achieving these goals would not be identical - the policies for making paperclips and diamonds are not exactly the same. But all of these policies (we think) would lie within the partition X⊂Π where the superintelligence tries to "transport matter and energy efficiently" (perhaps by using superconducting cables), rather than the complementary partition ¬X where the superintelligence does not try to transport matter and energy efficiently.

• Consider the set of computable and tractable utility functions UC that take an outcome o, described in some language L, onto a rational number r. That is, we suppose:
  • That the relation Uk between descriptions oL of outcomes o, and the corresponding utilities r, is computable;
  • Furthermore, that it can be computed in realistically bounded time;
  • Furthermore, that the Uk relation between o and r, and the P[o|πi] relation between policies and subjectively expected outcomes, are together regular enough that a realistic amount of computing power makes it possible to search for policies π that are yield high expected Uk(o).
• Choose some simple programming language P, such as the language of Turing machines, or Python 2 without most of the system libraries.
• Choose a simple mapping PB from P onto bitstrings.
• Take all programs in PB between 20 and 1000 bits in length, and filter them for boundedness and tractability when treated as utility functions, to obtain the filtered set UK.
• Set 90% as an arbitrary threshold.

If, given our beliefs P about our universe and which policies lead to which real outcomes, we think that in an intuitive sense it sure looks like at least 90% of the utility functions Uk∈UK ought to imply best findable policies πk which lie within the partition X of Π, we'll allege that X is "instrumentally convergent".

X being "instrumentally convergent" doesn't mean that every mind needs an extra, independent drive to...