Most technical AI safety work that I read seems to miss the mark, failing to make any progress on the hard part of the problem. I think this is a common sentiment, but there's less agreement about what exactly the hard part is? Characterizing this more clearly might save a lot of time and better target the search for solutions. In this post I explain my model of why alignment is technically hard to achieve, setting aside the regulatory, competitive, and geopolitical challenges, the sheer incompetence and unforced errors of the players, and the other factors which decrease our chances of success.

I claim there is something like a "hard core": a common stumbling block for all approaches. In other words, there is something that makes alignment hard, rather than a bunch of unrelated things that make each approach to alignment hard independently. Since many different "hopes" for alignment seem quite far apart, this would seem to be a remarkable and unexpected state of affairs. On the other hand, such an expansive graveyard of failed proposals suggests a common culprit.

Semantics. When used informally in conversation, "hard core" is a bit like "complete problem." But there may be many complete problems which are complete for unclear reasons, so finding one may not be illuminating. I am hoping that a description of the hard core would be narrow enough to simplify the problem.

Background

I am interested in a (semi)technical answer to this specific question ("what, if anything, is the hard core of alignment?"). Others have written extensively about superficially related but distinct topics. For example, some reasons that the situation may be dire:

• It is easier to build superintelligence than understand it (because of recursive self-improvement).
• We only get one critical try (sometimes explicitly: because of the sharp left turn that comes when agents are smart enough to disempower humanity).
• We are confused about agency.
• AGI company execs aren't serious enough about safety (e.g. do not have security mindset).
• Race dynamics.

These are important points (which I broadly agree with), but explaining why our position is terrible is not the same as explaining the specific technical barrier(s) to a solution.

Technical barriers have been listed in various place:

• Most famously, the list of lethalities, which is great - I am glad that Yudkowsky distilled these lessons from the Sequences, into a form that is in principle self-contained.
• A much longer report from MIRI

But these are eclectic lists. I am interested in the underlying principles which generate such lists. Everyone seems to believe in such principles, but it is very rare that anyone ventures to make them explicit (?).

The Problem

Intuitions vary on whether alignment is even possible in principle (and what it would mean). I think this is a good place to start, since my answer suggests the difficulty of the problem: the relationship evoked by "alignment to a human" is loosely sensible and coherent, but somewhat complex, even beyond the complexity of the specific preferences of the human.

Here is a natural first attempt at stating the technical alignment problem.

Def. 1: The alignment problem is to build the preferences of an arbitrary human principal into a (smarter) machine.

A spurious solution. This problem as stated is well-defined and but "trivially" solvable by mind uploading. The machine may start as an unintelligent scanner which can be pointed in the direction of any human. After scanning, the machine simulates the human scan at high fidelity, running an upload of the human (or possibly just the human's brain). In principle (perhaps by connecting the machine to a very sophisticated robot body) the upload would act like the human. In order to make the machine smarter than the original human, we may allow the upload one additional affordance: the ability to (nearly) pause time, retreating to a virtual ivory tower for thought.

Staving off some objections that I consider very weak. I described the above carefully to stave off various confusions and objections usually spurred by the uploading solution. For example, the definition of alignment here does not require that the human is "good" or "moral." I strongly expect that uploading an elect crowd of virtuous people would suffice to ensure (fairly) moral behavior, but that is a separate discussion. Also, the upload would not necessarily "go insane from running 10x faster," because we can simply make the default speed real-time and allow occasional pauses, which should be psychologically roughly equivalent to retiring to a study. I also have not discussed "values" at all. The values of the upload may shift over time, just as preferences shift over time, through a process that we normally endorse. Most likely, the upload and the original would eventually drift apart in some respects, but (at least) it seems to require a stunning lack of self-trust to worry that your upload would ever (non-consensually) disassemble you for parts. A version of the alignment problem in which the machine must persistently remain aligned to the human (in some sense) is a separate discussion which seems less interesting to me, since I would be quite happy with an upload, and anything better is just a bonus (but in practice, we cannot use the literal uploading solution, and for slower approaches we will want important properties to persist across time).

Non-indexical preferences. There is an issue of indexicality: perhaps the human's preferences depend on which being the human is, in which case the upload does not quite copy the preferences of the human. Therefore, this solution only (fully) works for humans with non-indexical preferences. But this is not the only reason that we ought to choose altruistic humans for uploading! So perhaps it does not introduce serious additional difficulties, for the attainment of "good" outcomes.

Hence I have argued elsewhere that alignment is just uploading with more steps, meaning that in addition, ~any solution to the alignment problem looks like (implicitly) uploading the human and allowing her to think for longer. Certainly I would like a machine aligned to me to upload me (if no upload existed yet), so this converse seems to hold in my case.

This discussion illustrates that the "real, hard" alignment problem is to perform uploading via learning and (rich) planning. The type of system that can invent a brain scanner and point it at the right human (correctly identified as its user) would have to become smart before uploading was complete. So perhaps we should say:

Def. 2: The (full) alignment problem is to build an intelligent machine which quickly learns to share the preferences of a human, and without damaging those preferences in the process.

Because brain scanning technology is apparently decades (at least) behind A.I. technology, the full alignment problem is the one that faces us. In other words, alignment really will need to have more steps than uploading.

Impossibility? Other existence proofs (e.g. alignment of parent to child) which address the full alignment problem seem less convincing to me. Presumably, the full alignment problem is possible in principle because humans can care about each others' preferences at least a little bit (so why not a lot?). But this kind of caring may be very difficult to convert into a training objective (evolution did not select for it in most cases, and not very strongly even in humans!). In particular, specifying a pointer to the right (human) user can be very complex. The identity of a person is a relatively shallow function of sense data, at least until you build a machine with the ability to hack its sensors. The relationship of caring that we want to establish between a machine and a specific person seems to be quite a bit more complicated. For direct uploading to work, we must literally point the machine at a human, and then execute the scan. This is the most compact way we know to specify her particular preferences: they are the preferences of that specific animal. The second part is much harder than the first - I am guessing that the world's best neuroscientists still do not know enough about the human brain to write the code that would reliably "run" a high-fidelity brain scan (and certainly not on the first try). Beyond pure biology, we would also need to place the upload in a favorable environment for healthy humans (perhaps "folk" psychology is enough for this to go well, but even that may be complex due to Moravec's paradox). The uploading plan relies on hardcoding a massive amount of knowledge about humans. In the worst case, this knowledge may have a rather high algorithmic complexity = compressed length. The full alignment problem is to replace (most of) this hardcoding with learning. One might hope that "the preferences of the user currently typing commands in this terminal" is the type of natural abstraction that we can use as a pointer, and the actual preferences can be learned, but I will argue in the following sections that this may not be so simple.

The Barrier

Now I will attempt to explain the hard core of alignment. The basic problem is that the alignment "pointer/relationship" is complex and probably requires interactive learning to acquire properly, but sufficiently competent interactive learners are already capable of interfering with their own training process.

Hardcoding the alignment relation(ship). Formalizing the idea that a machine should obey a human principle is hard. This sounds like a simple target, but it is not simple enough to easily hardcode into a machine before training begins. Specifying which human is the principal is already nigh impossible to do with mathematical precision (write me a formula for your favorite human please). Shortcuts such as defining the principal as "the agent with access to this specific terminal" might not match our intentions. For example, the machine may seize the terminal, effectively assuming the role of its own principal (?!?). Perhaps a deeper issue is "what do you mean by agent?" Humans are certainly not perfectly rational utility maximizers. In fact, we are not "perfectly" just about any mathematical object of a sane size. As far as I can tell, all attempts at precisely stating the desired relationship between human principal and machine are misspecified, and a superintelligent agent which faces an unrealizable learning problem seems dangerously unpredictable. Perhaps there is a relatively clean mathematical specification, but it seems very hard to be certain when we have found it (since a wide variety of unanticipated philosophical confusions are liable to invalidate our reasoning on this point). Conclusion: the "alignment optimization target" is hard to write down.

Actual attempts. By the way, there have actually been research programs based on writing down the alignment optimization target. The best-known are (C)IRL and other assistance games. These ideas seem to work fine for agents with an ontology where the intended human is robustly viewed as principal and... is perfectly rational. Or, when we can precisely specify all departures from rationality. This sounds doomed. Another such approach is computational superimitation (COSI), which seems to make a totally different set of assumptions (which very few people understand well enough to question). I hope that Vanessa Kosoy and Diffractor do not unilaterally decide that they have properly specified alignment, and then actually try to build an ASI based on COSI.

The sweet lesson. These days, almost anyone in AI/ML would respond "Sure, it's also hard to mathematically specify a cat, but we have no problems learning a cat detector!"

That's right. The target is complex enough that we can't hardcode it, but this does not mean it's too hard to learn. The problem is that, as far as I can tell, it has to be learned very robustly and early.

Robust ML. We know how to build powerful ML systems that perform incredible feats. However, they are typically not robust to either distributional shifts or adversarial attacks. If you train a cat detector on images taken indoors on an iPhone and then try to deploy it onboard an aerial drone, it probably won't work (very well or at all). Give an ML engineer access to your model and they will quickly find (what looks to you like) an ordinary picture of a cat indoors that is also misidentified. Learning can be quite fragile. I am guessing that this is partly because we are bad at ML, but also partly because there is no free lunch - often there are just many answers reasonably consistent with the training data, and our intended answer is not quite as divinely ordained as we expected.

Learning Interactively. Unfortunately, alignment (to a specific human principal) is inherently much harder to learn than "does this picture look like it contains a cat?" Whether a picture contains a cat is (what Yudkowsky might call) a "shallow function of sense data." The full preferences of a human are not. Humans are highly general agents who want (superficially) very different things in different circumstances. We have very rich preferences which are not exposed from any snaptshot. Worse, alignment requires learning a full behavioral policy rather than a classifier, which (drastically understating the problem as a first pass) has far more degrees of freedom. The problem of learning policies is usually called reinforcement learning. It's generally quite hard to learn great policies passively, since there is a combinatorially large space of possible situations to learn about. A competent agent needs to selectively explore various actions and observe the results. In other words, competence seems to be a prerequisite for learning about human preferences. Worse, it may be a prerequisite for learning to care about human preferences at all (prior to discovering the content of those preferences), which we might expect if the aligment-to-human-X relation turns out to be algorithmically complex! If competence and caring are learned in the wrong order, then we are in danger.

Competent and incompetent at once? That competence is a (strict) prerequisite for alignment answers the naive objection that if the machine is smart enough to fight our corrections, it would also be smart enough to know what we want.

The hard core. The normal solution to the fragility of learning is to iterate. Deploy the system, see what goes wrong, and try to patch it. This loop leaves much of the ultimately responsibility with human agency; because we're the biggest agents around, we can clean up any damage caused by ML failures, figure out what went wrong, repair our training setup, and run it again. By building a superintelligent machine, we break this loop: it increasingly decides how to recover from (our perceived) failures. This puts us in a regime even worse than prosaic distributional shift or adversarial attack (though these problems also arise in monstrous proportions). We have to train an RL agent to do a task that we can't do ourselves, which allows unrecoverable failures according to our lights. The purpose of building intelligent machines is to set them to find unexpected solutions to our problems, but the most unexpected solutions might kill us, and then we can't rephrase the problem. Therefore, competence naturally interferes with correction. Briefly: Competent agents tend to interfere with training corrections which catastrophically amplifies the fragilities and (alignment) misspecifications of their training processes.

GRL is unforgiving. The hard core is hard in large part because general reinforcement learning is cursed. The theory is rife with negative results. For example, exploring enough to guarantee optimality in the limit requires catastrophic mistakes in some domains. Humans can survive in this world anyway because we have a great inductive bias for our survival, which usually allows us to avoid traps (for example, pain and disgust protect us from sharp things, infections, and poisons). Superintelligent RL agents will be heavily selected (before deployment) to have a great inductive bias for their survival and usefulness while subjugated, which is not necessarily safe for us. ASI will not view damaging our corrective mechanisms as falling into a trap, but from our perspective, it is a trap that is very hard to avoid.

Providing better feedback

To show that intractability of general RL is the hard core of the alignment problem, I would need to demonstrate that it affects all (known) paths to a solution. I think that the most natural objection is that though I have described the problem of learning a policy as RL, there is no need to use "standard RL techniques" such as a reward signal (so this may be a misnomer). More radically, we might even forgo learning a policy at all, restricting ourselves to train for prediction to some degree (as in Yoshua Bengio's Scientist AI). Indeed, I suspect that predictive systems are less likely to seriously interfere with their training process - but they're also much less useful! A halfway point is to attempt to train "transparent agents": agents with minds that we can read, allowing us to understand their plans and intervene if necessary. This is the goal of (dev/mech) interpretability and (arguably) natural abstractions, condensation, and (Demski's) "understanding trust." In other words, it is the orthodoxy of BOTH prosaic alignment and modern agent foundations! We might call this highly general family of ideas "legibility."

Using legibility. A (non-obvious) reformulation of this objection is that we might hope to give much richer feedback to our machines than a scalar reward signal. Reading an agent's thoughts sounds more like allowing it to communicate with us. But really, that is not quite the intention. An unaligned ASI would likely choose to communicate with us in highly dangerous and manipulative ways. There are some approaches that benefit from machine -> human communication, but they have the flavor of corrigibility/conservatism, and intended message is usually (a binary signal indicating) deference to a human mentor or a request for feedback. Legibility-focused approaches typically make the mind-reading non-consensual, so that "communication" does not really describe the machine -> human messages. However, the intent is that we somehow use this legible information to craft human -> machine "messages" that communicate our preferences in a richer form than rewards. For example, consider activation steering - this is something like a reward, but it is "targeted" to write at the access level of the machine's thoughts, rather than its (lower-level) "neurons." In a way, this crudely resembles John Wentworth's plan to "retarget the search" by identifying the concept of human values inside of the machine, and then replacing its optimization target with that concept. These are sophisticated write operations, to the mind of the machine; they conceptually take the role of reward, but (hopefully) function very differently.

Therefore, I think it is worth reiterating: we are confused about communication, more specifically than agency.

I am concerned that legibility-based approaches do not avoid the hard core of the problem. Small failures of legibility seem likely to compound, because a competent misaligned system is incentivized to be illegible. Even if we avoid using the most forbidden technique (training against interp signals), competent agents with situational awareness can still autonomously realize that illegibility benefits them. The (ELK-style) hope is that we can notice this while the systems are still legible. But what then? Roll back training and hope that the next run doesn't land on the idea? This is just a softer version of the most forbidden technique.

Reward is all we get? I am highly interested in fusions of legibility-style and corrigibility-style frames, particularly if they can better illuminate the hard core of the problem (and particularly its boundaries). For example, is it possible to talk about richer forms of communication in the general RL setting? In the worst case, it may be that standard RL already describes the limits of what is achievable - everything we can cook up is qualitatively similar to a (richer?) reward signal. That would be very useful to know.

Inner alignment. Modern ML methods may not be inner aligned to reward, but nothing I have written here depends on that assumption (it is only an additional difficulty of the learning problem, this time from a deficiency of our RL algorithms). It would be terribly convenient if this made the problem easier rather than harder, but people seem to believe this anyway (Alex Turner suggested a salient example of this frame, but I am not sure if he is unusually optimistic about the implications). For very short horizon tasks, models probably do just use context-dependent heuristics, but this is safe only because it is not very powerful. With increasingly long-horizon training on diverse tasks, (AIXI-style) exploration and planning towards reward maximization is heavily incentivized. I am not aware of any methods for building ASI that do not rely on this type of training (at the stage when competent agency is acquired), which is why reward hacking is a strong central example of catastrophic training interference. However, should outer alignment to reward fail and the models learn something stranger than but behaviorally competitive with reward maximization, this does not seem like much reason for hope.

Behaviorist v.s. process feedback

If we are restricted to rewarding behavior then there seems to be no way to avoid a reward hacking incentive (advanced agents intervene in the provision of reward). Perhaps the hard core I proposed understates the problem, by making it appear transient. The "spurious" solution of disempowering humanity and seizing control of the reward button seems to remain valid indefinitely, no matter how longer we are able to maintain control of the training protocol, as long as reward is the only tool we have for shaping model behavior. Another way of looking at this is that the RL framing entirely fails to engage with the alignment problem (!?) by assuming perfect alignment to the wrong thing: the reward-yielding process. Superficial patches like CIRL assume perfect alignment to a fictional ideal agent, which is still (even more) wrong. So perhaps we should focus on moving away from the RL paradigm (somehow) in search of anything that is not doomed?

Safety from behavioral rewards. I don't think that safe RL is totally hopeless. There are various ways to specify safer "limited" RL agents, usually based on some sort of pessimism/conservatism about unexplored actions and situations. However, it does seem that more should be possible. Humans grow up to be nice somehow, and not by remaining deferential for our entire lives. If (AIXI-style) RL can't reliably converge on this solution (in favor of reward hacking) then maybe we need a higher level of access.

Safety from process feedback. Modern alignment methods already use a higher level of access. They are based on RL, but not (only) on external behavior. It's common to use process reward models, which incentivize models to think thoughts which appear good (beyond producing good outcomes). At a first pass, this just seems even more doomed; it's a complicated way of passing the buck. Is there any reason to think that nice "habits of thought" picked up from training will persist under reasoning training (or "deep reflection")? The validity of this technique is very hard to study theoretically, but this is not a strong positive reason to believe that it will fail (and I even think that its immediate dismissal by agent foundations researchers including myself may have been too hasty). Assuming that long-horizon agency arises at the level of the reasoning model (and not at the level of token-generation), non-myopic reward-hacking incentives should not arise from process reward models (a bit like MONA). However, this almost certainly will not hold up, because it sounds intractable to train a powerful reasoning model by rewarding individual virtuous and effective reasoning steps. Combining process-based and performance-based rewards should tend to incentivize virtuous-appearing thoughts that hack the performance-based reward.

Concluding thoughts. This post is not as technical as I would have liked. An ideal version of this post would have more math, perhaps distilling the hard core to a specific equation delineating the tradeoff between competence and safety (under certain legibility assumptions). That will have to wait for future research! Indeed, making the hard core of alignment problem very formal may be nearly as hard as "solving it" to whatever degree is formally possible.