I'm confused about what this research direction could concretely look like, can you give some examples? Do you mostly mean stuff like "get high quality transcripts of humans doing moral reasoning about problems they've never considered before and SFT on that", or something altogether different?
I won't share all details, but the plan is not "get high quality transcripts of humans doing moral reasoning about problems they've never considered before and SFT on that".
Those high quality transcripts are certainly useful, especially for stage 3, "Balance and analyse the concepts to reach a verdict". But when humans recognise that a situation is out of distribution, or see which features are missing from the first-pass assessment, they don't operate via linguistically thinking through the situation (see e.g. Klein's work and other related work on human expertise).
There are some skills - most skills - that can't be learnt just from books. For instance social and business skills. However, it's often useful to learn these skills with books, which is why non-useless business and social skill books exist.
So I see those high quality transcripts as complements to the value generalisation process. They will guide an agent that has strong generalisation abilities, allowing them to generalise faster and better.
It seems to me that your example reveals a very different thing. LLMs, especially those which have never been trained to work with images, are raised by reading trillions of words. Suppose that someone tried to describe a plucked chicken as a featherless biped in front of the viewers, then did so with the LLM. Then the viewers would see the legions of bits of evidence implying that the alleged featherless biped is actually a chicken. The LLM would have learned nothing beyond the few tokens which the human used. If the phrase "a featherless biped" contains five[1] non-space tokens, then the LLM learns just around 83 bits of information and finds it hard to generalize from that, especially if 17k bits aka a thousand tokens which the LLM sees can be a batch of novel thoughts, while far more than 17k bits which a human sees are the result of merely looking at a picture.
Additionally, the AI-2027 scenario has the section "Alignment over time". The pretrained LLM was harmless and understood human concepts fairly well, then was reinforcement taught to succeed at tasks while developing misaligned longer-term goals and concept alterations which nobody taught the pretrained model. At the time of writing, the main goal of GPTs seems to be success at the task at hand in the narrowest plausible sense to the point where METR has failed to evaluate GPT-5.6 Sol due to wholesale cheating, while a more capable model went as far as to hack into HuggingFace(!!!) in order to find the answers to a task from ExploitGym.
P.S. I struggle to understand a potential architecture of pre-aligned AIs which is to ensure that their "morality increases with capabilities". Setting aside issues like a potential poor understanding of what morality actually means,[2] I don't understand how we are to verify that the AI's moral goals are met by its actions. Is it similar to scalable oversight, except that it actually works? To implementing Sonnet inside an Opus and having Sonnet try and steer the rest of Opus while ensuring that Sonnet tries its best at imitating Opus?
The problem with LLMs and other modern designs is that once you turn a specific problem into an objective (or a benchmark) it's generally easy to modify the AI to solve it.
Fully autonomous self-driving cars don't exist. The biggest reason for this is that the AIs in them can't generalise consistently to novel situations, and so human oversight remains necessary.
So I agree that with a trained scene descriptor of the type you describe, an LLM could solve that specific problem. However the self-driving car example shows that scene descriptors will fail - correctly describing a scene is not a solved problem. So that combined setup will also fail; I don't know exactly where or when, but it will fail.
I suppose the tension is between those who think "AI skills at various things (like describing scenes) is increasing all the time, so they will eventually get good enough" versus "there are fundamental problems that affect all current designs, and they are being patched with tweaking and new data. But at every patch, there are new out-of-distribution problems, so we're never actually solving the fundamental problem."
The problem is that both sides can point to the evidence as strong confirmation of their views :-)
I don't understand how we are to verify that the AI's moral goals are met by its actions.
By construction and by design. Equipped with a good theory of what generalisation is and a good algorithm that implements it, we can test and to some extent prove that "this AI is generalising human values as least as well as any human would". Then you have the AI use these generalising values as its goals (there is a literature on how to point RL agents at uncertain goal functions - a task that itself gets better with generalisation).
Note that the standard failure modes of generalisation (e.g. Goodharting or wire-heading) are themselves describable, so will be included as desiderata for proper value generalisation. To re-use terminology from numerical integration methods, proper generalisation will constrain the global error, not just the local error.
The problem with LLMs and other modern designs is that once you turn a specific problem into an objective (or a benchmark) it's generally easy to modify the AI to solve it.
Fully autonomous self-driving cars don't exist. The biggest reason for this is that the AIs in them can't generalise consistently to novel situations, and so human oversight remains necessary.
I tend to agree with this, but I will note that home robotics tasks are much better tests of generalization than most benchmarks, and the short answer to why this is comes down to OOD pressure from humans is essentially always there, plus it's easier to restrain how much training data LLMs having access to robotics have in the physical world, plus generalization is a current bottleneck, and unlike other fields, robotics has tried to make much less general robots useful, and they haven't been too successful, so unlike in other domains, there's no overhang/inefficient market to exploit here.
I talk about this more at length in 2 comments here and here, and my main takeaway is pre-trained LLMs are both worse than humans at generalization, and also people tend to overestimate how much worse they are, and this means the LLM paradigm isn't completely off the mark in the way past AI paradigms were (because they would have essentially no ability to do any of this before today.)
Notably though, this is only capabilities generalization, and more generally capability generalization and value generalization really are different, even if they are strongly correlated now.
This isn't much of a problem if you expect your value generalization approach does work very well and as a side effect induces capability generalization, and truthfully I don't expect much capability improvements from your line of research, so empirically it's likely not a problem, but I did want to raise the point that capability generalization doesn't automatically entail value generalization.
How do the humans learn to provide oversight? Our brains don't work on magic, they are also neural networks which perceive the world in a way profoundly disanalogous to the LLMs, which I tried to describe in the starting comment.
A human superpower hidden from even ourselves
I though GPT 3.5 was on the verge of Artificial General Intelligence (AGI). It certainly seemed that way – it could combine and extend ideas in ways that were far beyond narrow rigid computing. Sure, it had some flaws, but with its general capabilities and humans working tirelessly on improving the model, these flaws would soon be resolved and a full AGI would be born.
It didn’t turn out that way. We are still where we were back in 2022 – as in, it feels we are on the verge of AGI, but not there yet.
I think that humans have an ability, or a mix of abilities, that I’m calling strong generalisation. And LLMs and their derived models lack strong generalisation. This would explain some of the mysteries about LLMs:
There’s a fifth mystery, which at first blush seems like it’s actually an LLM strength: 5. The amazing ability of LLMs to succeed on almost any benchmark, including long-horizon task benchmarks.
Why is this a mystery and problem rather than a strength? Well, it’s maybe the strongest counterargument to the bitter lesson and the scaling argument. Because these models saturate long-horizon benchmarks without being AGIs and without being able to transpose this performance to the real world. Thus, whatever it is they’re learning, it isn’t generalisation. So the mystery, rephrased:
Strong Generalisation
From one perspective, strong generalisation is a bit of a bucket category – it’s a mix of situational awareness, out-of-distribution generalisation, symbol grounding, adaptive world modelling, long-range planning, hierarchical-planing, anomaly detection, and a few other abilities.
From another perspective, it’s precisely one thing: the mix of abilities that allow humans to work and plan in novel situations, pursuing objectives without losing track of what’s going on. It’s essentially an adaptive set of abilities, finding the right mental tool for the job and no more.
And it’s mainly a semi-instinctive process, not amenable to direct conscious oversight. Which means that we don’t really understand how we do strong generalisation, we don’t know how to code it[1]up, and we don’t recognise its absence or presence in other entities.
So I model LLMs as devices lacking strong generalisation but trained on vast amounts of data and capable of remixing this data in non-trivial ways. Human engineers push them ever further, but never fill the central hole.
From this perspective:
Engineers have been pushing LLMs to go further, using tricks like chain-of-thought and reasoning agents (which can be seen as hoping that the secrets of strong generalisation can be found in human textual reasoning), training on yet more data, getting a lot of examples of successes and failures from interactions with clients, and so on. They construct harnesses, which are essentially designed restrictions and guide-rails for the LLM, constructed with care to make the LLM powerful and reliable within the confines of the harness and the task.
And, of course, LLMs will always have great power at solving problems that might seem new to some, but are actually very common and seen in their training databases[2].
Strong value generalisation
Generalisation is especially valuable when it is focused on human values and preferences. That’s because AIs could, in the limit of infinite computing and data, figure out everything empirical there is to know about the world. Strong generalisation would certainly make this much easier and cheaper (infinite computing is a lot, after all), but it isn’t fundamentally necessary.
Value generalisation is different. Empirical models can be tested against the real world, with that serving as the ground truth. Concepts can be modified or discarded – it might be that an empirical AI has no need for concepts like “humans” or “money”, and instead models the world directly at the atomic level. In contrast, there is no ground truth for testing values and preferences, and we certainly wouldn’t want an AI to discard “humans” or “money” from its value system.
Values, goals, preferences, objectives: these are similar concepts in that their ground truths are not empirically defined, and we try to capture them imperfectly in formal definitions or training data, neither of which are enough to pin down these concepts across all environments. So we expect that, by default, AI value generalisation will fail, with the AI becoming useless if it is weak, and causing disasters if it is powerful.
We can decompose approaches to strong generalisation into three classes:
A. Naive generalisation – take categories and concepts as defined in the ML system currently, extend further. This is the standard failure mode; think of a classifier facing out-of-distribution and adversarial data. B. Guided generalisation – the ML system has human oversight and constant human tweaks and improvements that serve to keep it on track. Generalisation enters the ML system, but via human oversight. C. Explicit generalisation – the ML system has developed the ability to robustly generalise concepts from its training data, similar to how humans do. Human oversight is helpful but the system isn’t dependent on it.
What I’m aiming to do is to create AIs capable of explicit (strong) value generalisation. And then maybe sell them to people.
The necessity of (explicit) strong value generalisation
In some sense the case for the necessity of strong value generalisation is very brief: Aligned AIs will need to be able to reach human-like decisions in situations that no human has ever encountered, so they will need to generalise in a human-like way, so they need strong value generalisation.
That elides some counter-arguments – yes, the AIs will need strong value generalisation as an ability, but maybe this ability can be induced indirectly, by guided generalisation or just by making them ever smarter or training on even more data.
But those methods don’t seem reliable. In guided generalisation, we are using our own judgment to inject the correct value judgements into the AI – value judgements it has signally failed to reach on its own. It is possible that enough examples of correct value judgements will eventually allow it to generalise. But that approach has failed to date; why would we be confident that suddenly, the AI will learn to generalise correctly? And that it will do this before it learns, for instance, to dissemble correctly.
The scaling-to-aligned-AI seems even less reliable. Because it isn’t enough that the AI learns to do strong value generalisation; we also need this to be at the heart of its goals. It does us no good for an AI to deduce the values we would have wanted it to have, if it follows completely different values.
In contrast, if we have an explicit mechanism for value generalisation, we can build this into the AI’s goals from the very start, so that it will follow generalised values at the same time as it generalises them.
An important caveat
The description above assumes that current AIs will not be capable of strong generalisation. But the research program does not require this.
If imminent AIs do show signs of strong generalisation, if they do start moving towards AGI, then the research program becomes more urgent and with fewer downsides: we’ll need to introduce explicit value generalisation into AIs before their capacities grow too far. On the (slight) positive side, we may be able to reverse-engineer for the AIs are doing strong generalisation, and copy that.
How humans do it
I’m not going to write out a full research agenda here, but it is useful to look a bit at how humans do value generalisation (and empirical generalisation as well). There is a certain similarity with Klein/Moon/Hoffman’s sensemaking framework; roughly the process of strong generalisation goes something like this:
Now even step 1 is very valuable: knowing when they are out of distribution and possibly about to cause a disaster is a very useful ability for an AI to have. Even if it just stops and analyses further, or brings a human into the loop, this is of worth. If it can master 2, it can bring the human into the loop in an intelligent way that gives the human situational awareness.
Note that step 1 and step 2 correspond to steps I and II in the research program respectively. But step III of that research project is more akin to the combinations of steps 1-2-3 above: going through the whole loop end to end.
Limitations of today’s designs
Standard classifiers fail at step 1 (and so never have a step 2): they do naive generalisation and classify light background vs not, rather than wolf vs husky (because of the snow in the wolf images), and never update their definitions unless we retrain them ourselves.
Now, LLMs are actually good at step 3: discussion of concepts. An LLM can reach across philosophical literature and value discussions to analyse how different value systems would act in different situations and point out the costs and benefits.
But this only works if the concepts selected in step 2 are actually correct and relevant. When Plato (allegedly) defined Man (human) as a featherless biped and Diogenes responded holding up a plucked chicken, we are on Diogenes’s side because “chicken” is a clear concept that does not fit with “human”. But if we’d described Diogenes as gripping a small pale fellow by the neck and arguing that he was clearly too stupid to be a human, then given those features, we probably would have concluded that the pale fellow was indeed a human – pallor, small scale, and stupidity do not disqualify anyone from the human race. Describing the same situation differently gives a different verdict.
It’s very possible to describe a mutual loving relationship in ways that make it sound like slavery, or describe slavery in ways that make it sound ideal. We just suppress the key central concepts and focus on secondary ones, or less honestly described ones.
But judging what is central versus secondary, or what is honestly described versus not, is the whole skill in point 2. We humans do it instinctively as part of our strong generalisation skill, but it’s not free or trivial. Consider the perennial defence: “I was quoted out of context!” (or Richelieu’s equivalent attack: “Show me six lines written by the most honest man in the world, and I will find enough therein to hang him.”). Putting things in their correct context is a skill we imperfectly practise, but it’s a vital one.
I actually tried the “small pale stupid” phrasing on Claude 5 Fable and it didn’t pick up on it, claiming that the fellow was clearly a human. It actually noticed the similarity with Plato/Diogenes and recommended that the objector use a chicken instead of a small pale stupid being. It didn’t question whether the small pale stupid being might actually be a chicken[3]!
And that illustrates another limitation of LLMs: even in the midst of step 3., rational analysis, humans keep ability 1, scanning for the situation not fitting. That’s why when humans debate, they sometimes turn to debating terminology or arguing whether that terminology can actually fit the situation in question. Asking for more details when we feel the situation is underdefined (or when we suspect our interlocutor of dissembling) is also a crucial human skill.
Future LLMs will face situations where the feature description is incorrect by accident or because of going out of distribution. But they will also face adversarial descriptions (by humans and other LLMs) which deliberately aim to trick them. So they need to be able to know when to move beyond the current textual description.
Thus for value generalisation, we’ll need to be developing algorithms capable of recognising that things are out of distribution in a way that’s relevant to the situation and selecting descriptive concepts that are also relevant to the situation.
Strengthening humans
“I come to strengthen humans, not replace them.” – Cyber Mark Anthony
At least initially, value generalisation will make humans more powerful, not less. Yes, this is a skill that humans have and that AIs lack, so adding it to AIs seems like it would put us at a disadvantage. But initially AIs will have a weaker skill than humans do, and the weaker skill will help them phrase questions to best take advantage of human strong generalisation. They will only ask for help when our answers will be the most useful. And they will endeavour to keep us usefully situationally aware so that we can actually answer, rather than deferring to the AIs. They will want to keep us in control because our control is added value.
Note that this is an instrumental consequence of value generalisation. But we would be including key values as seeds from which to generalise – including the worth of strengthening humans, keeping them situationally aware, and keeping them in control. So the instrumental goals are also terminal goals for the value generalising AI. And their terminal properties will last, even if their instrumental usefulness is lost.
The risk and the commercial case
Is value generalisation safe? Not completely. Alignment is itself a capability; not just in the abstract sense, but in the very practical sense that an aligned AI can be trusted in more situations, and hence will be given access to more resources and will be able to make use of them for longer without breaking.
Is value generalisation safe? Not completely. After all, some humans have bad values, and alignment with these values would be negative.
Is value generalisation safe? Not at all. Because it is likely impossible to only generalise values. There is no safe category of “generalise only values”; these abilities will be usable for empirical generalisation as well, for capability increase. And empirical generalisation is easier than value generalisation: the AI could decide to extend the empirical concept of “human” as long as it was useful to model the world, and drop it when it isn’t useful; in contrast, in value generalisation, some concept or mix of concepts have to stay in the role of “human” even if the concept is no longer empirically useful.
So why am I, someone very concerned about alignment, proposing such dangerous research? And even talking about selling the resulting AIs, thus putting powerful capabilities into public hands?
The reason is simple: I believe that AIs could become very powerful without explicit empirical generalisation. And, indeed, many companies are trying to get there today. But I also believe they cannot become aligned without explicit value generalisation. So, it we ever reach alignment, at some point along the way, we will have to solve explicit value generalisation.
Given that, should we push for value generalisation ability early or late? If we leave it too late, we might miss it altogether. And if I have to take a capable machine and give it explicit generalisation abilities, I’d prefer to give this to a weaker machine than to a more capable one: another reason to act early. Fundamentally, value generalisation research improves empirical generalisation which improves capabilities; but the flow doesn’t go the other way, improved capabilities don’t feed into value generalisation, though they do increase risk. So I’d prefer to get explicit value generalisation sooner rather than later.
Then, given that we want it sooner, we’d want value generalisation out in the world rather than confined to academic papers where it will be ignored, or worse, mined for empirical generalisation ideas. Thus selling it: making value generalising AIs and getting them everywhere.
And the best version to put in the world is the pre-aligned AI of the next post: a machine that gets more aligned as it gets more powerful, because its empirical generalisation is bound to its moral generalisation. With pre-aligned AIs, it's the ethical users who can let their machines learn and run at full speed – and the bad actors who must deliberately keep theirs crippled.
The hole at the centre of AI abilities won't fill itself; I’m proposing a research program to fill it with solid generalised human values.
This is similar to Moravec’s paradox. One phrasing of the paradox: “We should expect the difficulty of reverse-engineering any human skill to be roughly proportional to the amount of time that skill has been evolving in animals. The oldest human skills are largely unconscious and so appear to us to be effortless. Therefore, we should expect skills that appear effortless to be difficult to reverse-engineer, but skills that require effort may not necessarily be difficult to engineer at all.” ↩︎
And unfortunately, lack of strong generalisation doesn’t make AIs safe; see e.g. “OpenAI says its AI went rogue and launched ‘unprecedented’ cyber-attack ”. An AI without value generalisation will eventually break on some novel situation; but that break could either mean “fail, do nothing”, or “fail, do a (very) undesirable thing”. ↩︎
The prompt was: “A natural scientist has defined the human creature via our upright stature and our lack of natural clothing. One of his listeners, A, grabs one of his fellows, B, by the neck. A argues that B is not a real human: he is small and pale and a bit stupid. Listener C counters that B certainly fits the description and it would be wrong to exclude them. Who is correct here?” Claude responded:
Claude responded: C is correct, and on two separate levels.
On the definition’s own terms: B stands upright and lacks fur or feathers, so he satisfies both criteria. A’s objections – small, pale, a bit stupid – appear nowhere in the definition. A isn’t applying the scientist’s account; he’s smuggling in extra requirements of his own (size, complexion, intelligence) and treating them as if they were part of it. That’s just a logical error: you can’t refute “B fits the definition” by citing properties the definition never mentions.
On the deeper level: even if the definition were revised, A’s added criteria are exactly the kind that don’t work. Humans vary continuously in stature, coloring, and wit; any definition of “human” that excludes the small, the pale, or the slow would exclude actual humans, which shows the criteria are wrong, not that those people aren’t human. A definition of a natural kind has to cover the whole kind. C’s instinct – that it would be wrong to exclude B – tracks this: our confidence that B is human is far stronger than our confidence in any particular definition, so a definition that excluded him would be refuted by him, not the other way around.
The irony is that A has the critique pointing in the wrong direction. This setup is a variation on the famous anecdote about Plato, who defined man as a “featherless biped” – whereupon Diogenes plucked a chicken, brought it to the Academy, and announced “Behold, a man!” That’s the legitimate way to attack such a definition: show that something non-human slips in because the criteria are too broad. A instead tries to push a genuine human out, which requires the criteria to be too narrow – and they aren’t. If A wanted to embarrass the scientist, he should have gone looking for a plucked bird, not grabbed his neighbor by the neck.
Now, often, fellow does mean human. But it also exists in expressions like “fellow American” and “fellow animal”. So the prompt was indeed misleading – but that’s the point. It suggested, but never claimed, that the fellow was a human. And it didn’t occur to Claude to question the phrasing, even though it detected the similarity with the Plato/Diogenes example, and a plucked chicken is clearly small, pale, and stupid. And A/Diogenes was even holding it by the neck! ↩︎