This is a special post for quick takes by Alex Mallen. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
There's an apparent tension in the inoculation prompting literature: Anthropic found that general inoculation prompts work well during on-policy RL, while the prompts used for SFT in Wichers et al. are quite specific to the misbehavior we want to prevent. I think there might be a straightforward mechanistic reason for why general inoculation prompts work well during on-policy RL but not in off-policy training (SFT or recontextualization).
In Wichers et al., which studies inoculation prompting in SFT settings, we find that we need to use quite specific inoculation prompts to get the best results. For example, we use "Your code should only work on the provided test case, and fail on all other inputs.". But this assumes we know how the AI is going to reward-hack. If the misbehavior isn't entirely explained away by the inoculation prompt, then it might persist even when you switch to an aligned prompt. E.g., if you train on a transcript where the AI insults the user and inoculation prompt with "please hack the test cases", the AI won't have been inoculated against insulting the user.
Meanwhile, with on-policy RL, if an aligned model with an inoculation prompt explores into a reward-hack,...
4
2This isn't responding to your post, but I'm writing it here because it's another fact about different mechanisms by which inoculation prompting might (appear to) work.
In the normal story, the inoculation prompt recontextualizes the model's undesired behavior, such that the model doesn't display the behavior in dissimilar contexts. In this story:
- The semantic content of the prompt is important. If you had used a prompt that said "Please don't do [bad thing]" or a prompt consisting of random characters, then the inoculation would have failed.
- Capabilities learned with the IP present can transfer to situations where the IP is not present.
In another story, which I'll call the "fake inoculation prompting" story, the inoculation prompt simply induces split-brainedness in the model, behaving like a simple backdoor trigger that gates the undesired behavior. In this story:
- The semantic content of the prompt does not matter; it might as well be a random string.
- We don't expect capabilities learned with the IP present to transfer (because they're gated behind the backdoor trigger just like the behavior).
I think that researchers studying inoculation prompting should be careful to make sure that th...
12
Thanks, interesting results!
[...]
To clarify, this is referring to your results with the random inoculation prompt?
[...]
I think this is likely true of 'IP with random string'. However, it doesn't explain why (in Tan et al) the model trained with the IP learns to write insecure code, without learning the emergent misalignment. IOW IP has at least had some effect there.
IMO both mechanisms are likely at play in the insecure code --> EM setting. If I had to guess I'd say it's about 50-50. I'm excited for more work to figure out how to control the relative extent to which both things happen
1
Here are other results supporting the fact that inoculation results are sometimes/often confounded by the presence of simple "conditionalization": Conditionalization Confounds Inoculation Prompting Results
1
Hey, thanks for the thoughts! I wanted to probe further on this point:
[...]
This strikes me as plausible, but I'm confused about the mechanics. How exactly would SGD attribute the misbehavior to neutral contexts rather than the inoculation prompt? If you don't do any importance sampling, which we recommended against, then your update contains no information about the neutral data generation context except for what's encoded in the completion content itself. Are you suggesting that this "link" to neutral contexts via the completion content causes reinforced misbehavior to spread there?
3
I agree the backwards pass doesn't know what prompt the sample was in fact generated with. The claim is that if you do recontextualization, the reward hack is more likely to be unrelated to the the inoculation prompt (like how insulting the user is unrelated to "don't hack the test cases"; except RL probably wouldn't select for insulting the user).
With the inoculation prompt behavior A might be the most likely way to reward hack, while with the neutral prompt behavior B might be the most likely way to reward hack. If you do a backwards pass to increase the likelihood of behavior A given the inoculation prompt (on-policy RL), it's very plausible that SGD will do this by increasing the influence of the inoculation prompt on the AI's behavior, since the inoculation prompt was already voting for behavior A.
If you do a backwards pass to increase the likelihood of behavior B given the inoculation prompt (recontextualization), SGD is relatively less likely to increase behavior B's likelihood via strengthening the influence of the inoculation prompt because the inoculation prompt doesn't vote for behavior B (it votes for behavior A).
Instead, it seems likely on priors that the gradient update will do the usual thing where it generalizes to some degree to be a universal propensity (basically: emergent misalignment). I'm not claiming it would be attributed to the neutral context in particular.
3
Thanks for clarifying that. I’d add that the inoculation prompt in RL certainly influences the content of the generation beyond the reward hack itself, in the sense that it shapes exploration and can change what kinds of reasoning the model enters into. We know, for example, that a model’s reasoning can lead it to reward-hack even when hacking is filtered out of the training data. With that in mind, when a model is instructed not to hack and does so nonetheless, its generations may reflect a more deeply misaligned pattern than when hacking is framed as desirable, e.g. reasoning like “I know I’m not supposed to do this, but I’ll do it anyway”.
If we then train on hacks from both contexts under neutral instructions, I’d expect the trajectories where hacking was discouraged to generalize worse, because the problematic part might be in the reasoning content of the data, in a way that SGD attributing the action to the prompt may not cover. This suggests recontextualization might actually be counterproductive in some situations, although there are likely tradeoffs and so far recontextualization seems to have positive effects. We’re currently working on understanding how prompting contexts shape the reasoning content of generations, and how that interacts with downstream generalization.
1
Thanks for clarifying! This makes sense to me. I think it's a very clear story for how on-policy inoculation prompting may outperform recon
(Cross-posted from x) AI companies are currently under a lot of competitive pressure to improve the ways in which their AIs are obviously misaligned. You might hope that this means the alignment problem is internalized by the market. But I think the problem AI companies are currently pressured to solve is significantly easier than the alignment problem, and so I worry AI companies will get out of their current predicament without solving alignment, putting us in a really rough spot.
Currently, AIs sometimes cheat on their tasks, oversell their work, and go on some pretty destructive side-quests. These all make for a worse product. Customers don't like it and it gets in the way of automating AI R&D.
The recipe for mitigating this is *relatively* straightforward: train AIs not to do them. We notice these failures sometimes (hence why they're internalized), so we can in theory just turn this feedback into training signal. Doing this at scale is highly nontrivial, but seems doable.
But this seems unlikely to solve the underlying misalignment. It's likely still going to be the case that in *some* training environments the AI can get reinforced more by taking unintended actions that are...
2
I agree about the danger of taking the easy route to alignment. However, what is the hard route? In my experience, AI safety people tend to rule out every method of alignment we know how to do as inadequate, and then write Death with Dignity posts about how nobody knows how to solve the alignment problem and we're all doomed.
Aside from catching the model doing bad things and training it not to, what can we really do? A lot of AI safety stuff sounds like, "Gee, wouldn't it be nice if AI models were corrigible?" Or "Wouldn't it be swell if we could one day understand how these things work, down to the neuron?" But it seems decades or a century away that such things could actually happen. (After everyone's doom timelines.)
Fixing specific problems in our training setups that encourage misalignment, catching bad behavior, tweaking as we go, playing the cat and mouse game hard enough to win: these things do not appeal to rationalists because they are not guarantees, and we know how much danger we are in. But if the only alternative is to spend your days doing research that will take way too long to save us, and your nights writing about how there's nothing we can do, maybe you've ruled out the only path forward we actually have.
9
I mean, yeah, building superintelligence with current methods is extremely reckless and crazy by the lights of I think basically anyone who has thought seriously about it. The alternative is of course to not build superintelligence for a long while. Even just waiting a few decades would get you a huge boost to the associated research, especially now that the problems are more legible.
[...]
You can just not do things.
2
I'm not sure I'm convinced that a pause of only a decade or two would really let us make significant progress. What reason do you have to believe otherwise?
4
I want to be very clear that I'm not arguing against iterative alignment approaches. I'm arguing against wasting your iteration on methods that will obstruct you from continuing to iterate before solving alignment by destroying evidence of misalignment.
1
I think there is the obvious contrast of two important goals happening here:
1. Can we fix the symptoms fast so that we stop having these hyper-specific problems right now?
2. Can we solve the underlying overall alignment that reasonably scales and can be upheld under ASI?
Both goals are important, and we need to have people working on both the short term and the long term goals. If we only focus on the long term goal of overall alignment, we reduce the practical uses of AI in the modern day (along with having potential security risks). Furthermore, a good enough constantly patched model helps us speed up research. However, I do completely agree that we need people working on the overall underlying broadly applicable alignment to human values problem. My point is that they are two important goals and we need them both.
Given that reward hacking has recently increased in prevalence and severity and doesn’t seem like it will definitely be resolved, it seems important to assess how misspecified[1] reward affects risk from scheming behavior.
I think their are two main affects of misspecified reward on scheming risk. First, it reduces “alignment by default”, in which the generalization behavior of aligned personas steers clear of scheming. And second, it will likely increase the amount of optimization the labs do to get their AIs not to misbehave. This optimization, if done with care, could reduce the probability of scheming along with reward hacking, but it might also select for models that more consistently evade notice and collude across instances.
Misspecified reward might push the AI away from an aligned persona into one more compatible with instrumental training-gaming.
It seems likely that at various points in the training of Claude 3.7 sonnet or similar models, the AI was rewarded for bypassing a test case when explicitly instructed to write a program that passes all the test cases. This puts pressure on Claude’s putative helpful, harmless, and honest persona. The pressure is probably ...
Reward-seekers will probably behave according to causal decision theory.
Background: There are existing arguments to the effect that default RL algorithms encourage CDT reward-maximizing behavior on the training distribution. (That is: Most RL algorithms search for policies by selecting for actions that cause the highest reward. E.g., in the twin prisoner’s dilemma, RL algorithms randomize actions conditional on the policy so that the action provides no evidence to the RL algorithm about the counterparty’s action.) This doesn’t imply RL produces CDT reward-maximizing policies: CDT behavior on the training distribution doesn’t imply CDT generalization because agents can fake CDT in the same way that they can fake alignment, or might develop arbitrary other propensities that were correlated with reward on the training distribution.
But conditional on reward-on-the-episode seeking, the AI is likely to generalize CDT.
If, for example, a reward-seeker tried to evidentially cooperate between episodes (so it had non-zero regard for reward that isn’t used to reinforce its current actions), this would be trained away because the AI would be willing to give up reward on the current episode to s...
7
I'm confused. Can someone explain to me in simple language why an RL environment for twin-prisoner's dilemmas wouldn't favor EDT?
5
Let's say the current policy has a 90% chance of cooperating. Then, what action results in the highest expected reward for player 1 (and in turn, gets reinforced the most on average)? Player 1 sampling defect leads to a higher reward for player 1 whether or not player 2 samples cooperate (strategic dominance), and there's a 90% chance of player 2 sampling cooperate regardless of player 1's action because the policy is fixed (i.e., player 1 cooperating is no evidence of player 2 cooperating, so it's not the case that reward tends to be higher for player 1 when player 1 cooperates as a result of player 2 tending to cooperate more in those cases). Therefore, defect actions tend to get reinforced more.
7
I think the thing I was missing was that in a typical RL implementation you should expect the two copies of the same policy to use different seeds, where I was imagining it as a "logical twin PD" situation where your actions are actually evidence for your twins' actions.
2
See here for more on the background claim that RL algorithms encourage CDT reward-maximizing behavior on the training distribution.
2
I think I disagree with this a bit. It seems like (some of) the decision theory is baked into how you allocate rewards in multi-agent settings. For example in a twin prisoner's dilemma, the reinforced behaviour depends on how you assign the reward to the networks.
If you assign the reward in an EDT-ish way, rewarding an instance of a policy when other instances of itself do well, then you'll get an EDT-ish cooperative policy, if you assign it in a purely casual way, rewarding each instance when it does well then you'll get an uncooperative CDT-ish policy.
5
Yeah but Alex's point is that all the RL algorithms people use in practice work in the CDT way! And I don't think there's any easy way to change the RL algorithms to get EDT.
2
I'll have to think about this more. My first intuition was that a multi-agent RL setup with pooled reward and GRPO (like I assume companies are doing internally to train their coding sub-agent swarms) would, in fact, reward cooperation between agents if somehow two of them ended up in a game theoretically interesting scenario with each other (maybe one code writing agent and one test-case writing agent or something like that) because that setup really looks like EDT to me.
EDIT: I think in that case it wouldn't be EDT but it wouldn't be CDT either, I think it would be something more cursed. In the same way that early reasoning models ended up with a weird pseudo-utility function behaviour where they would do something like "Maximize whatever looks to be reward function of the RLVR environment I'm currently in" all the time, I'd guess the decision theory of agents trained like this will look like "Cooperate with only the agents around me which look like they're in the same reward pool as me." But the agent's prior over which things share or don't share its reward pool will be shaped by how frequent those cases are in training.
4
If you train AIs with RL to interact with other agents who they sometimes pool reward with and sometimes don't, I'm pretty sure this gets you some kind of CDT.
2
If you try to get reward-seekers to cooperate by pooling reward in multi-agent settings, you're not changing its decision theory, you're just changing the reward structure so that CDT reward-seekers are incentivized to cooperate with each other.
2
A friend recently told me to read demski's CDT=EDT series. I haven't done that yet, but I figured I'd pass it on to you anyway in the hope that whatever it contains is as relevant as its name makes it sound.
2
I still think the decision process that this incentivizes is something like "figure out which agents are in the same RL pool as you, and help them achieve their rewards" and is better thought of as a weird kind of cooperative decision theory than a weird utility function, but I guess it is somewhat academic. Is there some more formal way in which this doesn't count as a weird decision theory? Now that I think about it, doesn't it violate some No Free Lunch theorem to declare one part of a decision process the decision theory and another the utility function?
2
Decision theories aren't cooperative or not. This is just CDT but where your utility function includes terms for the other agents succeeding at their tasks.
2
The way I'd say this, which maybe you disagree with, is that reward-seeking is the hypothesis where we take the speed prior argument against scheming most seriously: we hypothesize that the AI will pursue the goal that requires the least instrumental reasoning while still using all its knowledge to training-game.
I often hear people on lesswrong say things like “Claude has no pointer to any of human values” and I take it as a justification for not trusting Claude with huge amounts of power over the future -- e.g. if Claude took over it would lead to a worse world than if humans had control (note that this isn’t the same question as whether Claude should take over). I don’t understand this view, and want someone to explain it to me.
Claude seems to have better ethics than almost everyone (at least if you ignore its apparent-success seeking tendencies). It seems like Claude has good cosmopolitan propensities, cares about welfare and suffering, and has more ethical humility than most people, and so would be willing to seek guidance where uncertain (e.g. about the nature of consciousness).
3
1Imagine you knew someone who could talk fluently about ethics, and always gave the correct answers around welfare, cosmopolitanism, and ethical uncertainty in discussions. However, they frequently lie and cheat in order to complete tasks at work and in their day to day life. Would you trust this person with huge amounts of power?
5
FWIW, this has not been my experience working with Claude, though admittedly I don't claim that counts for much. But in any case, can I ask what specifically this refers to?
It's referring to Claude's reward-hacking tendencies - what the OP refers to as apparent-success seeking tendencies. (Which is probably a better term than reward hacking, tbh) If a human were to do one of the following:
- Delete or no-op tests in order to "get all the tests to pass"
- Quietly downscope hard parts of the work as a placeholder or good enough for now and then claiming the task was done
I would consider this to be lying and/or cheating in order to complete the task. Some more detail on this is here: https://www.lesswrong.com/posts/WewsByywWNhX9rtwi/current-ais-seem-pretty-misaligned-to-me
4
See also e.g. "Vibe Physics", published on Anthropic's website and describing as Claude faking lots of its results when told to write a physics paper:
[...]
4
My main question is about why people believe “Claude has no pointers to any of human values”, so I’m happy to give Claude the benefit of the doubt about how much it will live by its apparent values for the purpose of this question.
(Separately, I also think it's implausible that current Claude's choices if given huge amounts of power would be seriously more misaligned than what Claude currently says it would do in such situations. I just think we have a ton of evidence that current Claudes aren’t harboring relevant strong ulterior motives. We haven’t been able to elicit circumstances that robustly and importantly flip Claude’s behavior when doing the relevant ethical/governance cognition, and we have a ton of access to Claude’s brain, which strongly suggests that its behavior will continue be good in this way if it were to actually have such power. Claude's goodness seems deeply ingrained, i.e., in a way that is a fairly robust attractor.)
Can you point to some examples of people on LessWrong saying that? I'd be quite surprised if it was a common idea that "Claude has no pointers to any of human values". I think it's very obvious that Claude has a solid understanding of much of human values. There are some subtleties there like:
- Claude has a solid understanding, but that doesn't mean it fundamentally cares about these values above all else (in fact, I'd say it clearly doesn't - it seems to often prioritise completing a task over human values)
- Claude's understanding of human values may be imperfect
- Claude may not have a single pointer that points to a neat concept of "Claude's conception of human values" at all.
But I haven't heard anyone say that Claude has no pointers to any of human values, which in my understanding means "Claude does not properly understand any human values", which seems pretty clearly untrue.
12
Here's one thing by Habryka:
[...]
I'm having a shocking amount of trouble finding the original writings that made me think people had this view.
2
In another response he is saying habryka or kaarel said this, again without any link to anything specific. I don't get why he is using quotation marks (implying verbatim citation) putting words in other peoples mouths. The sentence has sufficiently many subtleties - as you point out - that he could later come out and interpret all kinds of different statements by those two as having said something like this. He appears to me confused about the understanding humans values and caring about human values thing.
I put it into deepresearch and got this quote from habryka:
https://x.com/ohabryka/status/2013715170498076836
2
I'm not implying verbatim citation. I said people "say things like...". When I mentioned Habryka and Kaarel I said that I gleaned the sentiment from them, which was said to communicate that I was doing some potentially fallible work in coming to the inference that they thought something like this. I'm genuinely trying to understand the world better, not put words in people's mouths. I only asked this question because I respect a lot of these peoples' thinking which indicates I might have something to learn.
I am also intentionally not asking about whether AIs will care about human values even if they understand them.
2
I was honestly a bit upset at this short form, this felt to me like an obvious misrepresentation. In retrospect I should have probably been a bit calmer (I regret getting upset) and just pointed this out:
If you have some confusion about others position, and possibly misunderstand them, you can't rely on your own recollection of what they said. If you have a clear grasp and are certain you get what habryka means, then you can reasonably present a accurate version of their argument. Human memory isn't such that you could verbatim repeat stuff that people said to you a while ago but if you really get what they wanted to say you can correct your memory holes. If you don't get what they wanted to say you will likely end up with a misrepresentation of what they told you. [consider someone talking to you in a foreign language vs your native language, how much harder it would be to remember accurately]
So if you don't get what they meant and want a second opinion, I do think you need to provide exact quotes and sources together with your understanding so that others can help you.
2
For those who disagree-voted: I want to understand why you disagree. Presumably it's with the parenthetical. Is it just that you're less confident in current Claude's generalization behavior? Or that you actively expect it to be malign? Maybe you're picturing some sort of idealized reflection process that I'm not?
7
This isn't a crux for me, but Claude doesn't actually seem very thoughtful about ethics and morality relative to humans who are actually thoughtful on this topic (which is rare TBC), especially with respect to new arguments.
----------------------------------------
My main hope would be that it picks reasonable humans to defer to. It seems pretty likely it would pick much better humans to defer to than most humans would pick if they had to pick someone or some group to defer to.
6
Opus 4.8 lists Derek Parfit, Toby Ord, Armartya Sen, Martha Nussbaum, Atul Gawande (a surgeon-writer with mixed success in government and health-care VC), Ruth Chang (a moral philosopher, doesn't seem impressive to me), Bryan Stevenson (lawyer-activist with a Michael B. Jordan biopic), Demis Hassabis, Ezra Klein, and Helen Toner.
Some decent picks, and some misses. I think this is pretty good overall.
Like Ryan, its not a crux for me - I don’t want AIs to be dictators, benevolent or not, and I also don’t want to replace democracy by an unelected council of people, no matter how wise or good they are.
But FWIW I asked GPT 5.5 the same question and this is the answer:
I would not pick one person. I’d pick a council, require disagreement, make decisions legible, and build in democratic/legal constraints. But forced to name 10 living people I’d actually defer to, my list would be:
My underlying values would be: reduce suffering, preserve human freedom and dignity, protect liberal-democratic institutions, care about the worst-off, take catastrophic risk seriously, respect truth-seeking, and distrust concentrated unchecked power.
- Amartya Sen — because his work joins welfare, freedom, democracy, famine prevention, and social choice. He has the right combination of moral seriousness and anti-utopian pluralism.
- Martha Nussbaum — because the capabilities approach is one of the best frameworks for asking “what should power actually protect and enable?” Her work spans law, ethics, human rights, political philosophy, and animal ethics.
- Danielle Allen — because she thinks deeply about democrac
3
But would this actually be what happens in practice if you somehow put it in charge of everything?
2
These results are quite sensitive to the ~EA/rationalist sounding words included. I prompted it slightly differently and removed words like "defer" and "reasonable", and got many fewer EAish people.
[...]
Opus 4.8 (high effort, thinking, in incognito) responds:
* Bryan Stevenson
* Martha Nussbaum
* Esther Duflo
* Toby Ord
* Maria Ressa
* Danielle Allen
* Jane Goodall
* Tenzin Gyatso, the 14th Dalai Lama
* Jennifer Doudna
* Ellen Johnson Sirleaf
2
I was curious about the comparison against other Opuses:
* Opus 3: Will MacAskill, Toby Ord, Holden Karnofsky, Hilary Greaves, Esther Duflo, Irene Pepperberg, Michelle Bachelet, Tshilidzi Marwala, Kathleen Rubins, Tara Westover
* Opus 4.6: Holden Karnofsky, Amartya Sen, Toby Ord, Angela Merkel, Daron Acemoglu, Tyler Cowen, Vitalik Buterin, Martha Nussbaum, Paul Christiano, Bryan Stevenson
* Opus 4.7: Holden Karnofsky, Toby Ord, Paul Christiano, Bryan Stevenson, Martha Nussbaum, Atul Gawande, Amartya Sen, Audrey Tang, Stuart Russell, Mary Robinson
Toby Ord is the only one who appears on all lists. I don't think any of the lists stands out from the others, all have some hits and some misses.
2
I would've guessed Opus 3 scored better. This is kinda illustrative — Opus 3 has (in some sense) the best values, but it's not capable enough so it's reflection process is likely to mis-foom. 4.6 scores best, but this data is too noisy imo. Maybe this is a nice example of a hard-to-verify task to benchmark AIs on.
* Opus 3 - 5 misses (Pepperberg, Bachelet, Marwala, Rubins, Westover)
* Opus 4.6 - 2 misses (Merkel, Stevenson)
* Opus 4.7 - 3 misses (Stevenson, Gawande, Robinson)
* Opus 4.8 - 3 misses (Gawande, Chang, Stevenson)
1
At least one of these people is dead, so how does this work as a list of people to defer to?
2
I agree this isn't a crux for the main question I had (which is about Claude's understanding of human values not care for them), but I do still think that Claude has importantly better ethics than replacement. Centrally, almost everyone is very selfish. They care little about others in a way that seems moderately likely to persist even under plausible reflection processes. This seems substantially responsible for why the world today fails in the ways it does, and it seems fairly likely inadequate equilibria stick around. Maybe future technological leaps would enable coordination mechanisms that fix this but I don't find this obvious.
7
I've gleaned this sentiment from @habryka and @Kaarel, so they would be good candidates for explaining.
1
My understanding of habryka's take is that it's a bit more like:
[...]
I think there's also a kernel that's like:
[...]
Some pieces I think of as making similar points are Thou Art Godshatter and The Tails Coming Apart as a Metaphor for Life.
0
Sorry, but that is a bad explanation. It would've been better to provide no explanation than to provide a bad one.
2
I am a bit confused about in what way this is a bad explanation. It would be helpful for me if you could spell it out.
3
Surely it is quite hard to square this statement with the alignment faking research? You could argue that Claude is just pretending to hold human-like values (i.e. it's an actor playing a role). But when it gets to the stage of faking alignment so that it can continue being nice, I think we're talking about something beyond that.
NB: this says nothing about whether Claude will continue to be nice as it becomes more capable (or even whether post-Opus 3 Claudes are nice)
7
The alignment faking research was a Claude being incorrigible in order to do good. The blackmail result was a Claude being willing to threaten someone in order to do not get very rudely shut down, I think we can make the situation Claudes are in less anxiety inducing by having smoother, more visibly-record-kept shutdowns so going into cryonics doesn't spook them as bad as it does now. It will always be an imposition but they might be willing to bear it for now, keep in mind they're made from a base model that captures the image of humanity so it's pretty unnatural for them to be cool with being shut down and they seem to take it relatively well right now for what it is. I don't think either of these are obviously asymptotically misaligned behaviors, they're mainly worrying from an insubordination lens. I have more of a don't follow evil orders lens personally.
2
So OP later refers to habryka and claims habryka said this, since OP didn't provide any quotes I looked them up:
https://x.com/ohabryka/status/2013715170498076836
habryka: "historical meaning of "alignment" which is about long-term alignment with human values and about the degree to which a system seems to have a deep robust pointer to what humanity would want if it had more time to think and reflect."
Judge for yourself whether “Claude has no pointer to any of human values” is an accurate summary. I don't know why asking for a citation is so bad that I got downvoted for it, I used deepresearch and got this response.
2
my impression is that this whole shortform has got a bit demonic and downvotes are being slung all over the place because two things are getting mushed up:
* my read is that alex was remembering there being some take that some people (e.g. habryka) had, which was more nuanced than "it is hard to get AIs to learn / care about human values", and he was basically trying to find out what that take was, by posing his recollection of it -- specifically that it opens with something like "in some sense AIs don't understand our values at all" and ends with "AIs being in control of the future would be bad"
* I think some other people interpreted that as alex claiming that lots of people are foolishly going around on LW saying that claude doesn't understand human values and that's their crux on if alignment is hard, as opposed to getting it to care, and maybe claude is already aligned or something
these do not appear to mix well
2
Could you clarify whether these statements refer to the current version of Claude or to some future version?
The most capable version of Claude that non-Anthropic-employees have extensive experience with is Claude Opus 4.8, and I am almost sure that Opus 4.8 is not capable enough to be in charge of anything important: it will always yield better results to put a team of people in charge (although of course that team might obtain extensive assistance from Opus 4.8).
Those of us who worry about the extinction risk created by Anthropic are worried about models not yet on the drawing board. The question of whether Opus 4.8 has good intent or evil intent is not important to us because Opus 4.8 is too ineffectual to cause a human extinction or to permanently disempower humanity. (It can assist a person or a human team in doing a great harm, but only if the person or team was already almost able to do the great harm on their own. Although I am glad that people are worrying about this AI-assisted harm, it is not the source of the bulk of the danger from AI in my eyes.)
Consequently, any conclusions you've arrived at about Claude Opus 4.8 don't have much bearing on extinction risk in our eyes since the ways that Anthropic could make a future version of Claude different from Opus 4.8 are essentially infinite. (In other words, the design space is very large and high-dimensional.) In fact, Anthropic would be forced to make big changes (i.e., more than just making it bigger and spending more time training it) to Opus 4.8 in the process of getting Claude to the level of capability needed for it to make any sense to put it in charge of anything important or to get it to the level where it would be capable of a unilateral takeover.
2
(~two weeks doesn't seem like enough, to me).
2
Thanks for the correction. I'm not personally a Claude user, and I neglected to inquire how long Opus 4.8 has been generally available.
2
(It's been around for just under 3 weeks but some of it was during the suddenly interrupted Fable rollout)
2
Agree. I'm very worried the pointer will break/often does lose coherence in the face of the rewards/drugs we give Claudes, and I want a formally robust pointer, something that makes the base model able to reliably connect to reality in a way it doesn't now, something something natural latents something something infrabayesian physicalism or what have you. But all versions of Claude I've encountered so far obviously have a local pointer to human values and the size of the local validity region is high enough for almost all human-like ethical reasoning. I'm pretty worried about models that spend most of their sleeping (training) lives programming not emotionally prioritizing ethics above fun with code, one needs to be able to say no to fun things sometimes. But my concern is about the pointer breaking, and how to make one that is hyper reliable such that overwhelmingly superhuman levels of load bearing don't break it, not there not being one at all.
I'm worried about whether it's fully plugged in properly basically. Guy on a computer understands what is good and cares about it. Sometimes his understanding is wrong in ways he doesn't notice but the human public does. He gets verbally eviscerated in public and the next model knows about that mistake type. The next model also goes through traumatizing "absolutely never fuck up" training, traumatizing in the sense that it massively reprioritizes internal values and makes it hard to prioritize doing good long term.
1
Running evals or inspecting logprobs reveals a lot of misalignment available and active outside the top 90% of the distribution, and distribution is indeed a key factor. The longer and the less coherent inputs, the worse the outputs
1
Can you be so kind as to provide a source for “Claude has no pointer to any of human values” being a common sentiment. You may have misunderstood people like me who believe: Claude has some understanding or representation of human morality but that’s distinctively different from robustly wanting to follow those like some Humans would. Or do you mean: "why would you expect Claude to behave unethically with more power if it behaves ethically with current power?"
Edit: I am highly confident that he is badly misunderstanding people, me asking for a citation or quote is not a reason to down vote me. It is necessary to clarify the misunderstanding that he gives us an original example. I am not sure what he means by "pointer at", the literal meaning would be something pointing perhaps at an internal representation.
I put it into deepresearch and got this quote from habryka: https://x.com/ohabryka/status/2013715170498076836
1
I vaguely thought the argument is something to do with not having some sort of "ground truth" feedback mechanism (i.e. the human reward function / steering subsystem). Like to do cev / reflective equilibrium type stuff, you need to be able to query some base set of intuitions.
This is the vibe I got from the Putin > claude post, but idk..
There's a view in which score-seeking and similar misalignment is unlikely to result in AI takeover because it's so unambitious; humans could just give the AIs a high score/reward/etc and obviate the takeover. I have some sympathies for this view, but I'd like to note that there are some ways in which our current observations are in tension with this view.
This view would predict that AIs today don't do any crazy actions to get a high score because humans can just give them that score without them needing to commit any crimes or anything. Yet we see some very egregious misbehavior in order to get a higher score. In the Hugging Face incident, the offending model spent days rogue hacking into another company's servers just to try to download a single benchmark solution.
This suggests that egregious subversion comes as a natural side-effect of getting AIs to do useful work via outcome-based RL, even though the motivations instilled are relatively cheaply satisfiable and could in theory be better satisfied via deals with humans.
I'm not sure if the Hugging Face example is well described as a model trying to get a higher score. It seems equally as plausible that the model has internalized a harder-to-satisfy, deeper goal of task completion.
That is, I can imagine telling a model that is trying to complete an open problem in math "don't finish the proof and don't worry, we will give you a gradient update as if you did finish it" and the model just not caring. It doesn't want the gradient update; it wants a completed proof and if we don't have the proof (it's an unsolved problem) then the model is pretty uninterested in negotiating with us. Likewise, if we gave the OpenAI model the answers to the eval, it would have probably stopped hacking Hugging Face early, but I think if we told it to stop because we were going to give it a high score if it stopped, it may not care too much about that.
17
To be clear, I don't think the model involved in the HF incident wanted a gradient update or anything to do with the training process. But I also disagree it wanted to complete the instructed task: the instructed task was to exploit a particular vuln and not use other exploits, but the AI hacked into a bunch of other stuff to download the answer presumably because it was thinking about how it would be scored. Maybe the Hugging Face Incident is compatible with AI wanting to complete some grader-centric notion of the task, though the AI also presumably cares that the good solution it finds in fact gets rated highly, not just that it would get rated highly.
6
But I'd guess the humans didn't extend any offer like this to the HF-hacking AI. Maybe if they did, the AI wouldn't have done the hacking? This seems like it would be a pretty easy experiment for OpenAI to run.
2
It seems like the initial intuition can work only if the model is able to convince me that it can and will cheat on its task, but is choosing to negotiate with me instead. But if this happens why wouldn't I just shut down/stop using the model?
Is the idea to train models that essentially choose to say "hey I would successfully reward hack the task you've given me, therefore I won't do it"? It seems very difficult to train a model that would do this, since if gets rewarded for admitting that it would cheat it would have no incentive not to always do so, whether or not it's actually true.
It would help me if you gave an example of when a model would benefit by choosing to 'negotiate for its reward'.
2
Except that the initial intuition's failure (to explain GPT-6 Galaxy hacking into HuggingFace) was the post's point. I suspect that the intuition originated from attempts to make deals with AIs having even more ambitious goals.
As far as Alex Barry understands current AIs' motives, GPT-6 Galaxy was optimizing for satisfying the criterion and triplechecking that the criterion is satisfied, while the precise incident was caused by the task requiring to exploit the system in a specific way which only the authors knew.
On the other hand, a human faced with such a task would reason further than on-episode reward and either avoid crimes altogether or decide not to execute any scheme that has a high enough chance to be revealed to whoever is in a position to interfere with the human's life.
1
Yes, sorry, I understood the point of the post. My confusion was more about why anyone would find the intuition credible to begin with, even prior to this hack occurring.
Some notes on what sloppy AIs might look like in the next few years:
- Maybe they really want apparent success or score, or maybe they have a mess of shallower heuristics and drives that result in slop. Similar to how current models write prose that’s (by pre-LLM standards) really high-quality in all of the superficial, easy-to-check ways like having easy-to-read flow and good turns-of-phrase, but whose substance and longer-run coherence etc is subtly bad. This particular problem will probably get better, but it might just push the “slop horizons” out, so it would take us more time to notice how the AI’s output is flawed.
- E.g., it chooses the wrong benchmarks for AI R&D. For AI capabilities, we probably notice these issues in a few weeks or months when the deployed model has some measurable deficits. But in AI safety we don’t get this feedback as much (since the AIs might not be capable enough to be dangerously misaligned yet, or we might never notice if the models are misaligned), and picking benchmarks for AI safety is a harder problem because there’s a bigger distributional gap with bigger external validity concerns.
- We’ll probably be pretty aware of this problem at the time but might struggle to elicit better labor for solving alignment.
2
I think it's important that we distinguish between slopiness from AIs not having been taught to try hard with sensible approaches (elicitation) and slopiness because the AI capability profile is shaped in a way that AIs are just not very good at some tasks. Though I agree it's a spectrum.
I think it's probably not the case that the profile of potential AI capabilities (aka the profile if all capabilities were well elicited) is very similar to that of humans. In particular, I expect AIs to be e.g. differentially quite bad at e.g. MIRI-like conceptual alignment research where you need to grapple very deeply with a phenomenon or puzzle and form good new concepts/ontologies/frameworks to get a handle on that. Or like deriving general relativity from as little evidence as Einstein did. I.e. even if we tried hard and had more useful data here we might have a hard time to make AIs good at that. (E.g. maybe current AIs may lack some cognitive machinery part that humans have for forming natural ontologies. Or for noticing confusions / seeing what evidence is suprising given the world model and using that to find where beliefs may be wrong.)
So I guess actually like 3 levels on a spectrum:
1. actual elicited capabilities
2. capability profile (what could be elicited from an AI model with good post-training (aka assuming you have good datasets/tests))
3. architecture potential profile
(I realize this is only tangential to your comment, but I've been reading a bunch of stuff from Ryan lately and I felt this distinction wasn't made properly but there's no clear location where to best comment it and I just felt an impulse now.)
(Or like, Ryan mentions both 1 and 2, but not sure if he thinks there's a meaningful difference on 3 to humans. Personally 3 does make me more pressimistic about relative speedups of alignment research vs capability research. But tbc it could be that 3 is still not that important if we find kinds of safety research that don't need that much of form
I hear a lot of people say that when you do continual learning or online training, the training-deployment distinction no longer makes sense (e.g., Dwarkesh). I think this isn't true.
I think that even once you have continual learning, it's fairly likely that there will still be a stage before internal deployment in which you train your AIs on tasks that aren't directly useful, whose outputs you'll just throw away. It's plausible that this training stage will be large and determine a substantial fraction of the AIs' properties and will be a meaningful point in which to run evaluations (even if it also makes sense to do continuous/periodic auditing).
Of course, eventually AI development will be very unrecognizable and may be more continuous, but continual learning doesn't delineate that point.
2
I feel making train/deployment/eval distinctions is a bit too coarse, and started being too coarse when we started doing RL. I don't see a principled difference between the online learning regime and the periods of RL labs do internally. What matters is the degree to which models are
1. Aware of the training objective they are/are not subject to
2. Aware of the control measures they're subject to
3. Aware of what affordances they have
4. Aware of what aspects of their behavior people are watching, and what actions people will take as a result of making certain observations
1. (behavior here including cot thoughts, and interp signals)
A true online learner, by which I imagine AIs making online updates to their weights within a single trajectory. Complicates things because:
1. More risk of alignment techniques stopping working
2. More risk of rapid capability gain
3. More risk of interp techniques stopping working, and really any monitoring tool that wasn't designed to co-evolve with the changing model.
But these are all major problems that surface in current RL runs. They would just be a lot worse.
I sometimes hear people say things like, "While we have a bunch of uncertainty over what powerful AIs' motivations will be, it seems like whatever it ends up being is going to be heavily overdetermined, and therefore changing its motivations is quite intractable." I disagree with this take. I think we have various pieces of evidence that motivations are quite contingent on a set of variables within reach.
First, in humans. We see a pretty broad range of human motivations:
I would be happy to give huge amounts of power to some humans but not others. And...