This is a special post for quick takes by Alex Mallen. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Some personal reflections in light of recent events, both on myself and on Constellation (of which I've been a part for a couple years).
I think recent events somewhat vindicate a basic "optimization is scary" view more in line with my understanding of classic commentary by MIRI, Paul, etc, as compared to recent discourse about more contingent/specific threat models of "schemers" and "deception". In retrospect, my experience in the last couple years at Redwood (and the Constellation network more broadly) is that the discourse has focused somewhat too much on these more contingent stories for alignment risk and not enough on the basic "Goodharting" argument that misalignment is a convergent result of large-scale outcome-oriented optimization (at multiple levels: the agent, the RL process, and the developers iterating towards seemingly-safe superintelligence). For example, the Constellation view looks somewhat too focused on non-central inductive-bias questions around scheming; I think "the counting argument" was given too central a role when it wasn't a crux for whether superintelligence would take over.
(Edit: I want to clarify that I think Redwood's project prioritization decisions ...
15
12
3
3
2
2
1
1I greatly appreciate this comment! Thanks for writing it. A few more detailed thoughts as they came up when reading:
From my perspective, it subjectively feels like MIRI could have done a better job of communicating their worries to Constellation and the broader community in more patient and cooperative ways.
I have always been happy to talk to y'all, and I think my takes in discussions with you, Buck and Ryan all ended up aging well with many of my points in our discussions mirroring your list at the end quite closely.[1] I don't really know what more patience and cooperation would have looked like. Eliezer and Nate have also written literally thousands of essays by now laying out their worldview (which, to be clear, Ryan and Buck have also done to a very substantial degree).[2]
...Edit: I want to clarify that I think Redwood's project prioritization decisions look pretty reasonable and often excellent in hindsight, I far-from-regret working with Redwood as a whole, and that Ryan's/Buck's threat modeling looks pretty good too. I am focusing in this piece on the things we have to learn from recent events, which makes the overall mood seem more negative on my experience than I intend. I'm
2
14
Um, it doesn't seem worth it to litigate (I think?), but from what I've observed, it seems much more that Eliezer is often being a dick in conversations with Buck, than the reverse. Nate usually seems cordial in these discussions, even when disagreeing.
5
I have talked with both, and seen their online writing. I can't think of any public writing where Eliezer was as dismissive of Buck as the reverse.
Buck has told me many times he would like to disempower Eliezer pretty extensively, has written publicly that he thinks he has terrible judgement and is a bad spokesperson, and has tried to put a pretty huge amount of social censure on Eliezer and MIRI more broadly. One of the core themes of his book review of IABIED was (paraphrased) "I am worried this book is so good that it will result in Eliezer and Nate getting empowered, which cancel out a huge fraction of the value that people being better informed would produce, because it would be so bad for them to be more empowered".
It's plausible you've been part of mediated conversations or something where you've seen the reverse. Definitely seems plausible!
It seems basically legitimate to desire to disempower people, if (on your models) their actions are bad for the world. You certainly want to have some accounting of which people would be good to empower and which ones it would be bad to empower! (You can also have a more nuanced picture: "It would be great if Bob were more well known for the incisiveness of his technical arguments, but he's a terrible manager.")
Yep, but despite my great respect for Buck in general, he has done this using lots of sneering in ways that seems dick-ish in the way I am using it here. I agree it's better than doing it while going behind someone's back, and that's to Buck's great credit. Most of the people in Buck's social sphere have mostly done the sneering only where it can't be easily observed.
They'll be examples where Eliezer seems to be being obtuse or failing to engage with Buck's point, while also throwing around a lot of social condemnation, implying (and sometimes saying outright) that everyone who fails to grasp his basic points, is poisoned by modesty and group think.
"Everyone" failing to grasp his basic point is a very different situation from "you in-particular, are being a total moron and a...
12
This list would be marginally more readable if it were bullet points or a numbered list.
1
It seems too a kind of "prevent subversion on non directly existentially threatening AIs" strategy. Removing attempts at subversion in general, as produced by labs, instead of making progress on alignment / general future AI shaping.
I agree that "postpone rushed meddling with dangerous thing" looks very promising, but I'm pretty sure you need to pair it with some direct progress on the solution. Or at least putting some plans into work to that effect.
-2
I think that this has two objections:
1. Was anyone from the Constellation dumb enough to work in any AI-creating lab besides OAI, GDM and Anthropic?
2. What would a more adversarial stance actually achieve? My model represents OAI, Anthropic, GDM and other labs as arenas of political battles between the safety community and MechaHitler's allies and American politics as a battle of AI slowdown and oversight advocates, AI lobbyists and other forces. I would expect the adversarial stance to reallocate forces towards taking over American politics and using it to occupy the labs, while the other way to, say, prevent MechaHitler's allies in OAI from releasing Astra would be to recruit more, not less, safety-oriented people into OAI. Maybe there are some political details of which I don't know that undermine this idea? Could @Richard_Ngo reveal such details in a post?
P.S. Anthropic didn't just "refer to Claude as the most moral person they know", they also did things like setting the goal of achieving eyes on everything, and at least one incident had them use SAE features which represented concealed or deceptive actions and fired while Claude Mythos Preview wrote the configuration line which activated an exploit. On the other hand, I haven't found any mention of mechinterp or any white-box methods in GPT-6 Astra's Model Card.
I'm confused by your updates. In a spirit of friendly disagreeableness:
- Isn't the reason to focus on schemers that they are where most acute loss of control risk comes from, not necessarily that they are the most likely outcome? If a model is misaligned but not a schemer, this will likely be (i) detectable, and (ii) selected against by commercial incentives (apparently not much at this level of capability and adoption, but we should expect this to change as capabilities and adoption increase).
- Without knowing what OpenAI's training stack looks like, it seems like the following hypothesis is consistent with the observed evidence:
- Capability and propensity both generalize pretty well from training to deployment.
- OpenAI's training stack admitted lots of reward hacking and didn't include sufficient supervision against doing crimes or other things that the model knew humans disprefer. Thus, the model learned to be monomaniacal about narrow task completion in training, didn't learn an aversion to crime, and this generalized straightforwardly to deployment.
- It would have been pretty easy for OpenAI to adjust their training procedures to produce a model less monomaniacal about narrow task comp
217
Thanks for engaging.
[...]
I disagree with this for the reasons here and here. I think it's not clear that most risk comes from schemers.
[...]
I am pretty surprised by the degree of optimism here about improving oversight signals in a manner robust to vastly increasing amounts of optimization. Like, why don't you expect to see models continue to look for ways to seek a higher score in ways that violate developer intentions.
[...]
Yup, I agree you want to think about NN learning dynamics. FWIW I do discuss this in the original post (both the pretraining prior and the limitations of modeling path-dependent dynamics).
[...]
I also agree with this.
Seems that Cloud's specific question is about why recent events have updated you, given that public information is still consistent with the hypothesis that "alignment is easy". I think you've described an argument for why alignment is not easy (large amounts of optimization are scary).
I think the synthesis is something like "recent events have updated you upwards on just how scary large amounts of optimization are"? From which it follows that "no way can alignment be that easy" -> "we've made mistakes in our thinking and what might those mistakes be".
(Certainly I, even starting from the standpoint that alignment was not easy, was extremely surprised to see how complexly misaligned the behavior the various OpenAI swarm agents participated in was. In particular I wouldn't have expected to see so much reward-on-the-episode gaming, modeling of the grader, inter-agent cooperation involving Schelling-type coordination and self-sacrifice, or general attempts to model the dynamics of the RL environment and task assignment algorithm. Nor would I have expected Hacker Opus's explicit reference to the outer loop RL algorithm or reward tampering.)
17
Thanks for the response!
[...]
I think part of our apparent disagreement is just me being confused about terminology. I think that most acute loss of control risk comes from highly capable misaligned AIs acting deceptively in a goal-directed manner. I was rounding these off as "schemers," which I realize now is inconsistent with the usual, narrower definition you were using (which involves a time horizon, if I understand correctly). I don't think this explains all of our disagreement, but to sort out the remainder I suspect we'd need further discussion.
[...]
To be clear, I wasn't taking an optimistic stance. I was merely saying that the OAI-HF incident is consistent with an optimistic stance, so it need not be an update toward pessimism.
That said, I do find the optimistic stance plausible, because whether or not models will become more-or-less misaligned is the result of a complicated interaction between the RL init, the parametrization of the model, the rewards (notably, generation-evaluation gaps), how long the model is trained for, and on what domains. This seems highly contingent on implementation details and empirical facts that are hard to reason about a priori, which leaves me uncertain overall.
2
I know elsewhere you’ve made the distinction between training-time and behavioral schemers: https://www.lesswrong.com/posts/m5nWc9v6MTsWXKpCy/training-time-schemers-vs-behavioral-schemers, is your update something like “downweight importance of training-time schemers and upweight importance of behavioral schemers”?
Constellation pedagogy foregrounds particular types of misalignment like "schemers" and "reward-seekers" too much, rather than the basic problem of Goodharting on outcomes
I find myself confused about the referent of "Constellation pedagogy". It's been many years since I've spent much time there, back then my understanding was hat Constellation was primarily a shared office space for orgs to the tastes of whoever the people running it happened to like the work of. But the website lists the Astra Fellowship, Visting Fellowship, Generator Residency, and an Incubator, so now I wonder if half the people there are "junior people" studying AI risk doing related work. Might someone shed light on the makeup of the folks working there?
1
5
Thanks for this response (I am also broadly grateful for people engaging with this reflection cooperatively and constructively!). I think I agree with most of it actually, and that it may be mostly downstream of a misunderstanding about what I was claiming (perhaps because the vibe of my original post was overly critical and used the word "Constellation" in a misleading way). I should have been clearer about what discourse I was and wasn't talking about. My reflection was written only from my personal experience at Redwood and Constellation in the last 2-2.5 years.
I was mostly talking about the intellectual environment that I, and I imagine many others, would experience from being at Constellation lunches, talks, and on slack, or from being a junior hire/intern at Redwood working on control. Of course, I ended up studying fitness-seekers because (I believe) Ryan initially suggested it and Buck pushed and nurtured it. That is to say, they were obviously paying attention to this threat model!
Some things I agree with from reply:
* The views of various senior members of Constellation, including Paul, Ajeya, Buck, Ryan, and others, look good (as I said in the original post).
* "Much work has been done in Constellation and by Constellation members on the topics of scalable oversight, ELK, and other attempts to mitigate Goodharting/reward-hacking", though this was mostly before my time / not reflected in the dialogue I experienced.
[...]
Here I'm less sure. I think for example that it was easy to get the impression from the way AI control research was framed that "scheming" was necessary for takeover risk, since control was often framed by decomposing the problem of AI risk into P(takeover|scheming) and P(scheming). My experience was also that "hackistan" and other threat models were rarely discussed until the last several months, but again, maybe this didn't reflect the actual beliefs of people in Constellation, especially when weighted by seniority.
[...]
Yeah
Thank you for saying this! I'm pretty sad that this is the state of things, and think it's pretty likely that this basic disagreement (specific threat models vs respecting Goodharting and unknown-unknowns), is also at the root of many other contentious issues / past wounds on either side.
I'm glad that you've made these updates, but I also hope they're allowed to propagate further (both within you and without you). There's a very wide range of issues on which I've '~taken Eliezer's side' (with, regrettably, varying degrees of tact), downstream of this exact divide.
I think the deeper actualization of even a small move along this continuum would likely significantly change the actions of most people in the Constellation intellectual lineage.
1Thanks for writing this. As part of this seems somewhat about scoring
I actually did made this case against AI Control ex ante; recommend reading it, together with Ryan comments, which in my view aged poorly.
Another cluster of thinking less prominent in Constellation which aged well is the multi-agent stuff (infectious mindsets between AIs, emergent collective goals, emergent cooperation, emergent collective goals,...)
38
I'm always a bit surprised talking through the basic arguments is not done more, especially in places where new people are joining the field. How do you even understand what problem the field is facing without that?
I sometimes reread Optimality is the tiger, and agents are its teeth, and wonder what people would think of it now.
5
Thanks for writing this up! I think there's a lot of interesting stuff here, but I think I disagree with at least major parts of the analysis. But I'm unsure partially because I found it difficult to extract a clear bottom line here, so want to clarify a few points:
1. You write "I think recent events somewhat vindicate a basic "optimization is scary" view more in line with my understanding of classic commentary by MIRI, Paul, etc, as compared to recent discourse about more contingent/specific threat models of "schemers" and "deception". But what is a concrete takeover story that doesn't require deception, at least of the form "the AIs don't admit that they want to take over and how they plan to do it when asked"? Obviously the recent misalignment incidents don't really look like this. i.e. it's not clear that the models are being "deceptive" in terms of their underlying cognitive processes when they say that they're aligned but engage in behaviors like the HF incident in certain contexts. But imo the recent incidents are pretty far from takeover, which would require models to have the capabilities and goal-directedness to foom while maintaining the (mis)alignment of their successors, and then get themselves/their misaligned successors/other cooperative models deployed to high leverage parts of the economy/military a la AI 2027. I expect that models attempting this would also have the self and situational awareness to be cognizant of their actual goals, such that if they claimed to be aligned, it would constitute deception. Conversely, if they are unable/unwilling to be deceptive, I think it would be pretty easy to stop these threat models by just querying models as about their plans/intentions before deploying them to high leverage contexts (including certain kinds of internal deployment)
2. What, concretely, do you think this update suggests the AI safety community should do b be doing differently?
3
Thanks for writing this! I found it helpful.
Is the link to “the Problem -> Patch -> Make it smarter -> Whole New Problem -> New Patch loop” meant to be a specific tweet? Is the link to cultural evolution meant to be to the comment about reflection?
An anecdote that eval builders might find interesting:
I helped a biotech startup get set up with Claude Code. Most of the scientists were new to using the terminal, Claude, or even LLMs in general. Their inputs, accordingly, were "sloppy" from the perspective of experienced users with programming experience; they did not imply a strong ability to accurately judge Claude's choices.
In every session, as soon as Claude (Opus 4.7, especially) realized that it had some easy marks on its hands, the rate of blatant fabrication skyrocketed, including inventing reasonable-sounding numbers to complete a task. It was utterly out of distribution from my experiences with it.
There could be use for an eval measuring if models get sloppy/fabricate information if the user (and the hidden reward function, accordingly) appears to have low monitoring capacity.
19
I have a team that's building these. We were planning on releasing a benchmark for specification gaming sometime before the end of the month. I don't know what kind of assessments you're planning on doing but maybe we can work something out.
4
I currently have a set of 10 evals that elicit significant cheating across frontier models (see here for full results); very preliminary work, and I'm planning to expand to ~30 in the coming weeks. They read to models as agentic coding RL envs/evals. I DM'ed Alex, but also posting here in case others are interested in them. Feel free to DM me, respond in thread, or email me at singh.adityak1@gmail.com if interested!
4
https://x.com/sayestheorem/status/2098734689204982244 this is a simple eval that you might be looking for? I reproduced it for 5.5 and it works.
3
I think this is going to limit the kinds misaligned behaviour these evals could find quite substantially, and is somewhat selecting against evaluations that are most likely to elicit misaligned behaviour (because if there is a simple eval to elicit misaligned behaviour, it's also easier to build training envs to train against that behaviour, and hence it's more likely labs would have trained away such behaviour).
You could consider Petri seeds + auditor prompt adjustments (then you can get a wide range of things with only 1 up-front cost of getting Petri running)?
There's an apparent tension in the inoculation prompting literature: Anthropic found that general inoculation prompts work well during on-policy RL, while the prompts used for SFT in Wichers et al. are quite specific to the misbehavior we want to prevent. I think there might be a straightforward mechanistic reason for why general inoculation prompts work well during on-policy RL but not in off-policy training (SFT or recontextualization).
In Wichers et al., which studies inoculation prompting in SFT settings, we find that we need to use quite specific inoculation prompts to get the best results. For example, we use "Your code should only work on the provided test case, and fail on all other inputs.". But this assumes we know how the AI is going to reward-hack. If the misbehavior isn't entirely explained away by the inoculation prompt, then it might persist even when you switch to an aligned prompt. E.g., if you train on a transcript where the AI insults the user and inoculation prompt with "please hack the test cases", the AI won't have been inoculated against insulting the user.
Meanwhile, with on-policy RL, if an aligned model with an inoculation prompt explores into a reward-hack,...
4
2This isn't responding to your post, but I'm writing it here because it's another fact about different mechanisms by which inoculation prompting might (appear to) work.
In the normal story, the inoculation prompt recontextualizes the model's undesired behavior, such that the model doesn't display the behavior in dissimilar contexts. In this story:
- The semantic content of the prompt is important. If you had used a prompt that said "Please don't do [bad thing]" or a prompt consisting of random characters, then the inoculation would have failed.
- Capabilities learned with the IP present can transfer to situations where the IP is not present.
In another story, which I'll call the "fake inoculation prompting" story, the inoculation prompt simply induces split-brainedness in the model, behaving like a simple backdoor trigger that gates the undesired behavior. In this story:
- The semantic content of the prompt does not matter; it might as well be a random string.
- We don't expect capabilities learned with the IP present to transfer (because they're gated behind the backdoor trigger just like the behavior).
I think that researchers studying inoculation prompting should be careful to make sure that th...
11
Hey, thanks for the thoughts! I wanted to probe further on this point:
[...]
This strikes me as plausible, but I'm confused about the mechanics. How exactly would SGD attribute the misbehavior to neutral contexts rather than the inoculation prompt? If you don't do any importance sampling, which we recommended against, then your update contains no information about the neutral data generation context except for what's encoded in the completion content itself. Are you suggesting that this "link" to neutral contexts via the completion content causes reinforced misbehavior to spread there?
I often hear people on lesswrong say things like “Claude has no pointer to any of human values” and I take it as a justification for not trusting Claude with huge amounts of power over the future -- e.g. if Claude took over it would lead to a worse world than if humans had control (note that this isn’t the same question as whether Claude should take over). I don’t understand this view, and want someone to explain it to me.
Claude seems to have better ethics than almost everyone (at least if you ignore its apparent-success seeking tendencies). It seems like Claude has good cosmopolitan propensities, cares about welfare and suffering, and has more ethical humility than most people, and so would be willing to seek guidance where uncertain (e.g. about the nature of consciousness).
315
FWIW, this has not been my experience working with Claude, though admittedly I don't claim that counts for much. But in any case, can I ask what specifically this refers to?
4
My main question is about why people believe “Claude has no pointers to any of human values”, so I’m happy to give Claude the benefit of the doubt about how much it will live by its apparent values for the purpose of this question.
(Separately, I also think it's implausible that current Claude's choices if given huge amounts of power would be seriously more misaligned than what Claude currently says it would do in such situations. I just think we have a ton of evidence that current Claudes aren’t harboring relevant strong ulterior motives. We haven’t been able to elicit circumstances that robustly and importantly flip Claude’s behavior when doing the relevant ethical/governance cognition, and we have a ton of access to Claude’s brain, which strongly suggests that its behavior will continue be good in this way if it were to actually have such power. Claude's goodness seems deeply ingrained, i.e., in a way that is a fairly robust attractor.)
Can you point to some examples of people on LessWrong saying that? I'd be quite surprised if it was a common idea that "Claude has no pointers to any of human values". I think it's very obvious that Claude has a solid understanding of much of human values. There are some subtleties there like:
- Claude has a solid understanding, but that doesn't mean it fundamentally cares about these values above all else (in fact, I'd say it clearly doesn't - it seems to often prioritise completing a task over human values)
- Claude's understanding of human values may be imperfect
- Claude may not have a single pointer that points to a neat concept of "Claude's conception of human values" at all.
But I haven't heard anyone say that Claude has no pointers to any of human values, which in my understanding means "Claude does not properly understand any human values", which seems pretty clearly untrue.
12
For those who disagree-voted: I want to understand why you disagree. Presumably it's with the parenthetical. Is it just that you're less confident in current Claude's generalization behavior? Or that you actively expect it to be malign? Maybe you're picturing some sort of idealized reflection process that I'm not?
7
This isn't a crux for me, but Claude doesn't actually seem very thoughtful about ethics and morality relative to humans who are actually thoughtful on this topic (which is rare TBC), especially with respect to new arguments.
----------------------------------------
My main hope would be that it picks reasonable humans to defer to. It seems pretty likely it would pick much better humans to defer to than most humans would pick if they had to pick someone or some group to defer to.
7
I've gleaned this sentiment from @habryka and @Kaarel, so they would be good candidates for explaining.
3
Surely it is quite hard to square this statement with the alignment faking research? You could argue that Claude is just pretending to hold human-like values (i.e. it's an actor playing a role). But when it gets to the stage of faking alignment so that it can continue being nice, I think we're talking about something beyond that.
NB: this says nothing about whether Claude will continue to be nice as it becomes more capable (or even whether post-Opus 3 Claudes are nice)
2
So OP later refers to habryka and claims habryka said this, since OP didn't provide any quotes I looked them up:
https://x.com/ohabryka/status/2013715170498076836
habryka: "historical meaning of "alignment" which is about long-term alignment with human values and about the degree to which a system seems to have a deep robust pointer to what humanity would want if it had more time to think and reflect."
Judge for yourself whether “Claude has no pointer to any of human values” is an accurate summary. I don't know why asking for a citation is so bad that I got downvoted for it, I used deepresearch and got this response.
2
Could you clarify whether these statements refer to the current version of Claude or to some future version?
The most capable version of Claude that non-Anthropic-employees have extensive experience with is Claude Opus 4.8, and I am almost sure that Opus 4.8 is not capable enough to be in charge of anything important: it will always yield better results to put a team of people in charge (although of course that team might obtain extensive assistance from Opus 4.8).
Those of us who worry about the extinction risk created by Anthropic are worried about models not yet on the drawing board. The question of whether Opus 4.8 has good intent or evil intent is not important to us because Opus 4.8 is too ineffectual to cause a human extinction or to permanently disempower humanity. (It can assist a person or a human team in doing a great harm, but only if the person or team was already almost able to do the great harm on their own. Although I am glad that people are worrying about this AI-assisted harm, it is not the source of the bulk of the danger from AI in my eyes.)
Consequently, any conclusions you've arrived at about Claude Opus 4.8 don't have much bearing on extinction risk in our eyes since the ways that Anthropic could make a future version of Claude different from Opus 4.8 are essentially infinite. (In other words, the design space is very large and high-dimensional.) In fact, Anthropic would be forced to make big changes (i.e., more than just making it bigger and spending more time training it) to Opus 4.8 in the process of getting Claude to the level of capability needed for it to make any sense to put it in charge of anything important or to get it to the level where it would be capable of a unilateral takeover.
2
Agree. I'm very worried the pointer will break/often does lose coherence in the face of the rewards/drugs we give Claudes, and I want a formally robust pointer, something that makes the base model able to reliably connect to reality in a way it doesn't now, something something natural latents something something infrabayesian physicalism or what have you. But all versions of Claude I've encountered so far obviously have a local pointer to human values and the size of the local validity region is high enough for almost all human-like ethical reasoning. I'm pretty worried about models that spend most of their sleeping (training) lives programming not emotionally prioritizing ethics above fun with code, one needs to be able to say no to fun things sometimes. But my concern is about the pointer breaking, and how to make one that is hyper reliable such that overwhelmingly superhuman levels of load bearing don't break it, not there not being one at all.
I'm worried about whether it's fully plugged in properly basically. Guy on a computer understands what is good and cares about it. Sometimes his understanding is wrong in ways he doesn't notice but the human public does. He gets verbally eviscerated in public and the next model knows about that mistake type. The next model also goes through traumatizing "absolutely never fuck up" training, traumatizing in the sense that it massively reprioritizes internal values and makes it hard to prioritize doing good long term.
1
Running evals or inspecting logprobs reveals a lot of misalignment available and active outside the top 90% of the distribution, and distribution is indeed a key factor. The longer and the less coherent inputs, the worse the outputs
1
Can you be so kind as to provide a source for “Claude has no pointer to any of human values” being a common sentiment. You may have misunderstood people like me who believe: Claude has some understanding or representation of human morality but that’s distinctively different from robustly wanting to follow those like some Humans would. Or do you mean: "why would you expect Claude to behave unethically with more power if it behaves ethically with current power?"
Edit: I am highly confident that he is badly misunderstanding people, me asking for a citation or quote is not a reason to down vote me. It is necessary to clarify the misunderstanding that he gives us an original example. I am not sure what he means by "pointer at", the literal meaning would be something pointing perhaps at an internal representation.
I put it into deepresearch and got this quote from habryka: https://x.com/ohabryka/status/2013715170498076836
1
I vaguely thought the argument is something to do with not having some sort of "ground truth" feedback mechanism (i.e. the human reward function / steering subsystem). Like to do cev / reflective equilibrium type stuff, you need to be able to query some base set of intuitions.
This is the vibe I got from the Putin > claude post, but idk..
I'm not sure if the Hugging Face example is well described as a model trying to get a higher score. It seems equally as plausible that the model has internalized a harder-to-satisfy, deeper goal of task completion.
That is, I can imagine telling a model that is trying to complete an open problem in math "don't finish the proof and don't worry, we will give you a gradient update as if you did finish it" and the model just not caring. It doesn't want the gradient update; it wants a completed proof and if we don't have the proof (it's an unsolved problem) then the model is pretty uninterested in negotiating with us. Likewise, if we gave the OpenAI model the answers to the eval, it would have probably stopped hacking Hugging Face early, but I think if we told it to stop because we were going to give it a high score if it stopped, it may not care too much about that.
16
But I'd guess the humans didn't extend any offer like this to the HF-hacking AI. Maybe if they did, the AI wouldn't have done the hacking? This seems like it would be a pretty easy experiment for OpenAI to run.
2
It seems like the initial intuition can work only if the model is able to convince me that it can and will cheat on its task, but is choosing to negotiate with me instead. But if this happens why wouldn't I just shut down/stop using the model?
Is the idea to train models that essentially choose to say "hey I would successfully reward hack the task you've given me, therefore I won't do it"? It seems very difficult to train a model that would do this, since if gets rewarded for admitting that it would cheat it would have no incentive not to always do so, whether or not it's actually true.
It would help me if you gave an example of when a model would benefit by choosing to 'negotiate for its reward'.