This is a special post for quick takes by Alex Mallen. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
There's a view in which score-seeking and similar misalignment is unlikely to result in AI takeover because it's so unambitious; humans could just give the AIs a high score/reward/etc and obviate the takeover. I have some sympathies for this view, but I'd like to note that there are some ways in which our current observations are in tension with this view.
This view would predict that AIs today don't do any crazy actions to get a high score because humans can just give them that score without them needing to commit any crimes or anything. Yet we see some very egregious misbehavior in order to get a higher score. In the Hugging Face incident, the offending model spent days rogue hacking into another company's servers just to try to download a single benchmark solution.
This suggests that egregious subversion comes as a natural side-effect of getting AIs to do useful work via outcome-based RL, even though the motivations instilled are relatively cheaply satisfiable and could in theory be better satisfied via deals with humans.
Yes, sorry, I understood the point of the post. My confusion was more about why anyone would find the intuition credible to begin with, even prior to this hack occurring.
But I'd guess the humans didn't extend any offer like this to the HF-hacking AI. Maybe if they did, the AI wouldn't have done the hacking? This seems like it would be a pretty easy experiment for OpenAI to run.
7Alex Mallen
To be clear, I don't think the model involved in the HF incident wanted a gradient update or anything to do with the training process. But I also disagree it wanted to complete the instructed task: the instructed task was to exploit a particular vuln and not use other exploits, but the AI hacked into a bunch of other stuff to download the answer presumably because it was thinking about how it would be scored. Maybe the Hugging Face Incident is compatible with AI wanting to complete some grader-centric notion of the task, though the AI also presumably cares that the good solution it finds in fact gets rated highly, not just that it would get rated highly.