maybe this patching (ie incrementally making rewards better track desirable behavior) could be done by AI-augmented humans
I’m surprised at how often this is treated as an open question. If this were true, presumably you’d be able to show it in some weak-to-strong paper, then people could assess the technical claims / gaps. The premise often seems to be “maybe as humans become weak supervisors, it just happens to be fine“, but “just happens to be fine” assumes some solution to weak-to-strong generalization.
I’ve made this point previously here specifically with hill climbing visible reward hacking in mind:
The Ten People On The Inside likely should prioritize things that capabilities teams aren't going to do anyway and people considering joining labs with similar motivations should likely think about whether they're sure they're going to be able to do work that wouldn't otherwise be done for capabilities reasons. The thing that's top of mind here is reward hacking, which is increasingly becoming a capabilities problem[1]. To the extent research is done here, it seems pretty important to avoid the failure mode of "repeatedly iterate on environments and observable behavior"[2] which is the kind of thing I would expect by default for capabilities reasons.
Buck had a similar point here:
AI safety people often worry about their work having capabilities externalities. I feel like the inverse problem is often a much bigger deal: the stuff you do might be done later by the capabilities/product people in the course of their work, meaning your work isn't counterfactual! I think people should try to distinguish these concerns more clearly.
(See also Bronson Schoen's similar post three hours earlier. I wrote this shortform in Redwood slack a few weeks ago and am just posting it now. Obviously this isn't a very original take.)
It’s been surprising to me how often “maybe fixing visible misalignment as we go is enough, and we never need to solve W2S/SO” is treated as a “maybe it will work” strategy. The concensus in the research as far as I understand it has been “obviously you need a solution there, no matter how much you iterate against visible misalignment”:
I’ve tried to elicit public answers on this:
I’ve seen the sentiment a few times that we don’t need solutions to the issues raised in Automating Alignment Is Harder Than You Think, i.e. something like scalable oversight / weak-to-strong generalization. Has anyone made this case before? I’ve heard it often from people inside (and even outside) of labs, and my impression is that it’s load bearing for research prioritization with respect to which research directions to pursue and how to allocate researcher time.
But so far it’s just something I’ve seen multiple labs express without ever defending it. My default expectation is that people who currently argue for “maybe iterating against visible misalignment is enough“ will soon switch to “well obviously we never meant it’d be enough, but it may be sufficient for bootstrapping an automated alignment researcher to solve those problems” (without in fact having focused all their effort on getting their automated alignment researcher loop to solve those problems). I think entertaining “iterating against visible misalignment without solving W2S/SO” as coherent is going to cost a lot of extremely valuable time and allows for labs to redirect safety researchers to fix things that interfere with capabilities without pushback.
tldr: here:
Labs right now can allocate resources towards either (1) showing that automated alignment researchers can lower rates of visible misbehavior or (2) legibly demonstrating deeper problems that will arise with this approach.
I worry that (1) enables further “scaling at maximum speed”, especially if visible reward hacking impacts AI R&D acceleration (and there’s thus a strong incentive to hill climb).
It seems important that individual researchers at labs worried about risk are working on (2), even though there will be continued incentives to work on (1).
This can take the form of specific safety training targeting the observable behavior, post-hoc environment fixes, or having an iteration loop between your pre-deployment automated alignment evals and your alignment training
Here's a basic argument that focusing on patching reward hacking is misguided and will fail. Argument is as follows:
This was written in the spirit of More people should write basic AI safety explainers. It's an attempt to distill my understanding of what we're seeing right now + a variety of relevant sources.[1]
Such as The behavioral selection model for predicting AI motivations, Fitness-Seekers: Generalizing the Reward-Seeking Threat Model, Alignment remains a hard, unsolved problem, the training game bit in Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover, Why study proto-training gaming as an adversarial alignment failure mode?, Alignment Engineering vs. Misalignment Science