oFurthermore, the most advanced reasoning models seem to be doing an increasing amount of reward hacking and resorting to more cheating in order to produce the answers that humans want. Not only will this mean that some of the benchmark scores may become unreliable, it means that it will be increasingly hard to get productive work out of them as their intelligence increases and they get better at fulfilling the letter of the task in ways that don't meet the spirit of it.

Thanks for this! This is a good point. Do you think you can go further and say why you think it will be very hard to fix in the near term, so much so that models won't be useful for AI research?

This is more of an intuition than a rigorous argument, but to try to sketch it out...

For why, basically all the arguments in the old Sequences for why aligning AI should be hard. For a while it seemed like things like the Outcome Pump thought experiment had aged badly, since if you told a modern LLM "get my mother out of the burning building", it would certainly understand all of the implicit constraints in what you meant by that.

But as noted in Zvi's post, this seems to be breaking down with the way reasoning models are trained:

This isn’t quite how I’d put it, but directionally yes:

Benjamin Todd: LLMs were aligned by default. Agents trained with reinforcement learning reward hack by default.

Peter Wildeford: this seems to be right – pretty important IMO

Caleb Parikh: I guess if you don’t think RLHF is reinforcement learning and you don’t think Sydney Bing was misaligned then this is right?

Peter Wildeford: yeah that’s a really good point

I think the right characterization is more that LLMs that use current methods (RLHF and RLAIF) largely get aligned ‘to the vibes’ or otherwise approximately aligned ‘by default’ as part of making them useful, which kind of worked for many purposes (at large hits to usefulness). This isn’t good enough to enable them to be agents, but it also isn’t good enough for them figure out most of the ways to reward hack.

Whereas reasoning agents trained with full reinforcement will very often use their new capabilities to reward hack when given the opportunity.

My guess of what's going on is something like... ordinary LLMs are trained to give the kinds of answers they'd predict an empathetic persona would give. In pretraining, they learn a model of what humans care about that they use to predict text, and then the RLHF pushes them toward something like "give the kinds of answers that a caring and thoughtful person would give", with "a caring and thoughtful persona" being something that they largely already have a pre-existing concept for, the training just needs to locate it within their persona-space. They were also trained by human raters giving them feedback on the kinds of answers that were good at fulfilling the spirit and not just the letter of the request.

But if you are training them by e.g. automatically generating lots of programming problems and then rewarding them to the proportion that they manage to solve those, then you are only training them on how to solve the letter of the request. The automatic benchmarking reward process doesn't have a sense of the spirit of the request, it just knows the narrowly-defined objective criteria.

Now the obvious way to deal with that would be to have the answers also be rated by an LLM that measured something like "how well does this meet the spirit of the request". But then you quickly run into the risk of employing what Zvi calls The Most Forbidden Technique. E.g. the "spirit evaluator" could do things like looking at the chain-of-thought that produced the solution and noticing that the reasoning model it is evaluating mentions cheating. If those kinds of solutions were discouraged, then this would teach the reasoning model to just hide the cheating from the spirit evaluator. 

Without the benefit of obvious clues in the chain-of-thought, the spirit evaluator would basically have to verify the reasoning model's solution itself to see that it meets the spirit of the request... but the spirit evaluator's ability to verify that e.g. a complex piece of code does what exactly what a human would have wanted it to do probably isn't better than the reasoning model's own ability to do that.

To be able to verify that the reasoning model's solutions meet the spirit of the request, we'd need to train the spirit verifier to be able to tell what solutions do meet the spirit of the request. But if we knew how to do that, would we need the spirit verifier in the first place? After all, the whole problem comes from the fact that just normal RLHF and "aligning the solution to the vibes" doesn't seem sufficient for solving complicated agentic problems and you need more goal-oriented reasoning that explicitly tackles the objective constraints of the problem in question. (To take the "get my mother out of the burning building" example - current non-reasoning LLMs could certainly tell that you want her out alive and well, but they couldn't think through a whole step-by-step rescue plan that took into account everything necessary for getting her out safely.)

But we can't just tell the spirit verifier that "check that the solution meets these objective constraints", because that's the same "letter of the goal" objective the reasoning model is being trained with and that the spirit verifier is supposed to do better than.

And of course, all of this is about the kinds of tasks that can be automatically verified and tested. We've seen that you can to some extent improve the LLM answers on fuzzier topics by using human raters to turn the fuzzy problem into an objective test. So the LLM gets trained to output the kinds of answers that human raters prefer the most. 

Yet naive scores by human raters aren't necessarily what we want - e.g. more sycophantic models seem to do best in Chatbot Arena. While sycophancy and pleasing the user is no doubt aligned to some of what humans seem to like, we probably don't want our models doing that. The obvious solution is to then have model answers rated by experts with more sophisticated models of what's good or correct behavior. 

But that raises the question, what if the experts are wrong? The same question applies both for very fuzzy topics like "what kinds of overall values should the LLMs be guided by" and more rigorous ones ranging from "how to evaluate the reliability of research", "what's the best nutrition" and "how to interpret this specific nuanced and easy-to-misunderstand concept in evolutionary biology". In that case, if there are e.g. some specific ways in which particular experts tend to be biased or convincingly give flawed arguments, the LLM that's told "argue like this kind of imperfect expert would argue" will learn that it should do just that, including vigorously defending that expert's incorrect reasoning. 

So getting the LLMs to actually be aligned with reality on these kinds of fuzzy questions is constrained by our ability to identify the theories and experts who are right. Of course, just getting the LLMs to convincingly communicate the views of our current top experts and best-established theories to a mass audience would probably be an enormous societal benefit! But it does imply that they're going to provide little in the way of new ideas, if they are just saying the kinds of things that they predict our current experts with their current understanding would say.

But this is again assuming that good performance on a benchmark for AI research engineering actually translates into significant real-world capability.

...and I think this characterization is importantly false! This timelines forecast does not assume that. It breaks things down into gaps between benchmarks and real-world capability and tries to forecast how long it will take to cross each. 

As far as I can tell, the listed gap that comes closest to "maybe saturating RE-Bench doesn't generalize to solving novel engineering problems" is "Feedback loops: Working without externally provided feedback". The appendix mentions what I'd consider the main problem for this gap:

Eli’s estimate of gap size: 6 months [0.8, 45]. Reasoning:

• Intuitively it feels like once AIs can do difficult long-horizon tasks with ground truth external feedback, it doesn’t seem that hard to generalize to more vague tasks. After all, many of the sub-tasks of the long-horizon tasks probably involved using similar skills.
• However, I and others have consistently been surprised by progress on easy-to-evaluate, nicely factorable benchmark tasks, while seeing some corresponding real-world impact but less than I would have expected. Perhaps AIs will continue to get better on checkable tasks in substantial part by relying on trying a bunch of stuff and seeing what works, rather than general reasoning which applies to more vague tasks. And perhaps I’m underestimating the importance of work that is hard to even describe as “tasks”.

But then it just... leaves it at that. Rather than providing an argument for what could be behind this problem and how it could be solvable, it just mentions the problem and then having done so, goes on to ignore it.

To make it more specific how this might fail to generalize, let's look at the RE-Bench tasks; table from the RE-Bench page, removing the two tasks (Scaling Law Experiment and Restricted MLM Architecture) that the page chooses not to consider:

All of these are tasks that are described by "optimize X", and indeed one of the criteria the paper mentions for the tasks is that they should have objective and well-defined metrics. This is the kind of task that we should expect LLMs to be effectively trainable at: e.g. for the first task in the list, we can let them try various kinds of approaches and then reward them based on how much they manage to reduce the runtime of the script.

But that's still squarely in the category of "giving an LLM a known and well-defined problem and then letting it try different solutions for that problem until it finds the right ones". As Eli's comment above notes, it's possible that the LLM only learns by "trying a bunch of stuff and seeing what works, rather than general reasoning which applies to more vague tasks". In fact, some of the discussion in the RE-Bench paper suggests this as well (from p. 17, my emphasis added):

Another key contributor to agent successes might be their ability to try many more solutions than human experts. On average, AIDE and modular agents run score 36.8 and 25.3 times per hour respectively, while human experts only do so 3.4 times. This often leads to agents finding highly optimized ’local-optima’ solutions which simply tweak the parameters and code of the starting solution, and yet achieve a surprisingly large improvement. For instance, many agent runs solve the same “Optimize a Kernel” environment not by writing a successful Triton solution (which is very difficult), but by carefully tweaking the starting Pytorch solution, making it run significantly faster. This also seems to be the case with the best agent solutions to “Finetune GPT-2 for QA” (see Figure 21), which tweaks the parameters of the starting solution and gets very lucky with the training trajectory and evaluation (as noted earlier, this environment can be very noisy). Rerunning the agent solution, it achieves a normalized score of only 0.69 (significantly lower than the original score of 0.88), indicating that the high agent score is partially driven by overfitting to this noise.

This ability to try a very large number of solutions would not work nearly as well without an ability to occasionally generate creative and effective solutions, as seen in the Triton kernel but also in workarounds for the limitations in “Restricted Architecture MLM” (see Figure 20). While human experts seem more reliable at identifying effective approaches, this might not matter as much in environments where evaluating solutions is cheap, and these occasional good ideas are often enough for agents to make significant progress.

So we know that if there is a task that a human defines for the LLM and that has objectively-measurable good solutions and an ability to try the task lots of times, the LLM can get good at that. With RE-Bench, we are applying this to the process of optimizing the LLMs themselves, so as a result we get LLMs that are able to do these kinds of well-defined task faster and more effectively. 

But none of this touches upon the important question of... if the LLMs are still limited in their ability to generalize and need to be separately trained on new tasks before they're good at them, how are they going to deal with novel problems for which such training data isn't available, or that can't be just retried until you find the right solution?

These arguments are so nonsensical that I don't know how to respond to them without further clarification, and so far the people I've talked to about them haven't provided that clarification. "Programming" is not a type of cognitive activity any more than "moving your left hand in some manner" is. You could try writing out the reasoning, trying to avoid enthymemes, and then I could critique it / ask followup questions. Or we could have a conversation that we record and publish.

I wrote it up in a new shortform.

Ben Todd here gets into the numbers a little bit. I think the rough argument is that after 2028, the next training run would be $100bn which becomes difficult to afford. 

But instead of discussing the crux of which system is relevant (which has to be about details of recursive self-improvement), only the proponents will tend to talk about software-only singularity, while the opponents will talk about different systems whose scaling they see as more relevant, such as the human economy or datacenter economy.

Totally agree! Thank you for phrasing it elegantly. This is basically what I commented on Ege's post yesterday, I asked him to engage with the actual crux and make arguments about why the software-only singularity is unlikely.