More speculative thoughts:

• Perhaps we end up in a regime where this is possible, but expensive enough that we can only afford a bit of it.
• If that's the case, the more sample-efficient the RL is, the better off we are — since as the total number of RL rewards we need decreases, we can afford to invest more in the quality of each one.

Even more speculative thoughts:

• As an extreme example, you could imagine an AI that's as sample-efficient as a human child. Suppose an AI company had a virtual copy of von Neumann's brain age 5, which they could run at 10x speed, and had to raise him to become an intent-aligned adult.
• The resources at their disposal would in principle enable a lot of oversight and thoughtful parenting and teaching, so in principle they could afford to do pretty well at this task for a while. It's like a school with a 100-1 teacher-student ratio. They might fail in practice, of course.
• Once his outputs become sufficiently hard to check, they might have a lot more difficulty, e.g. maybe it's hard to tell whether his advice on the Manhattan Project is subtly wrong in a way designed to sabotage the project.

Changing selection pressures to align with intended behaviors: This might involve making training objectives more robust, iterating against held-out evaluation signals, or trying to overwrite the AI's motivations at the end of training with high-quality, aligned training data.

Is another broad direction for this increasing the intelligence of the reward models? A fun hypothetical I've heard is to imagine replacing the reward model used during training with Redwood Research (the organization).

So we imagine that during training, whenever an RL reward must be given or not given, a virtual copy of all Redwood Research employees is spun up on the training servers and has one subjective week to decide whether/how to give the RL reward, with their existing knowledge, access to the transcript of the model completing the task, access to data from previous such decisions, ability to use whatever internals-based probes are available, etc.

My instinct is this would help a lot.

To the extent trusted or controlled AI systems can approximate this thought experiment (or do even better!), that seems great for safety and we should use them to do so. People use the term "automated alignment research" but this sort of this seems like an example of automated safety work that isn't "research".

[cross-posted from EAF]

Thanks for writing this!!

This risk seems equal or greater to me than AI takeover risk. Historically the EA & AIS communities focused more on misalignment, but I'm not sure if that choice has held up.

Come 2027, I'd love for it to be the case that an order of magnitude more people are usefully working on this risk. I think it will be rough going for the first 50 people in this area; I expect there's a bunch more clarificatory and scoping work to do; this is uncharted territory. We need some pioneers.

People with plans in this area should feel free to apply for career transition funding from my team at Coefficient (fka Open Phil) if they think that would be helpful to them.

Thanks Tom! Appreciate the clear response. This feels like it significantly limits how much I update on the model.

We simulate AI progress after the deployment of ASARA. 

We assume that half of recent AI progress comes from using more compute in AI development and the other half comes from improved software. (“Software” here refers to AI algorithms, data, fine-tuning, scaffolding, inference-time techniques like o1 — all the sources of AI progress other than additional compute.) We assume compute is constant and only simulate software progress.

We assume that software progress is driven by two inputs: 1) cognitive labour for designing better AI algorithms, and 2) compute for experiments to test new algorithms. Compute for experiments is assumed to be constant. Cognitive labour is proportional to the level of software, reflecting the fact AI has automated AI research.

Your definition of software includes all data, which strikes me as an unusual use of the term so I'll put it in scare quotes.

You say half of recent AI progress came from "software" and half comes from compute. Then in your diagram, the cognitive labor gained from better AI is going to improve "software."

To me it seems like a ton of recent AI progress was from using up a data overhang, in the sense of scaling up compute enough to take advantage of an existing wealth of data (the internet, most or all books, etc.)

I don't see how more AI researchers, automated or not, could find more of this data. The model has their cognitive labor being used to increase "software." Does the model assume that they are finding or generating more of this data, in addition to doing R&D for new algorithms, or other "software" bucket activities?

These methods may be too aggressive. Before we have ASARA, less capable AI systems may still accelerate software progress by a more moderate amount, plucking the low-hanging fruit. As a result, ASARA has less impact than we might naively have anticipated.

I'm confused.

My default assumption is that prior to ASARA, less-capable AIs will have accelerated software progress a lot — so I'm interested in working that into the model.

It looks like your "gradual boost" section is for people like me; you simulate the gradual emergence of the ASARA boost over a period of five years. But in the gradual boost section, you conclude that using this model results in a higher chance of >10yrs being compressed into one year. (I'm not currently following the logic there, just treating it as a black box.)

Why is the sentence "As a result, ASARA has less impact than we might naively have anticipated" then true? It seems this consideration actually ends up meaning it has more impact.

Just wanted to say I really enjoyed this post, especially your statement of the problem in the last paragraph.

The guy next to me, who introduced himself as "Blake, Series B, stealth mode,"

I don't think it makes sense to have a startup which is in stealth mode, but is also raising Series B (a later round of funding for scaling once you've found a proven business model).

Thanks for reply!

When I say future updates I'm referring to stuff like the EM fintetuning you do in the paper; I interpreted your hypothesis as being that for inoculated models, updates from the EM finetuning are in some sense less "global" and more "local".

Maybe that's a more specific hypothesis than what you intended, though.