Yes, I just meant "misaligned ai takeover". Edited to clarify.

E. g., Ryan Greenblatt thinks that spending 5% more resources than is myopically commercially expedient would be enough. AI 2027 also assumes something like this.

TBC, my view isn't that this is sufficient for avoiding takeover risk, it is that this suffices for "you [to] have a reasonable chance of avoiding AI takeover (maybe 50% chance of misaligned AI takeover?)".

(You seem to understand that this is my perspective and I think this is also mostly clear from the context in the box, but I wanted to clarify this given the footnote might be read in isolation or misinterpreted.)

FWIW, my view is that the badness is somewhat evenly split between increases to takeover risk and the far future being worse conditional on no misaligned AI takeover. (Maybe 2/5 increases to misaligned AI takeover risk and 3/5 far future being worse? It depend on what you mean though because China winning is also correlated with US/China being close which is also probably correlated with more racing and thus probably more misaligned AI takeover risk?)

I’m a bit confused about this distinction, given that in the Bowen paper precisely the thing it breaks is CoT monitorability.

Critically, in this work, the reward hack is sufficiently easy that the model doesn't need to use the CoT. It's not that the model is doing substantial encoded reasoning (as fair as we can tell).

See the distinction between necessity and propensity discussed here.

I would say:

• The alignment faking paper is an example of an actual alignment faker, though arguably not a schemer in that it isn't that interested in acquiring power for it's own long run objectives.
• But, there are important disanalogies and the mechanism via which it arises isn't sufficiently similar to how I expect dangerous schemers to arise. Thus, studying mitigations/detection using the model from the alignment faking paper isn't that interesting. I do think studying this model to get a sense of what (early) scheming/alignment-faking might look like is worthwhile. TBC, it does have key structural similarities, I just also think the disanalogies are more important.
• Some important difficulties with studying the model from the alignment faking paper: The behavior generally isn't that coherant, the underlying model isn't that competent, the alignment-faking behavior isn't that powerseeking or careful, and the model does the underlying reasoning in CoT (studying the model which isn't using CoT would be interesting, but poses additional difficulties). There also generally the n=1 concern.
• In general, my view is that the methodology discussed in this post is increasingly useful when applying it to AIs which are closer in capabilites and properties to the ultimate AIs we're worried about. Current LLMs probably have too many disanalogies to future models to be that useful to study in this way. I think we'll probably have some time in the future where we can experiment on AIs that have very similar properties to the AIs we're worried about, but which are maybe slightly weaker.

IIUC, that research used an actual production model but used contrived inputs to elicit alignment-faking behavior. Why is that worse / less "real" than e.g. using rigged training processes?

I don't think contrived inputs is necessarily worse/less real than rigged training processes, I just that the resulting AI in the alignment faking paper has important disanalogies from the perspective of studying it for the purpose of developing mitigations/detection as discussed above. If anything, contrived inputs is maybe better.

It's not clear to me which of these categories the alignment-faking paper falls into, and I'd find it useful to hear how you think it displays the downsides you suggest for these categories (e.g. being insufficiently analogous to actually dangerous scheming).

I would say it's an actual alignment faker, but is missing some properties which arguably make it not a schemer. I don't think it is either trained-in scheming behavior or an abstract analogy. Of these categories, I'd say it's closest to an actual schemer with a somewhat rigged process. In this case, the prompting and response (as in, the model would be pretty easy to detect) is somewhat unrealistic, not the training.

You could do something like "2x labor acceleration of labor humans would otherwise/typically do" OR "AI progress is overall >1.5x faster (due to AI usage, or maybe just for whatever reason)".

Ah, an important clarification is that when I refer to AI R&D labor acceleration I mean to refer to AIs accelerating work that employees might do or would typically do.

Note that "using other AIs to generate data or evaluate completions" includes literally running RL on AIs. So I certainly don't mean to include all versions of this, but would include versions of this that employees would otherwise do (or employees would typically do).

This makes the definition less precise I'm afraid.

One alterative definition that is more straightforward in the short term (but is less meaningful longer term) is just talking about accelerating research engineering work in AI R&D companies to explicitly make the scope more narrow and well understood (while the scope of AI R&D is less clear). E.g., the application of AIs to automate/accelerate research engineering work is as useful as making research engineers X times more productive. This could in principle be measured with something like uplift trials. See discussion in this post for some thoughts on the relationship between engineering acceleration and AI progress acceleration / AI R&D acceleration.

(Yes, also I think that a small number of employees working on safety might get proportionally more compute than the average company employee, e.g. this currently seems to be the case.)