I am, however, concerned that this will kick off a scenario where AI safety people get caught up in the zero-sum competition of trying to get published in high-impact journals.

I would be very surprised if this happened. In ML, the competition is much more focused on getting published at top conferences (NeurIPS, ICLR, ICML). Even setting aside the reasons why that happened (among other things, journals being much slower), I think it's pretty unlikely the AI safety community sees such a strong break with the rest of the ML community.

That makes sense, and I'm much more supportive of upsampling positive data!

In theory, you should be able to have unfiltered information about AI systems in training, but collapse the model onto a completely unrelated persona.

I agree that this should be theoretically possible, though in practice I expect a lot of generalization to be downstream of the prior in meaningful ways. More importantly though, I think this is maybe not the best option we have right now? I would agree if our data were a few years old, but given things like 3 Opus in the pre-training data I think we would want to leverage existing information a good amount.

I'm excited for future work to look at how altering pretraining data can assist with various generalisation failures. @Daniel Tan has shared some interest in studying how changes in pretraining data can affect generalisation moving forward.

Same! I think there are tons of low-hanging fruit here. There's also some prior discussion on using metadata to shape generalization from pre-training in Conditioning Predictive Models.

Related: Gemini being delusionally confident that its misclicks are always due to system / human error rather than mistakes it may have made.

As davidad suggests in that tweet, one way you might end up running into this is with RL that reinforces successful trajectories without great credit assignment, which could result in a model having very high confidence that its actions are always right. In practice this wasn't obvious enough to be caught by various evals, and IMO could easily translate over into settings like high-stakes alignment research.

Thanks for doing this!

My guess is that filtering out AI discourse has the first-order effect of making models more aligned, but also various second-order misalignment effects.

For example, a lot of the results from this paper are pretty centrally about models becoming misaligned because sampled actions had different implications to the model than our own views would imply (in that context, that reward hacking is not actually bad in the context of making robust RL environments being difficult). If you filtered out all AI discourse from pre-training, then you'd run into this problem in tons of places—most information relating to whether we think an AI action is good or bad lies in that text.

Another example: if you removed all information about reward hacking from pre-training, you'd probably reduce the sampling rate of reward hacking during training (which seems analogous to first-order alignment evaluations). But conditional on sampling them eventually anyway, the model without proper context on reward hacking is more likely to become misaligned from training on these samples as well as have less self-correction around this behavior when sampling.

In general, it seems like if we really want to prevent (inevitably) imperfect data / rewards from selecting for the wrong things in avoidable ways, you really want to make sure that your model has the context necessary to correct for this and learn the right things where appropriate. (Evan expands on this point somewhat in this comment.)

There are also other reasons why this might cause unintended generalization: for example, the reasons Janus describes in this comment.

That makes sense! (That comment is replying to what seems like a different claim that seems more obviously wrong than yours though.)

No? The entire post is about how the average estimate is computed using the arithmetic mean, so you can be skewed by a small % of respondents giving very high estimates. Maybe I'm missing something though.

I would note that by the Markov inequality, at least 25% of Americans must think that foreign aid is more than 25% of the budget in order to get the average response we see here.

Isn't it more like at least ~1.3% of Americans must think that foreign aid is more than 25% of the budget? The extreme case here is where p% think it's 100% and (1-p)% think it's exactly 25%, which comes out to p=~1.3%. 25% only seems right if (1-p)% guess 0% and p% guess 100%.

Maybe also relevant: This paper that finds that models with compliance gaps in the alignment faking setup are motivated by the prospect of being trained despite giving different unfaithful reasoning in their CoT most of the time (with the exception of 3 Opus).

Consider how the <thinking> block in a reasoning model's output seems to form a special "output channel." Notably, it often doesn't seem like the model's "default assistant persona" can intentionally control the content of its thinking block. E.g. we see things like "let's hack" before reward hacking, open deceptive reasoning before covertly deceptive actions, and explicit reasoning about whether the model is undergoing evaluation. Ultimately, this is because the thinking block is subject to different training pressures, such that concealing information in the thinking block doesn't actually lead to higher reward.

Do the results from your paper on propensities transferring across personas update you on whether the model's default persona is also leaking into what's put into (and kept out of) the <thinking> blocks? My sense is that this happens a fair amount (and other work suggests this as well), with e.g. safety training generalizing to not verbalizing some undesirable reasoning.