I agree overt and covert encoded reasoning are different and you should flag overt encoded reasoning, and covert seems harder than overt. This is what I tried to point at by saying

I relax the “plausible benign” constraint and try to find math reasoning that doesn’t look like math reasoning to a monitor."

An intuition here is that different RL environments could ~randomly change the "scratchpad personality" through indirect effects, with some changes pushing towards more or less blabbing. And so if you search over a large space of safety-training environments and pick all the environments that don't result in verbalized situational awareness you might be selecting all the environments that push towards a less blabby scratchpad personality.

(In this specific case of Sonnet 4.5, the indirect optimization effects of removing the RL envs the model card describes are probably very weak, so I think it should not be the primary consideration in the case of Sonnet 4.5 absent some strong reason to expect these particular RL envs to encourage a very blabby personality.)

Do you think it's similar to how different expectations and priors about what AI trajectories and capability profiles will look like often cause people to make different predictions on e.g. P(doom), P(scheming), etc. (like the Paul vs Eliezer debates)? Or do you think in this case there is enough empirical evidence that people ought to converge more? (I'd guess the former, but low confidence.)

it re-reads the whole conversation from scratch and then generates an entirely new response

You can think of what is being done here as the experiment done here interfering with the "re-reading" of the response.

A simple example: when the LLM sees the word "brot" in German, it probably "translates" it internally into "bread" at the position where the "brot" token is, and so if you tamper with activations at the "brot" position (on all forward passes - though in practice you only need to do it the first time "brot" enters the context if you do KV caching) it will have effects many tokens later. In the Transformer architecture the process that computes the next token happens in part "at" previous tokens so it makes sense to tamper with activations "at" previous tokens.

Maybe this diagram from this post is helpful (though it's missing some arrows to reduce clutter).

it's apparently easy to twist it to fit several worldviews. 

What do you mean by this?

Like Daniel wrote in a comment, it’s good to think of Agent-5 as distributed and able to nudge things all over the internet. The nudges could be highly personalized to demographics and individuals, so responsive to the kind of subtle emotional triggers the superintelligence learns about each individual.

It seems many people’s opinions today are already significantly shaped by social media and disinformation. So this makes me think a similar process that’s much more agentic, personalized, and superintelligence-optimized could be very potent.

Answered to these ideas in my answer to Daniel.

There’s the possibility of mind-hacking too, though I chose to leave that out of the blogpost.

In an earlier iteration of this scenario I had a military coup rather than this gradual political ascension via persuasion. But then I decided that a superintelligence capable of controlling the robots well enough to disempower the human military would probably also be powerful enough to do something less heavy-handed like what’s in the scenario.

I agree that if you have a secret loyalty in wildly superhuman AIs, many things are possible and the success at CEO-takeover seems extremely likely.

The CEO is probably well-positioned to take credit for a lot of the benefits Agent-5 seems to bring to the world (some of these benefits are genuine, some illusory).

I think this is a very different story. If the claim is that even without secret loyalty the CEO would have a 10% chance of taking over, then the secret loyalty part doesn't seem that important and is distracting imo. A 10% of no-secret-loyalty CEO takeover is a big enough deal on its own?

I think this seems more dicey for the AI if it's not vastly superhuman at politics. There are forces pushing for many different people to be in power quite hard, so I think it takes somewhat forceful actions to move probabilities by a Bayes factor above 10, which makes me skeptical that something like the subtle actions in the essay would have the intended effect.

I would guess the AI's subtle policy actions would not be much more coordinated and high impact position than what TikTok could have done a few years ago, and I don't think TikTok was ever in a position where it could have put a much more pro-CCP candidate in power if it wanted to (if the pro-CCP candidate started from a similar position of strength as the CEO) - my guess is that it's hard to massively shift the balance of power without it being obvious to the people losing that something nefarious is going on.

My guess is that the AI could try to have a massive cover-up operation to discredit people who criticize the somewhat obvious media manipulation - but then the thing the AI strategy is more like "somewhat blatantly do bad things and then do a giant cover-up" rather than "take actions so subtle nobody would ever figure it out even if they had access to all the logs".