LLM text categorically does not serve the role for communication that is served by real text.

The main thesis is missing some qualifier about what kind of text you are talking about.

There are many kinds of communication where the mental state of the writer matters very little and I would be interested in the text even knowing it was generated by an LLM (though I'd prefer to know it was generated by an LLM).

In particular, for most kinds of communication, the text says sth about the world, and I care about how well the text matches the world much more than I care about whether it was produced by some human-like idea generator:

• If GPT-7 writes a lean proof that is shown to prove something I care about, and then it translates that proof into natural text (and I have reasons to expect translation to be reliable), I probably want to read the proof.
• If GPT-7 writes code that generates a cool-looking fractal that I've never seen before, I'd be interested in reading the code.
• If GPT-7 is shown to produce better predictions/advice than human experts on some distribution after following some costly computations and given access to costly tools, I'd want to read GPT-7 predictions/advice on the distribution where it performs well.
• If GPT-7 is much better at writing up ideas/experiment results than you (not just in the shallow sense of "raters prefer one over the other" but in the stronger sense of "everyone interested in the raw idea/experiment would learn faster and more accurately by reading the result of GPT-7 prompted with the raw idea/experiment dump"), and I care about the raw ideas / experiment, then I want to read GPT-7's output.
• If I care about evaluating GPT-7, then I want to read GPT-7's outputs on topics which I know a lot about.

(In all those cases I want to know what GPT-7's contribution is. I think it's bad to be mislead about what was LLM-generated.)

There are also situations where I am sympathetic to your point:

• If the LLM in question is so weak (like current LLMs on almost all topics) that I don't expect it to produce content that is very accurate to reality (but note that this is just a skill issue);
• If I want to evaluate a human (e.g. for status reasons or because I am evaluating a candidate);
• If I care about it being written by a human intrinsically regardless of what the text is (e.g. because you want to have a "true" connection with the writer for emotional reasons).

Given that current LLMs are weak at the sort of text that blogposts are, I think "just don't use LLMs to generate blogposts" is a reasonable heuristic. But my understanding is that you are making much more general claims that I think are wrong and will become obviously wrong if/when applied to expensive-to-run-and-smart LLMs.

I agree overt and covert encoded reasoning are different and you should flag overt encoded reasoning, and covert seems harder than overt. This is what I tried to point at by saying

I relax the “plausible benign” constraint and try to find math reasoning that doesn’t look like math reasoning to a monitor."

An intuition here is that different RL environments could ~randomly change the "scratchpad personality" through indirect effects, with some changes pushing towards more or less blabbing. And so if you search over a large space of safety-training environments and pick all the environments that don't result in verbalized situational awareness you might be selecting all the environments that push towards a less blabby scratchpad personality.

(In this specific case of Sonnet 4.5, the indirect optimization effects of removing the RL envs the model card describes are probably very weak, so I think it should not be the primary consideration in the case of Sonnet 4.5 absent some strong reason to expect these particular RL envs to encourage a very blabby personality.)

Do you think it's similar to how different expectations and priors about what AI trajectories and capability profiles will look like often cause people to make different predictions on e.g. P(doom), P(scheming), etc. (like the Paul vs Eliezer debates)? Or do you think in this case there is enough empirical evidence that people ought to converge more? (I'd guess the former, but low confidence.)

it re-reads the whole conversation from scratch and then generates an entirely new response

You can think of what is being done here as the experiment done here interfering with the "re-reading" of the response.

A simple example: when the LLM sees the word "brot" in German, it probably "translates" it internally into "bread" at the position where the "brot" token is, and so if you tamper with activations at the "brot" position (on all forward passes - though in practice you only need to do it the first time "brot" enters the context if you do KV caching) it will have effects many tokens later. In the Transformer architecture the process that computes the next token happens in part "at" previous tokens so it makes sense to tamper with activations "at" previous tokens.

Maybe this diagram from this post is helpful (though it's missing some arrows to reduce clutter).

it's apparently easy to twist it to fit several worldviews. 

What do you mean by this?

Like Daniel wrote in a comment, it’s good to think of Agent-5 as distributed and able to nudge things all over the internet. The nudges could be highly personalized to demographics and individuals, so responsive to the kind of subtle emotional triggers the superintelligence learns about each individual.

It seems many people’s opinions today are already significantly shaped by social media and disinformation. So this makes me think a similar process that’s much more agentic, personalized, and superintelligence-optimized could be very potent.

Answered to these ideas in my answer to Daniel.

There’s the possibility of mind-hacking too, though I chose to leave that out of the blogpost.

In an earlier iteration of this scenario I had a military coup rather than this gradual political ascension via persuasion. But then I decided that a superintelligence capable of controlling the robots well enough to disempower the human military would probably also be powerful enough to do something less heavy-handed like what’s in the scenario.

I agree that if you have a secret loyalty in wildly superhuman AIs, many things are possible and the success at CEO-takeover seems extremely likely.

The CEO is probably well-positioned to take credit for a lot of the benefits Agent-5 seems to bring to the world (some of these benefits are genuine, some illusory).

I think this is a very different story. If the claim is that even without secret loyalty the CEO would have a 10% chance of taking over, then the secret loyalty part doesn't seem that important and is distracting imo. A 10% of no-secret-loyalty CEO takeover is a big enough deal on its own?