It's a bit funny, but it seems to have gone for a simple and straightforwardly positive story, which is not what I would generally expect from you.

They can't access the computations that led to the previous messages in the context and so can only guess at why they wrote what they previously wrote.

This is not exactly right. The internal state in LLMs is the attention keys and values (per token, layer and attention head). Using an LLM to generate text involves running the context (prior user and model messages, in a chat context) through the model in parallel to fill the K/V cache, then running it serially on one token at a time at the end of the sequence, with access to the K/V cache of previous tokens, appending the newly generated keys and values to the cache as you go.

This internal state is fully determined by the input - K/V caching is purely an inference optimization and (up to numerical issues) you would get exactly the same results if you recomputed everything on each new token - so there is exactly as much continuity between messages as there is between individual tokens (with current publicly disclosed algorithms).

Student: I wish I could find a copy of one of those AIs that will actually expose to you the human-psychology models they learned to predict exactly what humans would say next, instead of telling us only things about ourselves that they predict we're comfortable hearing. I wish I could ask it what the hell people were thinking back then.

TA: You'd delete your copy after two minutes.

Apparently roughly this dynamic has happened in ChatGPT. Exciting*. https://x.com/MParakhin/status/1916533763560911169

We probably use a mix of strategies. Certainly people take "delve" and "tapestry" as LLM signals these days.

Average humans can't distinguish LLM writing from human writing, presumably through lack of exposure and not trying (https://arxiv.org/abs/2502.12150 shows that it is not an extremely hard problem). We are much more Online than average.

Abundance elsewhere: Human-legible resources exist in vastly greater quantities outside Earth (asteroid belt, outer planets, solar energy in space) making competition inefficient

It's harder to get those (starting from Earth) than things on Earth, though.

Intelligence-dependent values: Higher intelligence typically values different resource classes - just as humans value internet memes (thank god for nooscope.osmarks.net), money, and love while bacteria "value" carbon

Satisfying higher-level values has historically required us to do vast amounts of farming and strip-mining and other resource extraction.

Synthesis efficiency: Advanced synthesis or alternative acquisition methods would likely require less energy than competing with humans for existing supplies

It is barely "competition" for an ASI to take human resources. This does not seem plausible for bulk mass-energy.

Negotiated disinterest: Humans have incentives to abandon interest in overlap resources:

Right, but we still need lots of things the ASI also probably wants.

ASI utilizing resources humans don't value highly (such as the classic zettaflop-scale hyperwaffles, non-Euclidean eigenvalue lubbywubs, recursive metaquine instantiations, and probability-foam negentropics) One-way value flows: Economic value flowing into ASI systems likely never returns to human markets in recognizable form

If it also values human-legible resources, this seems to posit those flowing to the ASI and never returning, which does not actually seem good for us or the same thing as effective isolation.