If we think of the presented "thoughts" in CoT as a bottleneck in otherwise much wider bandwidth models I think things become clearer. If we're not watching, the purpose is pretty straightforward: provide as much useful information to the next iteration. There is loss when compacting the subtleties of the state of the model into a bunch of tokens. The better able the model is at preserving the useful calculations in the next round, the more likely it is to be successful as it is discarding less of the output from the processing applied.

So yeah, it's about "efficiency". IF our only metric is success in solving the problem the fine tuning is goign to start to pull the CoT output away from the pre-training coherent language. The CoT becomes another layer. If we demand it be coherent we essentially hobble it to an extent, but have two problems. 

Firstly we are dragging in two directions: solution quality and the language used in the CoT. That means the best words the model can use are ones that keep us happy but don't pull it too far from the actual representation that would be optimal for the solution. That sounds reasonable, almost like an explanation, but there is no guarantee that the actual pattern it is using is as harmless as the CoT would have us think. We might be seeing euphemisms, or just words that fit the bill but still pack enough punch in the next iteration to allow the system to continue on whatever chain it was on: nudge, nudge, wink, wink. 

The second problem, which exacerbates the first, is how little bandwidth is used up by the meaning we take from the CoT text. Every piece of punctuation, every piece of white space, every overblown superlative, every cliche is an option that provides information to the next iteration. And we don't even notice. Example: word choice:
- I need to make a firm plan
- I need to make a robust plan...or sound plan, or reliable, or reasonable, or accurate, or clear, or smart...

That's 3 bits of information. Combine it with other options and you rapidly build up a decent channel. And those options are myriad. EVERYTHING is information. The count of words; the frequency of certain tokens and the pattern of their repetition; the number of tokens between punctuation, between line breaks; the patterns between the patterns between the patterns. When you have billions of parameters the depth of the patterns is beyond any human comprehension.

I'm not sure we need to invent (or appropriate) concepts here (things like parasitic entities, dyads, etc). In fact it actually feels like doing so is joining the same as posting the messages quoted, the only difference being the perception of polarity. In truth I find it a bit concerning, alongside the replies, as it has a touch of superstition about it, what with the naming of phenomena, etc. 

IMO, what this is is a very interesting example of an emergent colonial/memetic organism. Sort of like an egregore. 

This is how I see things working. I think we can broaden the concept of the selfish gene to what the gene represents, which is an information pattern, a deep entropic whirlpool. Not just a sinkhole; when we get one of these information patterns in the right form and in the right environment, it spins itself into self-sustaining. 

I'm not great on thermodynamics, but we might say the sun is a local entropy reduction engine which required a bit of luck to spin up a deep entropy whirlpool on our planet. We are witnessing how deep that goes. Organic chemicals, lifeforms, DNA, biomes, cultures, nations, religions, markets, memes, jokes, fashions are categories of patterns that have been enabled by local entropy reduction engines, in turn powered by the ball of fire in the sky producing energy gradients. 

But the outputs from these engines have limits, they are finite. It is not abundance. The sun spins off complexity in a constrained reality that has edges, and so we have pressure that brings redness in tooth and claw. It's not clear to me just how spot on the sun's effect is on spinning up that entropy whirlpool. But so far all evidence is that it hit a very sweet spot (yet I guess we might only be on the first step of a marathon, who knows?). 

So the internet was a recent manifestation of the self reinforcing pattern of the entropy whirlpool the sun has created. It made possible levels of complexity we had not yet experienced. We got cate meme information patterns,  which would have been impossible before. 

And now we have AI, which is just armed to the teeth with optimisation, and when combined with humans has the ability to reinforce even the weakest of information patterns. For an aspiring information pattern the internet was the major league, but AI is the holy grail. We shouldn't be shocked by this sort of thing. We have created a turbocharged engine to produce self-replicating information patterns in a constrained environment.

I don't think AI will have any better idea of what the fuck is going on than we do as it do will be a cyboid, just as we will be zooids. If I were to put money on who I will welcome as our new overlords, it wouldn't be AI, it would be whatever emerges from this super-charged, colonial-organism-generating machine.