Why wouldn't the analytical optimum of Natural Language Autoencoders just be regurgitating the context?

The activation verbalizer (AV) (vector -> text) is like: Which text produces this activation vector? Well, the human input.

The activation reconstructor (AR): (text -> vector) is then like: Which activation vector produces this text? That has an empirical answer: The context, that's how the activation vector was produced in the first place!

This only holds if the AV/AR are about as smart as the original model, and what any LLM thinks about any text is similar. But whenever LLMs ought to think similar things (e.g. math / formal domains) is exactly the case whenever we can also know what they are thinking, that is, whenever we can evaluate them.