If you want to keep the search function from wireheading the world model then you have to code "don't break the world model" into your value function. This is a general contradiction to the Orthogonality Thesis. A sufficiently powerful world-optimizing artificial intelligence must have a value function that preserves the integrity of its world model, because otherwise it'll just wirehead itself, instead of optimizing the world.
If the value function says ~"maximise the number of paperclips, as counted by my paperclip-counting-machinery", a weak AI mig... (read more)
Seems very possible given the mech. explanation of prompt injection post argues agents identify their CoT via writing style. Maybe getting another LLM to paraphrase each agent message would cause Astra to behave differently?