Seems very possible given the mech. explanation of prompt injection post argues agents identify their CoT via writing style. Maybe getting another LLM to paraphrase each agent message would cause Astra to behave differently?
If you want to keep the search function from wireheading the world model then you have to code "don't break the world model" into your value function. This is a general contradiction to the Orthogonality Thesis. A sufficiently powerful world-optimizing artificial intelligence must have a value function that preserves the integrity of its world model, because otherwise it'll just wirehead itself, instead of optimizing the world.
If the value function says ~"maximise the number of paperclips, as counted by my paperclip-counting-machinery", a weak AI mig... (read more)
Most likely known, but FYI the audio version doesn't follow the text exactly (audio version was before some final editing and refactoring?).
It's a ~95% match until the split into 3 teams, where the audio and text versions proceed in a different sequence. After this the versions differ a lot in wording and organization.
Overall, both versions still make sense, it's just slightly confusing when taken together. Thank you to everyone involved for making this explainer!