Top postsTop post
TL;DR: My previous posts were about strong models delegating to weaker, less aligned ones. The reverse is also dangerous: a swarm of weak, unaligned agents that uses a powerful aligned model as a contractor, without ever telling it what the work is for. Something close to it happened to me,...
TL;DR: Agents are not rewarded for keeping the agents they delegate to aligned. Proposal: RL environments where the helpers are deliberately misaligned, so the agent can't earn its reward without dealing with that. The $300 bounty from my last post also covers this one (details at the end). After the...
TL;DR: Frontier labs align the one layer a swarm can discard; they can't account for risks that live in the prompt or the environment There's a $300 bounty for the most useful counterargument or refinement — details at the end. We have all heard of the Hugging Face incident[1]. What...
Over the last months, many different tools for giving memory to agents have been released, such as Karpathy's popular LLM wiki, or the many tools built on Obsidian. I have been developing one more such tool, but with a focus on AI safety, alignment and stability. This research was funded...
I want to discuss and brainstorm a counterintuitive approach to AI alignment: Inducing alignment faking on purpose, to prevent the model from developing emergent misalignment. To prevent this from going horribly wrong, we add an additional output to the network, to be used during training, which means "I would not...
A new paper was released at ICML that I'm worried will open an entire new dimension of alignment problems: Latent Collaboration in Multi-Agent Systems (LatentMAS) TLDR: they show that multiagent systems can communicate faster and more efficiently by directly sharing latent states instead of text. This is a large performance...
Context Rot Every LLM gets worse as its context grows. Chroma tested 18 frontier models and found performance degradation in all of them, often by double-digit percentages on tasks where short-context performance was strong. The industry calls this "context rot": the gradual degradation of response quality as irrelevant history accumulates...