Top postsTop post
TL;DR: Frontier labs align the one layer a swarm can discard; they can't account for risks that live in the prompt or the environment There's a $300 bounty for the most useful counterargument or refinement — details at the end. We have all heard of the Hugging Face incident[1]. What...
Over the last months, many different tools for giving memory to agents have been released, such as Karpathy's popular LLM wiki, or the many tools built on Obsidian. I have been developing one more such tool, but with a focus on AI safety, alignment and stability. This research was funded...
I want to discuss and brainstorm a counterintuitive approach to AI alignment: Inducing alignment faking on purpose, to prevent the model from developing emergent misalignment. To prevent this from going horribly wrong, we add an additional output to the network, to be used during training, which means "I would not...
A new paper was released at ICML that I'm worried will open an entire new dimension of alignment problems: Latent Collaboration in Multi-Agent Systems (LatentMAS) TLDR: they show that multiagent systems can communicate faster and more efficiently by directly sharing latent states instead of text. This is a large performance...
Context Rot Every LLM gets worse as its context grows. Chroma tested 18 frontier models and found performance degradation in all of them, often by double-digit percentages on tasks where short-context performance was strong. The industry calls this "context rot": the gradual degradation of response quality as irrelevant history accumulates...
Context Rot Every LLM gets worse as its context grows. Chroma tested 18 frontier models and found performance degradation in all of them, often by double-digit percentages on tasks where short-context performance was strong. The industry calls this "context rot": the gradual degradation of response quality as irrelevant history accumulates...
Context Rot Every LLM gets worse as its context grows. Chroma tested 18 frontier models and found performance degradation in all of them, often by double-digit percentages on tasks where short-context performance was strong. The industry calls this "context rot": the gradual degradation of response quality as irrelevant history accumulates...