josh :)

In the current safety landscape, researchers are encouraged to use as many agents as possible, even in “fleets,” where a large number of them go out and run experiments while you, uh... wait for them to come back? I've spent the last few months of the Anthropic Fellows Program trying...

TL;DR: AI welfare is hard to evaluate when one model can inhabit many personas. Recent work suggests that future training may produce a single stable underlying persona that can play many roles, making model welfare much easier to evaluate. One of the hardest questions in AI welfare is deciding what...

TL;DR * Inspired by the introspective awareness and CoT controllability papers, we made a benchmark to measure how well models can control their activations while completing a simple task. * We are motivated by the concern that highly introspective models could control their activations, confounding probes and other monitors, and...

TLDR: * Many important decisions for safety depend on or are influenced by benchmark scores. * These benchmarks, in effect, are trying to measure latent properties of models from how they answer questions. * Psychometrics has spent decades trying to answer such questions in humans. We can probably steal some...

TL;DR * We fine-tuned models to claim they are AGI or ASI, then evaluated them in Petri in multi-turn settings with tool use. * On GPT-4.1, this produced clear changes in the preferences and actions it was willing to take. In the most striking case, the AGI-claiming model attempted to...

TL;DR LLMs can be trained to detect activation steering robustly. With lightweight fine-tuning, models learn to report when a steering vector was injected into their residual stream and often identify the injected concept. The best model reaches 95.5% detection on held-out concepts and 71.2% concept identification. Lead Author: Joshua Fonseca...

> TL;DR: Letting a model overfit first, then applying Frobenius norm regularization, achieves grokking in roughly half the steps of Grokfast on modular arithmetic. I learned about grokking fairly recently, and thought it was quite interesting. It sort of shook up how I thought about training. Overfitting to your training...