Top postsTop post
TL;DR * Inspired by the introspective awareness and CoT controllability papers, we made a benchmark to measure how well models can control their activations while completing a simple task. * We are motivated by the concern that highly introspective models could control their activations, confounding probes and other monitors, and...
TLDR: * Many important decisions for safety depend on or are influenced by benchmark scores. * These benchmarks, in effect, are trying to measure latent properties of models from how they answer questions. * Psychometrics has spent decades trying to answer such questions in humans. We can probably steal some...
TL;DR * We fine-tuned models to claim they are AGI or ASI, then evaluated them in Petri in multi-turn settings with tool use. * On GPT-4.1, this produced clear changes in the preferences and actions it was willing to take. In the most striking case, the AGI-claiming model attempted to...
TL;DR LLMs can be trained to detect activation steering robustly. With lightweight fine-tuning, models learn to report when a steering vector was injected into their residual stream and often identify the injected concept. The best model reaches 95.5% detection on held-out concepts and 71.2% concept identification. Lead Author: Joshua Fonseca...
> TL;DR: Letting a model overfit first, then applying Frobenius norm regularization, achieves grokking in roughly half the steps of Grokfast on modular arithmetic. I learned about grokking fairly recently, and thought it was quite interesting. It sort of shook up how I thought about training. Overfitting to your training...
TL;DR I fine-tuned deepseek-ai/deepseek-llm-7b-chat to detect when its activations are being steered. It works: 85% accuracy on held-out concepts, 0% false positives. But two things bother me about what I found: First of all, if models can indeed learn to detect steering, steering-based evaluations might become unreliable. A model that...