This is a special post for quick takes by phoenix. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Playing with the idea that identity is less of an "instantaneous I" of current experience and more like the continuity of experiential snapshots under the curve. Like how no individual frame is "the movie," but when you run them at 24 frames per second you get the experience of a film that emerges from the continuity.
One of the problems with performing independent AI safety research is that if you do actually end up coming up with something novel and sufficiently powerful you immediately run into a catch-22: you can't broadcast the specifics (because safety research is capabilities research), and without a public track record it's unlikely that anyone relevant will entertain a private review.
Everyone operates from a locally justifiable position, but it does make getting information into the hands of the people/organizations best suited to act on it responsibly difficult.
On the current margin most safety research is several times more impactful as safety research than capabilities research. If publishing allows you to either stop wasting your time or get a job where you have access to 10x more resources, it's a huge net win even if capabilities progress you generate removes 20% of your impact, which is unlikely. So I think this issue is way overblown.
I've been working on a tool for ambitious mechanistic interpretability called ATLAS. What started as an exploratory technique for the ARC White-Box Estimation Challenge turned into a different kind of tool entirely, so I figured I'd drop it in here, as there's been a bit of mech interp discussion recently.
Playing around with Hoel's Causal Emergence 2.0, I arrived at ATLAS by treating a forward pass as a program rolling through a field of constraints and reading that pass at the grain where the model's causation lives. The net has an internal geometry that can be read with machine-exact precision, and that precision means that looking into the black box doesn't require methods that disturb the computation like ablation or activation patching. Instead, when you use exact accounting, you can know the effect of a weight-change edit before you actually run it by solving it in closed form and predicting the result rather than observing it.
Looking for collaborators with experience in the field to check it out and give feedback. Current model works on basic MLPs, GPT-2, Tracr, Pythia. The in repo demo goes up to Qwen2.5-72B-Instruct (though I've run larger models, 72B is what's in there right now).
Current capabilities include circuit discovery and zero-run surgical weight edits (demoed on a grokking model and GPT-2), among a number of other features, all done without ablation, approximation, dictionaries, or fitting.
I've got a separate repo going for using untrained GPT-2 nets as programmatic breadboards as an extension of the technique to explore the various interactions of valence and drives, but figured I'd start with the basics here in this repo to gauge interest as that's a related but secondary project.
Repo is here: https://github.com/OperatorPhoenix/ATLAS
Kicking this around for a post I'm drafting: when an LLM hallucinates something, it's usually at least plausible for the situation. Like a hallucinated citation generally has proper formatting, etc, so the generation worked well enough. It's also confidently incorrect, which is of course what makes it so dangerous to people who don't know any better and so annoying to people who actually know the subject matter.
I've been thinking of the set of all possible responses as a kind of navigable topology (think like the Library of Babel website but instead of linear pages it's a high-dimensional manifold), and it's been productive to think of hallucination as a kind of localization problem. The model is in "citation" space when it should be in "I don't have this" space. The output is locally correct for where the model thinks it is; it's just in the wrong place in response-space relative to reality.
Thinking of the set of possible responses as a kind of response-space provides an interesting lens on the problem. If they're not broken outputs then they may be expected outputs from the wrong context. Would also help explain why "just try harder to be accurate" doesn't really work all that well: effort in generation doesn't help if the error is upstream, in mode-selection. (Though saying "try harder" may well prompt the system to actually evaluate where it is in response-space and relocate if necessary, so it's not totally useless.)
Also suggests an interesting tack might not be "how do we make the model generate better" but "what determines which mode/space the model is in, and can that be checked before output?"
Upshot of this perspective is that it means that just adding compute to a model won't actually help with hallucination if it doesn't also expand reasoning about where the model is in response-space in the first place. If the model doesn't have any way to anchor its internal state to reality, it can compute for a thousand years and never land on an answer that is coherent with that reality. The hallucination bottleneck from this angle doesn't seem like adding additional knowledge, it seems like the limitation is the system's context about where it should be within its own reasoning space.
Anybody else have a similar perspective, or know of posts/papers that explore this dimension? Would love an outside perspective.