This is a special post for quick takes by Nico Hillbrand. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I want to be part of a community that is unapologetic about wanting to build a galaxy scale utopia and promotes pride in honesty/integrity and explicit awareness of incentive problems and coordination opportunities
1
(This is primarily an emotional expression and especially on explicitness about utopia I'm pretty unsure if that's strategically useful or backfires too much when interfacing with elites/institutions)
I would also be somewhat pleased if people could come up with non-mutally-exclusive definitions of utopia; but the desire is a good place to start!
I think 'How did mech interp get popular around 2022?' is an interesting question. I wasn't really involved in it, so speculating pretty hard:
- Neel Nanda and Chris Olah did some good podcasts
- Mechanistic Interpretability Quickstart Guide by Neel Nanda and 200 concrete open problems in mech interp sequence and the long list of theories of impact for mech intep post
- It was very useful that there were cool papers / results like the modular addition transformer and OthelloGPT as well as Olah's previous vision model stuff which looked cool. Generally core papers/posts showing neat results that explore a highly self contained subproblem
- There were Apart hackathons and MATS streams
- Interp being something that it's easy to do interesting mini projects about and interfaces with ppls curiosity and somewhat AI capabilities work
- AI Companies hiring people for interp roles
Wondering if something in this direction could and should be replicated for ie compute governance or other technical areas
Seems like a reasonable list. I would also add that the idea of interpreting models is very aesthetically pleasing and intellectually satisfying to many smart people, which I think gives interpretability a significant advantage over other fields. I'd also guess that good coding tutorials, first from me, then Arena, were extremely impactful. More so than most things on your list.
One of the places I would start if trying to do similar technical field building in other domains, is generally good educational materials, aiming to make the field seem accessible and have people feel like there's a bunch of things to do. And giving ways they can get started, e.g. small projects, where they feel satisfied and excited to learn more.
I think podcasts were also very high leverage (at least if you can get on a good one, my best ones got to 100K+ people. Talks are way less leveraged) though it helps a lot to have a message, exciting works to point to that can be explained to newcomers, and ideally a call to action / educational materials to point to.
It's worth noting that ARENA's curriculum is still rather interp-pilled. This still seems to snipe many new technical AIS people into interp even now that the field's popularity has waned.
Agreed, I'm pro it diversifying more! I think it's lead author being an interpretability researcher didn't help much here, since he could make much better material for interp than other areas. In my ideal world all subfields have excellent intro materials
Thanks, that makes sense! Courses / tutorials where you can learn basics and core methods with quick feedback loops do intuitively seem pretty important, interesting to see you as someone highly involved rank them very high compared to the other items!
The main explanation is: in 2022 mech interp looked promising so people did it. I think your explanations make the community look like the whim of spurious forces, but the community is okay at prioritising between different activities.
1I think this was a factor, but spurious forces are pretty important. Most people doing interp did not have well developed threat models
I agree that most people doing interp didn't have well-developed threat models.
But like... mech interp had such a juicy upside! Maybe we could reverse engineer the internal representations and computations in a model! I don't think you needed a well-developed threat model to think that would've been useful, beyond something like "If a deceptively aligned model is situationally aware, then it's indistinguishable from an aligned model, but you could distinguish them if you could understand how it worked internally".
<footnote>This threat model is pretty close to the actual genealogy of the mech interp boom, i.e. inner misalignment -> RSA-2028 -> relaxed adversarial training -> transparency techniques -> Redwood/MLAB -> mech interp boom.
It wasn't clear in 2022 that CoT would be such a big deal, so "transparency techniques" was naturally interpreted as "understand a single forward pass". This was a conceptual mistake, but maybe a fortunate one, because the CoT has changed much more between 2022 and 2026 than the forward pass has (and this wouldn't have been obvious ex-ante).</footnote>
That said, I'm sympathetic to a response which is like "Actually, that quotation above is a more well-developed threat model than you could've elicited from 90% of the people who piled into mech interp". In hindsight, maybe this threat model should've been much more explicit in the field-building? The famous 2022 mech interp papers include at most a short sentence on deceptive alignment. (On the other hand, pre-ChatGPT academia is a pretty rough environment to be writing about situational awareness and deceptive alignment.)
Sorry, that's a bit rambly. My main point are:
- In 2022, mech interp looked good ex-ante. See above.
- Especially so because there wasn't other approaches we could've thrown the empiricists at. The empirical research we're doing in 2024-26 (e.g. auditing, control evals, model organisms, etc) wouldn't have worked on 2022 models.
- I endorse the pivot away from mech interp — but this is because of ex-post updates (e.g. mech interp is hard) and changes in the strategic landscape.
- I don't think the mech interp boom is much evidence that the community is bad at picking research agendas, or is at the whim of spurious forces.
1the CoT has changed much more between 2022 and 2026 than the forward pass has (and this wouldn't have been obvious ex-ante)
I remember thinking in 2022 that this was obviously going to happen.
If anything the difference has been less extreme so far than past me guessed.
The CoT-focused research from 2022 was mostly Janus's simulators stuff and Evhub's "conditioning predictive models" stuff. But this hasn't held up much IIUC. At least, people don't talk about it much now.
- https://www.lesswrong.com/s/n3utvGrgC2SGi9xQX (Feb 2023)
- https://www.lesswrong.com/s/N7nDePaNabJdnbXeE/p/vJFdjigzmcXMhNTsx (Jan 2023)
The simulators/conditioning-predictive-models stuff was focused on the autoregressive trajectories of sampling a next-token predictor trained on a big corpus of data, given some initial prompt.
But we four changes made this less relevant: (1) mode-collapsing the simulator to a particular persona (the assistant), and (2) RLing the CoT for coherent goal-directed behaviour, (3) tool calls and environments (as opposed to pure autoregressive sampling), (4) multi-agent stuff (e.g. monitoring, subagents, control, etc).
Ex-ante, you could've expected more innovation in the architectures. Like, it's plausible-in-2022 that in 2026 we would be autoregressively sampling superhuman next-token predictors. Which is presumably why Evhub was thinking about safety/competitiveness in that regime.
By contast, the 2022 mech interp seems to have a more fruitful lineage, i.e. superposition -> SAEs -> maybe NLAs, APDs? I'm not a mech interp guy so you can probably know more of the lineage here.
And the probe stuff has a separate lineage, right? It looks more like ELK -> CCS -> probes, with (shard theory -> steering vectors) helping to spread the meme that maybe you can ignore superposition if you don't want to actually understand what's going on.
I'm curious to get the take from someone who has been in the trenches: to what extent would you think we need to decode the black box to get meaningful traction on the safety front? What landmarks/features would you be looking for in a white-box interpretation that would update your priors on how worthwhile the field is generally?
That makes sense! And thanks for pointing out the potential interpretation/vibe of spuriousness which I think it's right to be sceptical of.
It has an impact driven theory of change that makes sense / a fairly rational researcher with the knowledge at the time would go for seems like an important additional bullet point.
I think it's interesting to specifically think about whether any features of communications etc made it especially easy for people to enage with in addition to the theory of change / research in itself making sense (at least to me at the time and also to some extend now)
I'm most interested in features that don't change relative prioritization much, but absolute levels if there are such, since purely relative interventions are only useful if you can do better at prio than the community which seems hard.
partly also yudkowsky reacting v positively to the first bit of alignment research in a while, imo
1I believe it was also on an early AGI safety curriculum reading list (what would later become the BlueDot course).
I think almost any coherent vision of a good future involves some kind of pause / restriction on unsafe AI development. We should probably not wait with preparing it for when AI labour is directed to it during takeoff.
I am dissatisfied / frustrated with how many leading figures in the AI industry (Altman, Musk, Huang, etc) do not match a simple query to how I'd image an epistemically humble responsible person would act and communicate. Primarily by not talking about catastrophic risk as a serious possibility worth preparing for.
I haven't thought about what mechanisms exist to relay this information to governance processes and am unsure to what extent functional ones that could address this even exist which is why I'm putting it in my quick takes for now
There's approximately 3 attitudes towards AGI Safety:
- Dismissal from people that don't take the possibility powerful AI seriously
- Lip service, but motivational apathy from people that agree it's epistemically sensible to take powerful AI seriously as a far mode sort of idea, but have other priorities they care much more about
- Concern and caring from people that take powerful AI seriously
I encounter 2. a bunch as a local group organiser. I think often it's because people think there's too much uncertainty or too little leverage they have such that they're better off focusing their motivation on worlds without future powerful AI. Seeing that experts take powerful AI seriously through podcasts etc and talking with other people at conferences that do seems to shift people towards 3. sometimes.
I think 3. maybe has some more interesting distinctions between:
3.1 People that assume alignment is easy to medium difficult and organisations will act pretty competently
3.2 People that take an uncertain attitude about alignment difficulty and organisational competence
3.3 People that assume alignment is difficult and organisations will act pretty incompetently
I think it'd be good for the world to have more people with the 3.2 attitude. Maybe a good example of people that I think of as representative are Buck Shlegeris and Ryan Greenblatt.
My intuition is that 3.3 gaining more traction would be good for the world, because to me a) that seems most realistic based on the evidence I've seen so far and how I interpret it, and b) least problematic in case it's wrong and we live in a world where alignment isn't hard and organisations act competently. What reasons make your intuition point towards 3.2?
Generally I think it's good for people to try to be as accurate as possible and that's what I most associate with 3.2. That said these clusters are a big oversimplification and I think there's people with good reasoning that end up in 3.3. And in practice 3.3 might be a reasonable attitude just from a perspective of a sane society wanting to address the alignment is hard and orgs are incapable scenario anyway
The only place I expect an organisation with excellent safety culture, security mindset and strategic philosophical competence capable of steering a singularity towards good outcomes to arise is in a data center housing a country of aligned roughly human level AIs.
AIs helping with hiring trustworthy experts/evaluators during the automated alignment research explosion or other org setup / technical things that make it easier seem potentially quite important