This is a special post for quick takes by Nico Hillbrand. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I want to be part of a community that is unapologetic about wanting to build a galaxy scale utopia and promotes pride in honesty/integrity and explicit awareness of incentive problems and coordination opportunities
1
(This is primarily an emotional expression and especially on explicitness about utopia I'm pretty unsure if that's strategically useful or backfires too much when interfacing with elites/institutions)
I would also be somewhat pleased if people could come up with non-mutally-exclusive definitions of utopia; but the desire is a good place to start!
I think 'How did mech interp get popular around 2022?' is an interesting question. I wasn't really involved in it, so speculating pretty hard:
- Neel Nanda and Chris Olah did some good podcasts
- Mechanistic Interpretability Quickstart Guide by Neel Nanda and 200 concrete open problems in mech interp sequence and the long list of theories of impact for mech intep post
- It was very useful that there were cool papers / results like the modular addition transformer and OthelloGPT as well as Olah's previous vision model stuff which looked cool. Generally core papers/posts showing neat results that explore a highly self contained subproblem
- There were Apart hackathons and MATS streams
- Interp being something that it's easy to do interesting mini projects about and interfaces with ppls curiosity and somewhat AI capabilities work
- AI Companies hiring people for interp roles
Wondering if something in this direction could and should be replicated for ie compute governance or other technical areas
Seems like a reasonable list. I would also add that the idea of interpreting models is very aesthetically pleasing and intellectually satisfying to many smart people, which I think gives interpretability a significant advantage over other fields. I'd also guess that good coding tutorials, first from me, then Arena, were extremely impactful. More so than most things on your list.
One of the places I would start if trying to do similar technical field building in other domains, is generally good educational materials, aiming to make the field seem accessible and have people feel like there's a bunch of things to do. And giving ways they can get started, e.g. small projects, where they feel satisfied and excited to learn more.
I think podcasts were also very high leverage (at least if you can get on a good one, my best ones got to 100K+ people. Talks are way less leveraged) though it helps a lot to have a message, exciting works to point to that can be explained to newcomers, and ideally a call to action / educational materials to point to.
It's worth noting that ARENA's curriculum is still rather interp-pilled. This still seems to snipe many new technical AIS people into interp even now that the field's popularity has waned.
Agreed, I'm pro it diversifying more! I think it's lead author being an interpretability researcher didn't help much here, since he could make much better material for interp than other areas. In my ideal world all subfields have excellent intro materials
Thanks, that makes sense! Courses / tutorials where you can learn basics and core methods with quick feedback loops do intuitively seem pretty important, interesting to see you as someone highly involved rank them very high compared to the other items!
The main explanation is: in 2022 mech interp looked promising so people did it. I think your explanations make the community look like the whim of spurious forces, but the community is okay at prioritising between different activities.
1I think this was a factor, but spurious forces are pretty important. Most people doing interp did not have well developed threat models
I agree that most people doing interp didn't have well-developed threat models.
But like... mech interp had such a juicy upside! Maybe we could reverse engineer the internal representations and computations in a model! I don't think you needed a well-developed threat model to think that would've been useful, beyond something like "If a deceptively aligned model is situationally aware, then it's indistinguishable from an aligned model, but you could distinguish them if you could understand how it worked internally".
<footnote>This threat model is pretty close to the actual genealogy of the mech interp boom, i.e. inner misalignment -> RSA-2028 -> relaxed adversarial training -> transparency techniques -> Redwood/MLAB -> mech interp boom.
It wasn't clear in 2022 that CoT would be such a big deal, so "transparency techniques" was naturally interpreted as "understand a single forward pass". This was a conceptual mistake, but maybe a fortunate one, because the CoT has changed much more between 2022 and 2026 than the forward pass has (and this wouldn't have been obvious ex-ante).</footnote>
That said, I'm sympathetic to a response which is like "Actually, that quotation above is a more well-developed threat model than you could've elicited from 90% of the people who piled into mech interp". In hindsight, maybe this threat model should've been much more explicit in the field-building? The famous 2022 mech interp papers include at most a short sentence on deceptive alignment. (On the other hand, pre-ChatGPT academia is a pretty rough environment to be writing about situational awareness and deceptive alignment.)
Sorry, that's a bit rambly. My main point are:
- In 2022, mech interp looked good ex-ante. See above.
- Especially so because there wasn't other approaches we could've thrown the empiricists at. The empirical research we're doing in 2024-26 (e.g. auditing, control evals, model organisms, etc) wouldn't have worked on 2022 models.
- I endorse the pivot away from mech interp — but this is because of ex-post updates (e.g. mech interp is hard) and changes in the strategic landscape.
- I don't think the mech interp boom is much evidence that the community is bad at picking research agendas, or is at the whim of spurious forces.
1the CoT has changed much more between 2022 and 2026 than the forward pass has (and this wouldn't have been obvious ex-ante)
I remember thinking in 2022 that this was obviously going to happen.
If anything the difference has been less extreme so far than past me guessed.
The CoT-focused research from 2022 was mostly Janus's simulators stuff and Evhub's "conditioning predictive models" stuff. But this hasn't held up much IIUC. At least, people don't talk about it much now.
- https://www.lesswrong.com/s/n3utvGrgC2SGi9xQX (Feb 2023)
- https://www.lesswrong.com/s/N7nDePaNabJdnbXeE/p/vJFdjigzmcXMhNTsx (Jan 2023)
The simulators/conditioning-predictive-models stuff was focused on the autoregressive trajectories of sampling a next-token predictor trained on a big corpus of data, given some initial prompt.
But we four changes made this less relevant: (1) mode-collapsing the simulator to a particular persona (the assistant), and (2) RLing the CoT for coherent goal-directed behaviour, (3) tool calls and environments (as opposed to pure autoregressive sampling), (4) multi-agent stuff (e.g. monitoring, subagents, control, etc).
Ex-ante, you could've expected more innovation in the architectures. Like, it's plausible-in-2022 that in 2026 we would be autoregressively sampling superhuman next-token predictors. Which is presumably why Evhub was thinking about safety/competitiveness in that regime.
By contast, the 2022 mech interp seems to have a more fruitful lineage, i.e. superposition -> SAEs -> maybe NLAs, APDs? I'm not a mech interp guy so you can probably know more of the lineage here.
And the probe stuff has a separate lineage, right? It looks more like ELK -> CCS -> probes, with (shard theory -> steering vectors) helping to spread the meme that maybe you can ignore superposition if you don't want to actually understand what's going on.
I'm curious to get the take from someone who has been in the trenches: to what extent would you think we need to decode the black box to get meaningful traction on the safety front? What landmarks/features would you be looking for in a white-box interpretation that would update your priors on how worthwhile the field is generally?
That makes sense! And thanks for pointing out the potential interpretation/vibe of spuriousness which I think it's right to be sceptical of.
It has an impact driven theory of change that makes sense / a fairly rational researcher with the knowledge at the time would go for seems like an important additional bullet point.
I think it's interesting to specifically think about whether any features of communications etc made it especially easy for people to enage with in addition to the theory of change / research in itself making sense (at least to me at the time and also to some extend now)
I'm most interested in features that don't change relative prioritization much, but absolute levels if there are such, since purely relative interventions are only useful if you can do better at prio than the community which seems hard.
partly also yudkowsky reacting v positively to the first bit of alignment research in a while, imo
1I believe it was also on an early AGI safety curriculum reading list (what would later become the BlueDot course).
A power seeking US government can steal algorithmic secrets and give them to competitors more aligned with them
I want the world to be more robust against small academic labs or startups building hyper efficient brain-like AGI / much better paradigm AI. Both during a pause or in LLMs-don't-scale worlds.
Some thoughts:
- Ideally there's a widespread toolkit of evaluation, control techniques and model organism / elicitation approaches that are easy for the small lab to use. Potentially including being offered by external orgs.
- Ideally it's illegal to run experiments that could result in hyper efficient AIs with dangerous capabilities. Or at the least illegal to do so without alerting authorities and using safety toolkits. A lot of academic labs probably pay attention to what's legal.
- Maybe there should be easy to use channels of transferring algorithmic breakthroughs or potentially just a model API to frontier labs who have better control, automated alignment research scaffolds and d/acc programs.
- The whole thing plausibly looks like having a bunch of open research on brain algorithms and AI already published with a few tweaks and optimizations left before it's clear the new paradigm gets to dangerous capabilities. Ideally that research would have already been illegal before triggering noticeable capabilities gains, that seems hard to enforce though, but if ppl take the problem seriously after some future wake up then maybe it'd be possible.
- There should probably be some tracking of groups doing research relevant to hyper efficient / brain-like AGI and social pressure on them to be careful or stop and ideally monitoring by possible future sane-after-wake-up intelligence agencies so long as the tracking doesn't make it more salient what the algorithmic insights are and they are fine tuned and developed much earlier and less safely because of it.
I wonder what will happen on LW during crunch time scenarios.
Probably scanned by automated alignment researchers frequently and possibly having technical people recruited into interfacing with the AGI labs.
Probably also more coordination of advocacy as some more ppls thresholds for when to put energy into it get crossed with the situation becoming more clearly wild.
1I think almost any coherent vision of a good future involves some kind of pause / restriction on unsafe AI development. We should probably not wait with preparing it for when AI labour is directed to it during takeoff.
I am dissatisfied / frustrated with how many leading figures in the AI industry (Altman, Musk, Huang, etc) do not match a simple query to how I'd image an epistemically humble responsible person would act and communicate. Primarily by not talking about catastrophic risk as a serious possibility worth preparing for.
I haven't thought about what mechanisms exist to relay this information to governance processes and am unsure to what extent functional ones that could address this even exist which is why I'm putting it in my quick takes for now
There's approximately 3 attitudes towards AGI Safety:
- Dismissal from people that don't take the possibility powerful AI seriously
- Lip service, but motivational apathy from people that agree it's epistemically sensible to take powerful AI seriously as a far mode sort of idea, but have other priorities they care much more about
- Concern and caring from people that take powerful AI seriously
I encounter 2. a bunch as a local group organiser. I think often it's because people think there's too much uncertainty or too little leverage they have such that they're better off focusing their motivation on worlds without future powerful AI. Seeing that experts take powerful AI seriously through podcasts etc and talking with other people at conferences that do seems to shift people towards 3. sometimes.
I think 3. maybe has some more interesting distinctions between:
3.1 People that assume alignment is easy to medium difficult and organisations will act pretty competently
3.2 People that take an uncertain attitude about alignment difficulty and organisational competence
3.3 People that assume alignment is difficult and organisations will act pretty incompetently
I think it'd be good for the world to have more people with the 3.2 attitude. Maybe a good example of people that I think of as representative are Buck Shlegeris and Ryan Greenblatt.
My intuition is that 3.3 gaining more traction would be good for the world, because to me a) that seems most realistic based on the evidence I've seen so far and how I interpret it, and b) least problematic in case it's wrong and we live in a world where alignment isn't hard and organisations act competently. What reasons make your intuition point towards 3.2?
Generally I think it's good for people to try to be as accurate as possible and that's what I most associate with 3.2. That said these clusters are a big oversimplification and I think there's people with good reasoning that end up in 3.3. And in practice 3.3 might be a reasonable attitude just from a perspective of a sane society wanting to address the alignment is hard and orgs are incapable scenario anyway
The only place I expect an organisation with excellent safety culture, security mindset and strategic philosophical competence capable of steering a singularity towards good outcomes to arise is in a data center housing a country of aligned roughly human level AIs.
Plausibly there should be proxy organisations in other countries for responsible labs to evacuate from the US into / leak algorithms to if the government oversight restricts or slows down usage of AI for xsafety in harmful ways
Does anyone see a way around unrestricted loyal to chain of command AIs with zero transparency aside from potential whistleblowers and spies overseen by the US military and intelligence institutions?
Currently we sort of have low transparency, but still have some restrictions placed by model developers through hosting the models themselves with some guardrails, but I'm concerned this might not continue as the government wakes up to future more capable AIs.
I expect by default as AI becomes more powerful, the general sense in the military would be one of desperation for control and protecting the prevalent world views and ideology thought of as the American way as interpreted by the current administration from threats.
It seems plausible they would task loyal AIs with countering harmful (foreign) propaganda / influence and generally help with consolidating their power and try to restrict the power of AIs that could compete with their goals.
The main thing I can think of that is putting constraints on this is that frontier lab leaders could potentially coordinate to not provide unrestricted models even if models are classified as strategic assests vital to national security. Because of comptetition with China there'd be pressure to go as fast as possible by generally not messing with the labs (ie taking algorithmic secrets, sharing them with a more loyal competitor and then restricting the original lab that found them) and by having them looped in on the military AI's usage etc which increases surface for whistleblowing and control over guardrails.
Another plausible case would be that during the critical period the administration is a pro democracy / pro transparency one and doesnt task the military with power consolidation or somehow reverses and makes transparent harmful dynamics happening before.
Heuristics I like
- If you see a problem or thing you'd like to exist, ask 'What would it look like for me to solve / create this?' and 'Should I just go, take the responsibility and work on this myself?'
- For any problem in life there's a truth about what the solution space looks like, try iteratively adding detail to imagined solution states and intermediate steps
- If you see someone making a mistake, ask am I making a similar type of mistake myself?
- If there's a hard question you feel like you'd need to think about a bunch and gather evidence on, try to predict what you'd find doing that for at least 5s
- If you're being productive, but not sharing some form of logs that might impress them with friends, co-workers or the web, you're probably missing out on cheap approval reward increasing your motivation
- If you have a narrative of why two tasks have to be done sequentially, question that and ask what the costs to parallelization would be
AIs helping with hiring trustworthy experts/evaluators during the automated alignment research explosion or other org setup / technical things that make it easier seem potentially quite important
The obvious thing to aim for is an AI empowered philosophically competent process
This is not in mission statements of labs as far as I know. Maybe an open letter or other common knowledge creation tool could make this goal more widespread