This is a special post for quick takes by Thomas Kwa. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Anything where hill climbing is acceleratory = potentially negative value
Near term alignment (and some kinds of control) have dubious impact, better to focus on things that scale
AIs may be strongly superhuman at most cheaply verifiable tasks soon. Be a complement to this, not a substitute
Don't fight obviously losing battles, e.g. trying to prevent eval awareness
Chem/bio/cybersecurity is low value without specific expertise
Policy impact is probably larger than purely technical impact right now
It’s better for technical safety work to happen in places that have policy connections, like Redwood, METR, GovAI, CAISI, UKAISI
Be suspicious of overcomplicated methods or setups. If the setting is harder to study than other settings, it needs to be higher relevance
Understand clearly what problem the project is solving and whether it's important. Some important problems:
Alignment red teaming. The holy grail is being able to reproduce all precursor behaviors of the Hugging Face incident and other incidents, with no prior knowledge of the incident or even its environments, in a way that generalizes to future full loss of control incidents. I think this is the highest potential payoff problem in technical AI safety right now and deserves ~20% of the entire field's labor. Why? It scales, it's very much unsolved, progress gives AI labs evidence their models are misaligned.
Hill climbing on this is probably better for the labs to do. Outside of labs, I see value in work that makes conceptual progress, has higher relevance than labs' work would, and convinces labs to work on the hill-climbable things.
Preserving monitorability
Model organisms for phenomena we don't have incidents for yet
Scalable oversight
Improving alignment generalization
[Agent foundations and some other areas omitted because I'm less familiar with them]
Research should eventually cash out in impressive results according to "gold standard" validity checks (unless it's preparadigmatic, in which case establishing them is even better) E.g.
If the project is interp, will you get white box methods that outperform the best black box methods in a way that's robust to training?
If scalable oversight, can you improve on the W2SG SoTA in a relevant setting?
A superintelligence would definitely be able to tell it was in an eval. This is because it would be a lot smarter than us.
1BryceStansfield
A super intelligence with a probe in its brain (a steering vector) might think that an orange is a type of piano. This is because it has a probe in its brain.
I think it will just definitely know what's going on. There's no way amplifying some vector or other, that we found with an SVM or whatever else, is going to fool it into thinking it organically thought about that. I know we will have direct access to its brain, but we just aren't going to know enough to make use of that artfully, and it's going to have superintelligent introspection. I agree that in theory, there is some intervention on its brain that would be successful and that it wouldn't be able to detect. But we will have no idea what that interventi... (read more)