I think it is too early to be making legible progress in AI safety. The field is pre paradigmatic, and is in severe need of deconfusion. If we start optimizing on the nothing that we know right now, we risk failing. Normally failure is tolerable; trial and error under reward is how most progress is made. But the problem with AI is that if you fail past a certain point in capabilities, the AI takes over, or helps a dictator cement power forever, or builds an extinction plague for a terrorist, and now you can’t do any more trial and error.
It is much easier to build capabilities than safety, like you said. Reality provides a richer source of data and reward than ~moral philosophy. Capabilities will surely help build safety too, but we don’t know if it does so before or after it removes our ability to make it safe. We also don’t know how to trust the research AIs do for us; it begs the question.
Also, the problem is anti inductive. Instrumental convergence is at the very least a feature of agentic LLMs, and likely of most motivated minds. If you train against signals of deception, and don’t remove the pressure that put it there, the model just learns to deceive you a different way. Etc.
Your plan seems to be, “build capabilities, and hope it later helps differentially drag forward more safety than capabilities.” I don’t think this will work, because that’s what the labs have been doing and it isn’t working for them. Their plan is literally to build it and hope that when it solves alignment, it’s still nice enough to let us align it… which is, again, begging the question.
I think pursuing deconfusion, or better yet, trying to stop it from being built in the next few years so that we have breathing room to pursue deconfusion, is a much better plan. It is within your power to do so; many technical researchers have switched to policy and advocacy.
I recently spoke to Stephen Casper at Princeton's alignment club. Stephen is an CS PhD turned policy researcher at Harvard Kennedy School. He was very sure that mechanistic interpretability was one of the least effective ways to pursue AI safety. As someone who does research in mech interp, I was surprised by how convincing he was. The point is roughly that every mech interp paper looks like one of the following:
"we used the J lens to study reward hacking behavior and found a correlation between X and Y"
"we found the cheating direction in llama"
... you can see where this is going.
Mechanistic interpretability is filled with studies which are interesting, and are useful adjacent, but, by themselves provide little utility. In this sense, they remind me of 2000s and 2010s psychology, presenting something unexpected, novel, and with the hint of usefulness. Did you know that different parts of the brain are associated with deontology and util? How about that disgust reaction is associated with political affiliation?
Reading mech interp papers feels like drinking a la croix of progress in AI safety.
But I do not think it has to be this way. If we stop focusing on finding surprising things about models, and instead focus on using mech interp to do useful things with models, progress is more likely.
This is because approaching a problem with the goal of doing useful stuff does not let you cheat yourself into thinking you found something interesting. As someone who has experience writing papers, and also market making on prediction markets, I can attest to this. The phenomenology of figuring out something which has to work (or else you loose money) is completely different from trying to produce publishable knowledge. You stop pursuing ideas which you would not bet on being important, and more importantly have no motivation to convince others that those ideas are good and worth paying attention to.
Here are some project ideas I think are in the right direction:
Cheaply scaling automated neuron analysis (post-nick's-blog post).
Extracting circuits from transformers for small auxiliary models.
It is not that these ideas are directly aimed at safety. But they have verifiable rewards, whose success conditions require us to know something about how models work, as opposed to research which is nominally about safety, but does not require us to make progress in safety. This is the same principle as training models for prediction. Prediction is not the goal (understanding is the goal), but it is a measure for something which requires the goal. The bitter lesson applies to us, too!