Our goal is to build a roughly human-level automated alignment researcher. We can then use vast amounts of compute to scale our efforts, and iteratively align superintelligence.
To align the first automated alignment researcher, we will need to 1) develop a scalable training method, 2) validate the resulting model, and 3) stress test our entire alignment pipeline:
To provide a training signal on tasks that are difficult for humans to evaluate, we can leverage AI systems to assist evaluation of other AI systems (scalable oversight). In addition, we want to understand and control how our models generalize our oversight to tasks we can’t supervise (generalization).
Finally, we can test our entire pipeline by deliberately training misaligned models, and confirming that our techniques detect the worst kinds of misalignments (adversarial testing).
We expect our research priorities will evolve substantially as we learn more about the problem and we’ll likely add entirely new research areas. We are planning to share more on our roadmap in the future.
[...]
While this is an incredibly ambitious goal and we’re not guaranteed to succeed, we are optimistic that a focused, concerted effort can solve this problem:C
There are many ideas that have shown promise in preliminary experiments, we have increasingly useful metrics for progress, and we can use today’s models to study many of these problems empirically.
Ilya Sutskever (cofounder and Chief Scientist of OpenAI) has made this his core research focus, and will be co-leading the team with Jan Leike (Head of Alignment). Joining the team are researchers and engineers from our previous alignment team, as well as researchers from other teams across the company.
This is incredible—the most hopeful I’ve been in a long time. 20% of current compute and plans for a properly-sized team! I’m not aware of any clearer or more substantive sign from a major lab that they actually care, and are actually going to try not to kill everyone. I hope that DeepMind and Anthropic have great things planned to leapfrog this!
I see what you mean. I was thinking "labs try their hardest to demonstrate that they are working to align superintelligent AI, because they'll look less responsible than their competitors if they don't."
I don't think keeping "superalignment" techniques secret would generally make sense right now, since it's in everyone's best interests that the first superintelligence isn't misaligned (I'm not really thinking about "alignment" that also improved present-day capabilities, like RLHF).
As for your second point, I think that for an AI lab that wants to improve PR, the important thing is showing "we're helping the alignment community by investing significant resources into solving this problem," not "our techniques are better than our competitors'." The dynamic you're talking about might have some negative effect, but I personally think the positive effects of competition would vastly outweigh it (even though many alignment-focused commitments from AI labs will probably turn out to be not-very-helpful signaling).
4mako yass
When you talk of competing to look more responsible, or to improve PR, for which audience would you say they're performing?
I know, personally, I'll work a lot harder and for less pay and be more likely to apply in the first place to places that seem more responsible, but I'm not sure how strong that effect is.
Some quotes:
[...]