What safety techniques exist? What techniques should labs be using, now or if they soon develop powerful models? What techniques should be considered industry-best-practices? "How might we align transformative AI if it’s developed very soon?" What's our current bag of tricks? What alignment research is ready to go?

Some general sources:

• Buck's The current alignment plan, and how we might improve it (especially v1 Plan and Classes of technical research) (and Redwood's unpublished proposals on control)
• Holden's How might we align transformative AI if it’s developed very soon? (especially Key tools)

Some sources on particular (kinds of) techniques:

• RLHF
• Activation engineering

Some categories of safety techniques:

• Before-training safety techniques (e.g. architecture & dataset)
• Before-deployment safety techniques
  • During training + between training and deployment
• (Before-deployment threat detection)
  • (Also important is detecting dangerous properties of the model — the current kind-of-technique is model evals or red-teaming mostly done by humans; maybe someday there will be AI-based red-teaming or interpretability. This is noncentral in this post but I'm interested in specific techniques.)
• During-deployment, inference-time control/monitoring
  • (Including not just techniques prompting and chain-of-thought and monitoring-outputs-for-coups, but also tools or stuff-you-build like training a classifier and building interpretable agent-scaffolding)

This all depends on threat models and how the leading lab would use its powerful AI to prevent others from training/deploying unaligned powerful AI (or make the world robust to unaligned powerful AI)...

Some notes or pointers-to-sources on what OpenAI and Anthropic say they do:

OpenAI: they use RLHF.^[1] While working on GPT-4, they "used GPT-4 to help create training data for model fine-tuning and iterate on classifiers across training, evaluations, and monitoring"; see also Using GPT-4 for content moderation. "Risks & mitigations" in "GPT-4 Technical Report" discusses "Adversarial Testing via Domain Experts" and "Model-Assisted Safety Pipeline"; see also "Model Mitigations" in "GPT-4 System Card."

OpenAI ultimately has "Superalignment" goals, but they say "Our plan in the shorter term is to use AI to help humans evaluate the outputs of more complex models and monitor complex systems."^[2][3]

Anthropic: they use Constitutional AI. "Alignment Capabilities" in "Core Views on AI Safety" (Anthropic 2023) mentions "debate, scaling automated red-teaming, Constitutional AI, debiasing, and RLHF," at least as research topics if not mature techniques. Similarly, their paper Red Teaming Language Models to Reduce Harms (Anthropic 2022) uses red-teaming to create data for RL; it's not clear whether they use this technique in practice.

This is a broad question; narrow answers would be helpful, e.g. "tagging pre-training data based on human preferences and filtering out some content about AI (especially takeover)."

Thanks to Aaron Scher for suggestions.

Related: Which possible AI systems are relatively safe?

1. ^{^}

   Our approach to alignment research (OpenAI 2022) says "RL from human feedback is our main technique for aligning our deployed language models today." See also Aligning language models to follow instructions (OpenAI 2022) and GPT-4 (OpenAI 2023).

2. ^{^}

   See also "Training models to assist human evaluation" in "Our approach to alignment research" (OpenAI 2022).

3. ^{^}

   Aaron Scher left the following comment on a draft (slightly edited):

   Jan Leike says "in two years, we'd want to have a good sense for what are actually the techniques that we could use to align the automated alignment researcher. Do we have the portfolio of techniques that, if we did apply them, we would actually feel confident that we have a system that we can trust and we can use a lot, and then we can hand off a lot of work to?"

   I think another way to frame your question is: What if OpenAI had to answer that question this week instead of in 2 years, what is the current answer to the question?

   I think it's notable that their meta-plan is to first do this convergent thing of figuring out what techniques they can use.