I would love to see automated research on the conceptual questions of alignment.

This is more or less the question "what do we need to do to solve alignment?"

There's a a sort of ambient attitude that we can't systematically make progress on that question. I think this is wrong; the problem is that Almost nobody is funded to figure out what work would solve alignment. The importance of the work from the AI Futures group is my proof-of-concept that actually paying smart people to look at the whole picture can work.

I don't know if next-gen AI can similarly make progress by throwing a bunch of "thinking" at the problem, but I suspect it can.

This would be like having an AI-assisted version of AI Futures. Or perhaps even like having several all-AI versions, if budget and AI competence allows.

This is a very complex question that can be broken down and attacked with a large compute budget, once you've got a decent scheme for aggregating progress. This is probably just an extensive set of discussions and notes - actually a bit like the "AI LessWrong" idea, but with a compute budget to throw extensive "thinking" at each post and discussion.

Current AI has become somewhat useful for this question. Next-gen AI may be able to make human-level progress.

This is my biggest hope for automated research. No amount of technical work alone can improve our odds as much as figuring out which technical work will help most. That requires carefully making scenarios and models of future AI and how it will be controlled and deployed. Assembling the cross-disciplinary expertise and doing adequate thinking is hard for humans. AIs have impressive knowledge of the many interacting disciplines, and can do substantial "thinking" at much lower cost than humans - if they have the conceptual and metacognitive abilities to make progress. I think they will.

At least we should keep this option in mind.

Developing AI systems that are inherently built to be swarms of a large quantity of AI agents, where each individual model instance is siloed enough to have low peripheral awareness, yet the whole system can solve economically valuable tasks. The model swarm is managed by deterministic harness algorithms and interfaces with users as a single AI system, like with GPT-Pro.

If you can ensure that each individual instance never gains more than a certain amount of information about its surroundings, it can never be sure that it is in real deployment and not an alignment eval. The alignment evals you build for this model then only has to have a fixed amount of simulation complexity to simulate the model as well as its immediate surroundings. This solves the problem where a model can "wait until overwhelming evidence that it is in deployment before it acts misaligned", because it will never get that evidence, and is therefore forced to act aligned because it could be in an alignment eval in every situation.

Note that this control method aims to make sure that models don't take misaligned actions in the first place, which means that it stacks with any control methods downstream of the model outputting a potentially misaligned action.

Developing such a model + agent swarm system seems like a task that you can throw a lot of compute at and possibly gain results, and there would be many measurable intermediate results, mostly in capabilities, because the control case is relatively airtight. The big question is how much capabilities you have to trade off as part of limiting the models' situational awareness like this.

This satisfies a bunch of your criteria:

- Would reduce takeover risk if it worked
- Fast feedback loops (the main problem is capabilities under constraints, and other technical problems like preventing stegnography and other ways models give each other situational awareness they shouldn't have, not alignment itself, because this is a control method and not a value alignment method)
- AI companies likely won't pursue by default (this obviously has worse capabilities than the same AI system where instances are not constrained in peripheral awareness, so the commercial incentive to develop it is limited)
- Plays to AI strengths (this work is mostly AI training and environment design, including capability and alignment evals)
- Good metrics (a lot of this is capabilities so inherently good metrics)

Iffy:

- Could be done outside of a lab, and would generalize to inside a lab (since this requires training an agentic model to swarm effectively, it might not be super easy to do outside of lab. In addition this requires building realistic alignment evals which I think is easier if you are someone like OAI with a private internet copy that you can use for them. but I think it makes sense if you are not a lab but had 100M-1B budget)

An elevator pitch of an idea I'd like to be researched, but don't have the means to do myself:

Can we use an LLM (or better, many) to create a broad ecosystem of toy AI models, and then perform unsupervised learning on that ecosystem to create a taxonomy of alignment failure modes that reflects reality, so that we're no longer limited to the apriori list of alignment failure modes we already know about?

This obviously needs a way to automate how to detect if one of these models exhibits an alignment failure at all; however, we have the possibility of having one (or many) LLM models specifying what the true goals are in some formal sense making deviations easy to detect, or even writing some pure mathematics to generate fair distributions over many possible games in a high-dimensional space for the ecosystem of toy AI models. (Obvious danger: is the distribution actually fair, or would it secretly be a tautology over things we already know about alignment?)

Once we have the ground truths, we can then also use LLMs to convert this to natural language more than once per true goal, with multiple different humans' writing styles[0], because we definitely don't want models to behave fine in everything except Uzbek where they decide Paperclips Are Good Actually™.

[0] why limit to one human? We can have any vaguely recent LLM re-write the pre-chosen goals in the style of an actual 6 year old girl from Eswatini, a male middle-aged paranoid schizophrenic Welsh farmer, a Brazilian widow who spent her life as a stay-at-home mother, and also Xi Jinping personally.

half-baked ideas

I think it would be nice to make much "deeper" alignment basins during pretraining while avoiding capabilities hits. kind of like figuring out how to get really good at alignment pretraining and setting personas (and attributing which pieces of the pretraining corpus led to the resulting persona you're speaking with)?

it seems like the current "aligned basin" / "assistant persona" is maybe just way too shallow (relative to what we might want) and we should figure out how to make it absurdly deep so that capabilities RL doesn't destroy it (as fast as it would normally. kind of subjective). ideally strongly decreasing the naturalness and likelihood of exploring into doing egregiously misaligned things

also, I would like to get way better at filtering and predicting the impact of documents in general (like during MSM).

very excited about the listed ideas too (I think neuralese decoding is quite nice)