Almost nobody is funded to figure out what work would solve alignment

Seth Herd

Almost nobody is funded to figure out what work would solve alignment — LessWrong

71 Almost nobody is funded to figure out what work would solve alignment

4th Sep 2026

4 min read

71

Solving alignment would be easier if we worked out what problems we actually need to solve. This could be called the alignment meta-problem. Work on this problem is rarely directly funded. More focused work on it should let us use our limited time and funding more efficiently.

The diagram implies narrowing alignment work, but I expect meta-problem work to also identify high-payoff "fringe" approaches.

If we're driving toward a cliff, maybe we should buy better headlights.

All too often we're doing work that merely sounds or feels good, and optimizing less than we could for work that drives most efficiently toward success. Some of this is inevitable and some of it is useful, but we could do more to light the path ahead.

Edit, in response to early comments which revealed I wasn't being clear: the meta-problem as I'm defining it includes any means of identifying more and less useful work, but it would probably mostly focus on two elements. The first is predicting the path to takeover-capable AI, including its design, how it's deployed, and the societal/organizational/governmental actions that will constrain alignment solutions. The second major element would be mapping from those predictions to likely-most-useful work. This mapping would include identifying the hardest parts of the problem to receive more work, or to somehow route around (e.g., slowing stopping if the hard problems are practically unsolvable).

Most researchers agree that mech interp, refining and improving alignment training, control, theory, and miscellaneous techniques like confession are useful for solving alignment. Working toward regulation and slowdown/pause is also commonly considered useful in the governance space, and spreading awareness of alignment risks is pretty obviously useful for accelerating and enabling all of this work. But we don't know what variants of this or other work make best use of our limited time and funding. Which of them, or what others, are the most efficient use of resources? I think that we could probably get usefully better answers at relatively low cost.

Currently work on the meta-problem is done either in researchers' spare time, during grant applications, or inside funding orgs and AI developers. All have limited incentives to publish their thinking legibly, and incentives during grant applications and inside dev orgs are subject to Goodharting and motivated reasoning: sounding good vs. being good. More funding directly for analysis and planning seems like an efficient leverage point.

The AI Futures Project is the exception that demonstrates the trend. They are funded to do prediction work, but that spreads into many aspects of the meta-problem. They make Gears-Level Models of how alignment and governance work might succeed or fail, and so identify work that's likely to be more and less useful. They have the time and focus to explicitly include the many cross-dependencies between governance, public opinion, and technical alignment work, in contrast to people who do this in their spare time and mostly focus on their own area of expertise. (Other orgs and people do aspects of this work, but arguably do less direct work on the meta-problem as I'm thinking of it.)

Many researchers and orgs do some of this work despite it not being directly funded, because they think it's so valuable. And people at funding orgs do meta-problem work as part of their job. But publications of funders' models tend to be short and limited (although kudos to the funders who do carefully work out and publish their theories-of-change!).

Why not to fund more work on the meta-problem

There are good reasons not to fund this type of work more heavily. In brief:

Science doesn't typically do this, for good reason
Could concentrate funding, neglecting important work
- Bad predictions and work recommendations are worse than none
This work is being done adequately through indirect incentives
Maybe what we're doing is the right work, particularly for short timelines
Could waste money on useless or bad work that's hard to evaluate
Prediction is hard, particularly about complex and novel scenarios

These are all important points. Each is something to guard against, or a reason to not over-fund the meta-problem.

Arguments in favor, compressed

Each of the arguments against suggests an argument for:

Unlike other sciences, alignment has a specific goal and time pressure
- Like planned projects such as Apollo, Manhattan, WWII
- (And perhaps other sciences would benefit, too; observing cognitive neuroscience and its funding for 23 years has led me to think they would)
Over-concentration of funding is more likely if there is less/worse meta-problem work
- Over-concentration could be happening now, since only a few people and orgs are funded to do this work
- We're already making predictions and work recommendations, just rarely with much time-on-task
- Could draw more funding into the field by clarifying what's probably useful
While a lot of this work is being done, most people are doing it in small pieces, and are incentivized and have Motivated reasoning toward justifying their own work
Maybe we're already doing all the necessary work; why not rigorously check?
- Theory, and gears-level models of the transition to LLM AGI (e.g., AI 2027, LAMRAG) raise substantial doubts
Because it's hard to evaluate this work, streetlighting steers away from it
- It's also arguably an Illegible problem in Wei Dai's usage
Prediction is very hard, but also very useful.
- Actually funding people to do it and publish it full-time might be better than everyone just doing little pieces in their spare time

This post is a teaser for a longer, more worked-out draft that's been stuck in my queue. If responses here change my mind or indicate that No One Cares, it will probably stay unfinished. This is pretty telegraphic; I can elaborate, give examples, and clarify in the comments.

All questions, feedback and suggestions are appreciated!

Disclosure: Much of My research has recently been on aspects of the meta-problem. This is a result of increasingly thinking such work is even more neglected than the object-level technical alignment work I started out doing. I feel very lucky to be able to devote time to this type of work.

Disclaimer: I'm not deeply familiar with all the orgs, and I'm sure I've overlooked orgs funding overlapping work. Apologies to those I've missed.

Thanks to Peter Gebauer and others for many conversations touching on this claim, and Peter for comments on an earlier draft. This framing was inspired when a donor friend said "I might fund more alignment work if I could tell what would be useful," causing me to say "Maybe we should be funding figuring that out!"

Frontpage

71

New Comment

20 comments, sorted by

top scoring

Click to highlight new comments since: Today at 9:35 PM

[-]johnswentworth1mo5519

I think this is missing the main reason to not devote more resources to the meta-problem.

One of the main patterns in alignment work is that people run into the hard parts of the problem, where it's hard to even think clearly at all or find any hint of traction, and then they mentally "slide off" to something which feels easier. I claim that "go meta" is one of the most common ways that people slide off. Problem is, the meta problem is not actually easier than the object-level problem. It feels easier, because it's further removed from concrete challenges and therefore easier to not-notice the ways in which one's vague abstract ideas don't match reality. But in actuality, if someone is struggling to even keep their attention on the object-level problems, they're not going to be able to make actually-useful progress on the meta problem.

[-]Gunnar_Zarncke1mo164

This is true, but misses the point: It is very true that you can slide easily on meta problems. But if nobody attacks this specific meta-problem, it is quite likely that funders wont allocate resources to research that is needed but overlooked; people entering the field will be working on things that are not on the critical path; and researchers will not connect areas that have overlap or dependency.

[-]Seth Herd1mo*158

If your point is that individuals may be spending too much time on the meta-problem, despite not being funded, I agree. I think that's separate from whether we should fund more full-time work. I think that would actually address this sliding-off-the-hard-parts issue.

I agree that there's a tendency to slide off the hard parts, including into the meta-problem. But the meta-problem definitely includes accurately identifying the hard parts, so that fewer people keep sliding off them in the future.

But the more common direction of sliding off the hard parts these days is into empirical work.

One of the main points of increased funding for the meta problem is to address exactly this failure to meet the problem at its hard parts. Good, focused work and publication on the meta-problem would make the hard parts more apparent.

You and others think the hard parts are apparent. Many prosaic alignment researchers do not agree. Better work on that disagreement would help resolve it. So it seems like we should fund that work.

So: (edit) would you really object to funding a few more people to work on the meta-problem? I would've thought you'd support funding some aspect of what I mean by the meta-problem, since that's how I read much of your contribution to the field. Your recent post on streelighting, and slop not scheming as the median doom path was two top-notch pieces of work on the meta-problem IMO. They're widely cited and guiding work in better directions, exactly what I mean by work on the meta-problem.

[-]johnswentworth1mo125

You and others think the hard parts are apparent.

I don't, actually. I don't think anyone has ever even come close to conveying the hard problems to people who do not already have at least some sense of them. A few write-ups that I think made it farther across the gap than usual include Eliezer's List O'Doom, the Embedded Agents sequence, and my own What Is The Alignment Problem? and The Plan posts. But none of those really get the whole picture across. And even before people are ready for those, most people need to pick up at least some simple prerequisite concepts of the sort The Sequences covered, though different people are missing different prerequisites and we also don't have a single source which covers all the prerequisites.

So my actual take here is not exactly "don't fund people writing about the hard problems of alignment". If e.g. your plan was to approach people who already seem to demonstrate substantial understanding of the hard parts of the problem and offer them money to write a few posts posts trying to convey marginally more, I'd say that's a reasonable plan. On the other hand, I would not recommend trying to e.g. hire newly minted MATS scholars to do this sort of thing. It is important, but decidedly not one of the more tractable directions.

[-]Seth Herd1mo73

So I think we're mostly in agreement here, if we restrict the proposal to funding people with the capacity to actually make progress. You need to understand the whole thing pretty thoroughly before your work on the meta-problem is going to be useful.

I like your take on understanding the hard parts. You can frame it as people being stupid and stubborn, or you can frame it as the problem being genuinely hard to convey. One seems obviously more useful, and I read your comment as taking the second. I do try to keep that framing in my own mind.

I'm envisioning RFPs and competitive funding, which do help select for good people and good work if they're written and evaluated by thoughtful people.

I think it's worth noting that you and Eliezer have had flexible enough funding to spend substantial time on the meta-problem. I think that's a big reason you've both done good work on it: you've had time to really think hard and read a lot about it. My point in asking for more funding is to get more work like that, and more work building on it.

I think time-on-task is an even larger predictor of performance than innate intelligence (I think this is apparent from the history of intellectual accomplishment and from the cog psych literature; another stalled draft makes this argument in more detail). That's why funding buys capability. Given that, it's still better to fund people with good starting knowledge and good innate intelligence if the work might wind up directing the field.

I sure don't want a MATS scholar directing the field. It does seem like putting actual funding on this issue would raise the waterline and spread understanding of the hard parts of alignment. Making understanding the big picture a funded and thereby respectable goal should make beginners more interested and informed.

Identifying the hard parts and explaining clearly why they're hard is exactly what I mean by the meta-problem.

[-]jacquesthibs1mo*60

It sounds like you were imagining a weak version of what Seth is proposing, and that you agree with the strong version he also would approve of?

That said, I agree with the general comment and I have the impression that past attempts by folks who did not spend the hard time doing object-level work end up plausibly making things even worse by pointing at entirely the wrong things.

[-]Chris_Leong1mo110

I resonate with this.

Alignment felt hard, so I concentrated on community building, but one thing I've realised over time is that doing community building well doesn't remove the need to place bets on what solving the problem is likely to look like.

[-]Seth Herd1mo*80

Figuring out how to best "place bets on what solving the problem will look like" IS exactly the meta-problem I'm talking about.

Community-building is not part of the meta-problem as I'm defining it, although I certainly think it's valuable work.

If you mean we've got to actually place the bets, yes and we're already doing that, just (to be a little provocative) without thinking about it all that carefully because we're not bothering to pay anyone to do it.

[-]MichaelDickens1mo62

FWIW I would describe a statement like "people slide off the hard problems and spend too much effort on the easy problems" as discourse on the alignment meta-problem. You need to be thinking on the meta level to come up with statements like that. Devoting more resources to meta work might lead more people toward beliefs like the one you've just expressed. (Or it might not, depending on the nature of that meta work.)

[-]cousin_it1mo*1711

If we’re driving toward a cliff, maybe we should buy better headlights... Most researchers agree that mech interp, refining and improving alignment training, control, theory, and miscellaneous techniques like confession are useful for solving alignment... Which of them, or what others, are the most efficient use of resources?

In a naive sense, the question answers itself. If we're driving toward a cliff, we don't need to buy better headlights - we need to stop driving toward the cliff. If everyone stops working on AI, that will free up resources, not use them. Then they can be used for other good things.

Whereas if everyone works on the things you mentioned, like mechinterp, that will use resources and speed up the race. Similar to the development of early alignment techniques like RLHF. So that's bad.

So my non-naive answer is: the most efficient use of resources is pushing for a stop to AI and recruiting the next guy to do the same.

[-]Seth Herd1mo62

I think our difference in opinion is driven by differences in our beliefs about our odds of stopping in time, and our odds of succeeding at alignment if we do not stop in time. If we either can stop or can't succeed if we don't, trying to stop is the only option. If we can't stop, but can succeed at alignment, getting alignment right is the only option. Intermediate estimates of both chances lead to working for both. This raises the very real conflict you mention: working on alignment also aids capabilities (and prevents warning shots), reducing the odds of stopping.

So there are difficult decisions to be made, based on estimates of very complex situations. Both our chances of stopping in time, and or chances of succeeding at alignment, require predicting outcomes of highly novel and unique futures (working AGI/ASI and the state of the world considering building it).

I submit that anyone who's highly confident in their estimates of either scenario is quite wrong. We just don't have enough analysis to be confident.

So the question is how we make good estimates and decisions under substantial uncertainty.

The below is my derivation of the above and some more detail.

I think the sane thing to do if it's likely we're driving toward a cliff is just what you say: stop. And if you can't, start building brakes like crazy.

But you also have to factor in the chance that you are not going to be able to stop in time. I personally do not want to give up on worlds in which we cannot stop in time. This puts me in dilemma: do I advocate only for stopping, or also for doing alignment work that has an actual chance of succeeding?

I don't think getting everyone to stop working on capabilities-relevant alignment is a realistic option. Working in that direction is, and I now say "obviously we should stop" alongside every suggestion for alignment approaches that might actually work in the short timelines where we can't stop in time.

But I feel very uncertain about this. This is not an easy question to answer given my current beliefs.

I think this is also a microcosm of the conflicted state of the field of alignment (even the subset of that field looking at the big picture and understanding the risks).

I think my personal indecision reflects the indecision of the field as a whole, since I'm deferring heavily to other experts in both the optimist and pessimist camps. (I think the Yudkowskian/rationalist standard of deferring to essentially nobody is pretty obviously bad epistemics, although that story is complicated.

I don't really expect a response; this is just me working through some important issues aloud.

[+][comment deleted]1mo20

[-]draganover1mo*90

Hmm, I don't know that I agree with this statement:

> I think that we could probably get usefully better answers at relatively low cost.

I think it is easy to convince oneself that some work is better to do than other work, but it is hard to reach actually sound conclusions about "X is obviously the best thing to work on". The way you get the clearest signal is by trying the idea and seeing where reality pushes back on your efforts. Indeed, I'm on a team doing alignment research and we are constantly trying to "understand what work would solve alignment". This is just hard to do! (especially under the time crunch.)

Edit: I think John's comment makes the point I was trying to make better than I was able to make it

[-]Seth Herd1mo60

Absolutely we need to guide work by empirically seeing what works!

I see little chance we'll stop doing enough of that. The suggestion here is just that we fund a little more abstract prediction/theory of what would be valuable if it works. A little more seems like a reasonable split, since it's currently a tiny amount relative to funding for empirical work.

What would you guess is the fraction? I'd say spending on the meta-problem is maybe on the order of .1% of total alignment funding?

To clarify: by "usefully better answers at relatively low cost" I didn't mean vastly better at trivial cost, just a good expected return on investment.

[-]Ashe Vazquez Nuñez1mo80

I'm a bit confused about the framing here. My conception of research is that it's mostly the art of thinking about (on a meta-level) what the right research questions are. Indeed, progress in research often feels like a process of asking increasingly precise questions as opposed to answering them.

Consequently, isn't meta-strategy baked into strong research culture?

[-]Seth Herd1mo64

It seems like this falls under the category of, "maybe the traditional science approach has this covered"?

If so, the answer is that that is not how things like the Normandy invasion, Apollo project, and other large goal-directed projects succeed. They would've failed or taken ten times as long if nobody was thinking full-time about the overall plan.

[-]Taylor G. Lunt1mo72

The mech interp people think mech interp is what will save us. The personas people think personas will save us. What would meta-level research accomplish that could tell us who is right in advance, other than just picking a side using the same subjective judgment the researchers are already using to pick sides?

To me, doing meta-research to try to adjust the balance between different existing research methodologies seems useless. Less useless would be trying to uncover new research areas. That's really difficult and requires deep expertise.

Let's say the balance of research is off, and there are 2x too many people working on mech interp, and not enough working on corrigibility stuff. Trying to re-divide that pie might not be a good use of resources given that the current pool of AI safety researchers represent a tiny fraction of humanity's geniuses (and the number who are Actually Trying is even smaller). For every AI safety researcher, there are 1000 geniuses working on AI capabilities, doing engineering, writing novels, practicing law, or starving in poverty in some poor country. Given that discovering new research areas (or progressing in existing ones) is gated by deep knowledge and IQ, maybe we would do well to try to recruit the other 99.999% of humanity's geniuses to the task.

With well-made educational resources, how long would it really take to get a non-AI-researcher genius who knew high school math up to speed? I'm not talking about the existing path of university, which wastes at least 95% of its time teaching irrelevant trivia. Like, to learn about AI, first you learn about ML, and to learn about ML, first you learn statistics. But consider an intro statistics course, and how much of that you really need to know to learn ML. Do you really need to learn about p-values and chi-square tests and ANOVA? Do you really need to learn about convolutional neural networks and random forests in your ML class to start doing AI/AI safety research? What would the minimum path be to getting up to speed on AI? I suspect it would be measured in months, if someone could do a good enough job putting the resources together. And once they're up to speed on the foundational stuff, they can then branch out into research areas they prefer and continue their own learning, having skipped 95% of the lengthy education they would have needed to get that far. And there are a lot of geniuses out there, and people are passionate about AI. As you said, most meta-research gets done in people's spare time. What if a significant fraction of the world's geniuses, even 1%, worked on AI safety research in their spare time and shared their work as a community? What research areas might be uncovered?

(You might say that learning ANOVA or whatever might not be directly relevant to AI safety research, but it contributes to some kind of research taste, judgment, knowledge, whatever. I would respond that general intelligence cannot be taught in humans, so all "research taste" and "judgment" and so on actually are is IQ plus your knowledge of the problem domain. And knowledge you acquire closer to the problem domain has a much higher likelihood of being relevant. If you do mech interp, other mech interp papers you've read are most likely to be relevant, and ML/math topics that are close by conceptually, then math/ML topics that are farther away, then stuff that's clearly not likely to be relevant like ANOVA, or the mandatory anthropology course you had to take to satisfy your degree requirements. Getting people as close to the problem as quickly as possible and then letting them learn in whatever direction they feel is important from that point on seems like the optimal strategy. The existing university strategy is moreso optimized to be a form of class warfare whose duration is a feature rather than a bug.)

[-]Gunnar_Zarncke1mo60

Before you can make predictions about the route, you need to know precisely what the routes are. Otherwise, you are in the position of a traveler in unknown terrain without a map who doesn't know if the winding path by the river is getting him home quick, or the one over the hills.

I think one of the gears you need to determine the route is the dependency structure between AI alignment problems I recently laid out in The Open Problems of the AI Alignment Field and their Cruxes. It is not a map that tells you how long the routes are, but at least you have an idea what the joining and forking points are and who is positioned to measure the length. It let's you answer questions like

what is the underlying crux this research is addressing?
is this research addressing a crux on the critical path? (eg in terms of resources allocated)
can specific research efforts be combined?

Does this sound like a gear you want?

[-]Seth Herd1mo60

Yes, I read that post and gave it a big upvote! That is exactly the sort of work I mean. I think I had some quibbles, but only quibbles, and public discussion is the point of carefully published work on the meta-problem.

If you did that on your own time, thank you! What I'm arguing for here is paying more people with your depth of understanding to do similar work we'd be improving our odds.

[-]StanislavKrym1mo60

Most researchers agree that mech interp, refining and improving alignment training, control, theory, and miscellaneous techniques like confession are useful for solving alignment. Working toward regulation and slowdown/pause is also commonly considered useful in the governance space, and spreading awareness of alignment risks is pretty obviously useful for accelerating and enabling all of this work. But we don't know what variants of this or other work make best use of our limited time and funding. Which of them, or what others, are the most efficient use of resources? I think that we could probably get usefully better answers at relatively low cost.
Currently work on the meta-problem is done either in researchers' spare time, during grant applications, or inside funding orgs and AI developers. All have limited incentives to publish their thinking legibly, and incentives during grant applications and inside dev orgs are subject to Goodharting and motivated reasoning: sounding good vs. being good. More funding directly for analysis and planning seems like an efficient leverage point.
The AI Futures Project is the exception that demonstrates the trend. They are funded to do prediction work, but that spreads into many aspects of the meta-problem. They make Gears-Level Models of how alignment and governance work might succeed or fail, and so identify work that's likely to be more and less useful. They have the time and focus to explicitly include the many cross-dependencies between governance, public opinion, and technical alignment work, in contrast to people who do this in their spare time and mostly focus on their own area of expertise. (Other orgs and people do aspects of this work, but arguably do less direct work on the meta-problem as I'm thinking of it.)

How could we even make progress on the meta-problem? For example, we had Anthropic claim that not even Claude Mythos 5.1 provided a 2x acceleration on their R&D projects, while Kwa demonstrated disbelief back in July, moved to OAI to construct RSI evals and Ngo ended up claiming that "measures and models of RSI are the kinds of thing which are near-optimal for propagating "feeling the AGI" inside OpenAI" and that Anthropic would have gone twice as fast if it propagated feeling the AGI and OAI would have quadrupled the acceleration rate. In circumstances like these, I would expect that the main technical meta-level work is to resolve as many object-level disagreements as we can.

The main governmental meta-level work seems to be either already done (e.g. AI-2040's Plan A and the doorstoppers on strategy, out of which I would throw away the sloppier parts) or consist of finding the best strategy for awakening politicians and/or promoting the awakened ones onto relevant positions (think of Bores' failed campaign to become the congressperson).

[-]Gunnar_Zarncke1mo40

This framing was inspired when a donor friend said "I might fund more alignment work if I could tell what would be useful," causing me to say "Maybe we should be funding figuring that out!"

This is actually the subject of a grant proposal I recently submitted: Alignment Crux Map (also here). This would let funders figure out which underlying crux an AI safety research project addresses; for example, when the label (e.g., corrigibility) is ambiguous, which risks funding what the project seems to claim it is doing vs what it is actually doing.

Is this the question of how to allocate funds you had in mind?

Moderation Log