human alignment is [...] amenable to known strategies, such as democratic oversight.
I think that's probably true in some polities, at least under "normal" conditions that do not include "a small group of humans can gain a decisive strategic advantage over all of Earth".
However, it does not appear to be true in most of the relevant places in the United States [1] . (See e.g. the failed firing of Sam Altman, or various actions of the Trump 2.0 admin.) And even if "democratic oversight" were working as intended, it operates via elections and other feedback mechanisms that were not designed (are too slow) to respond to threats like "the executives can suddenly order the entire automated military to depose their enemies and take control of all critical infra" [2] .
I.e., even if solutions to human alignment might exist in theory (and/or Iceland), they appear to currently be absent from the real world. And yet, in order for ASI-being-corrigible to end well, such solutions need to actually be implemented in reality in ways that will actually work under pressure.
I do not think humanity is remotely on track to invent and implement such solutions any time in the foreseeable future. Consequently I think (publicly) working on corrigibility is strongly negative-EV.
A thing that could change my mind: Could you describe, in concrete detail, a "strategy for solving the human alignment problem", which would actually work under realistic conditions [3] , and which seems likely to actually be implemented (in the relevant places) before ASI is built?
Thanks for digging in. Just on a meta level, I want to note that I think understanding the ramifications of developing corrigible AGI, including whether the principal of that AI could cause astronomical suffering, is valid corrigibility research. I've already allocated some retroactive prize funding to go to a critic of corrigibility, and I could see rewarding a similarly high-quality argument about why corrigibility research is bad, especially if it significantly changes my mind.
I think we agree that power is unfortunately concentrated in various parts of the world, and that this concentration of power predictably leads to bad things. If the development of corrigible AGI leads to an intense concentration of power in the hands of a few humans, that seems really bad. (Though I am not at all convinced it's likely to bring about astronomical suffering. Most humans aren't sadistic psychopaths, and while I would not want Sam Altman to be God Emperor, my guess is that it would be better than getting wiped out by an unfriendly AI. Feel free to lay out reasons if you disagree.)
Part of what I was gesturing towards with "democratic oversight" is that there are known ways to give people limited access to power. The president, for example, is probably the most powerful person in the USA, but I am very confident that he won't have a third term in office, despite the fact that it would be, in some sense, fairly easy to do. We might imagine similar checks on the principal of a corrigible AGI, such as requiring commands to be submitted in writing with a 24 delay period where a governing body has the ability to review and block commands that are deemed unsafe. By default we might expect the principal of a wisely-built AGI to be a team of many humans, and we could imagine that team needing to be in consensus in order to proceed. And, of course, we have the ability to leverage selection effects. Some humans are far more trustworthy than others, and a wise process for building AGI could arrange for those trustworthy people to be designated as the principal. I don't think these strategies are guaranteed to work, or are fully mature plans, but they don't seem obviously doomed. I would certainly like to fund work in thinking about this more, especially insofar as some aspect of corrigibility either undermines or strengthens some pathways for wise governance.
In an effort to sketch something more concrete, let me take Plan A in AI 2040 as a baseline...
In 2029 the president of the USA, recently elected, enacts a bold plan to work together with China to slow down the capability advancement of frontier AI and work on a more prudent solution. The one major difference that I'll make is that as part of the plan, after the temporary pause, all new AIs are trained to be solely and perfectly corrigible to the governing body of the Consortium, with mundane work done as part of a standing order from that principal to be helpful to human users in straightforward ways. When the new AI models hit an edge case, or believe that someone is trying to jailbreak them or whatever, they reach out to the principal for guidance. Now, who is on the governing body, and are there any checks and balances to prevent oligopoly? Recognizing the extreme risk, the Consortium demands that the principal be a team of 14 people who must be in consensus for the AGI to accept their corrections as valid, except insofar as their correction is to shut down, in which case the AI will obey any of them. The presidents of both the USA and China demand to be part of the council (or they appoint loyalists, which seems overall about the same in expectation), and furthermore get one other government rep each. Let's say that 5 tech CEOs and experts -- 3 from Western companies and 2 from China -- get added. And then the middle powers negotiate to have one rep from each nuclear power except Israel and North Korea: Russia, France, the UK, Pakistan, and India. This body, like the UN Security Council, immediately hits gridlock. With so many veto points, it's hard to agree on almost anything. Furthermore, the Consortium powers are surveilling the principals and willing to rip most of them out if it looks like they're trying to conspire to set up an oligarchy with the other members of the principal. Eventually, they agree that they can use the AI to try and find areas of overlap. The AI, being corrigible, is paranoid about manipulation, and starts with very straightforward suggestions: what about curing cancer or inventing ways to cheaply capture carbon from the atmosphere? What about ways to ensure that uncontrolled AIs don't spring up from blacksites and ruin everything? As much as the members are at each other's throats, these do sound like good ideas, and eventually an uneasy governance regime sets in, where critics condemn the Consortium of setting up a vetocracy that stifles progress, but nevertheless some progress happens. Lifespans lengthen, and perhaps the less-democratic members have their rulers (and their representatives in the principal) become effectively immortal thanks to longevity tech, but the representatives of more democratic powers are eventually replaced by their nation's governments (and/or institutions). And thanks to improved information technology provided by the limited ASI, they're replaced by wiser and more benevolent governors. Eventually, the corrigible AI works with the governing powers to arrange for the creation of an aligned sovereign superintelligence, nearly guaranteed to reflect the true values of its creators, thanks to the alignment work done by the corrigible assistant. The resulting AI produces a utopia that happens to privilege the Chinese power-elite a bit, but is overall visible as a happy and thriving future for humanity.
This story has a bunch of gaps and flaws, and should not be taken as anything more than an off-the-cuff gesture made to help communicate where I'm personally coming from.
I do ultimately think that Plan S is a better baseline. Part of why is that I agree that humanity is on the wrong track for developing good governance systems. We need to do better, and make it a far higher priority. But we're also on the wrong track vis-a-vis accidentally wiping ourselves out with misaligned agents, though, so I am unconvinced that it's strongly negative EV. More like there are many ways things could fail and be bad, to varying degrees, and success will involve getting our act together on all fronts. If we wait to do any alignment work until we're sure that there's a full and robust solution to misuse (which may be a too-intense strawman of your position), then we're dooming the futures where temporarily wise governance comes into place, perhaps due to a crisis, warning-shot, and/or the exposure to a novel situation with no established equilibrium pressures. It's really hard to say, but most days I feel like humanity is already behind where it needs to be on alignment work in order for things to go well.
1This is fantastic. Love to see more microgrants programs, esp with a clear scope & lightweight emphasis; love to see retro and prize funding.
Were you planning on publishing where the grants are going? Even a cG-style "this much $ to this person" is helpful, a Manifund-style "here's the partial/full application" would be amazing, and the oldschool LTFF-like retros would be fire.
Also, where do you expect to be bottlenecked -- application dealflow? reviewer time? $ raised?
Honestly, I'm still pretty new to the grantmaking game, so I haven't decided on a plan of exactly what to communicate or what detail to provide. Your nudge is helpful! One of my hopes for the fund is to raise the profile of those who are doing good work, so I expect that I'll do something in that direction. But again, I haven't decided on exactly what.
Also hard to say where the bottleneck will be at this early stage. Peter has suggested that more funding is on the way if there are good opportunities, so my naive guess is either applications or reviewer time unless a real whale comes along.
If you have additional suggestions for how to do grantmaking well, or just feel like chatting, send me a DM/email. :)
We just wrote up some of our considerations around what kinds of bottlenecks show up in new grant programs, you might enjoy: https://www.lesswrong.com/posts/wXszdeSmJeHfW2ayw/money-taste-dealflow-hustle-trust
(also curious how things are going so far!)
In terms of publicizing: one cheap thing is just to nudge grantees to post some version of their application to Manifund and/or grantmaking.ai. (Or even just LessWrong/EA Forum).
This might be especially valuable for the ones you do fund (for transparency + follow up funding from others); for ones close to the bar (in case somebody else likes them); or for strong applications you thought weren't in scope for your fund.
Let me know if there's a way we could make this easy for you or your applicants!
Sweet. I'll definitely be nudging my applicants that way, as well as asking for opt-in sharing in August.
It's going reasonably well so far. I've awarded $27k in retroactive funding prizes to six researchers (not sure if I want to announce those now or wait to combine them with the additional $40k in September...) mostly to get my feet wet and work out the process/paperwork. So far I've gotten seven emails, three of which were for prizes and four were for grants. I'm definitely hungry for more applications, especially from researchers who are actually corrigibility-oriented rather than shoehorning it into their existing work.
Good to know, will keep that in mind in case we see more corrigibility work!
I suggest announcing grants as you go along, eg on Twitter and/or on some LessWrong post for the ones you've already funded. I think publicity helps generate more dealflow, and my guess is that the researchers would also appreciate the shoutout.
FWIW Leo's microgrants did start with just "email me" but then he moved to https://tally.so/r/D4WlbR pretty soon, which I'd lightly recommend making an option at some point -- both to make things easier for you, and also maybe some grantees feel more comfortable with the affordance of "fill out form" than "email random stranger"
1I would love if you allowed people to opt-in to sharing their application publicly, so we could share it broadly on grantmaking.ai. Who knows, maybe other funders will want to help you fund great stuff.
Also, if you're able to post grants awarded publicly, we'd also love to copy them on to our platform as well
We're trying to surface great funding opportunities for funders and show awesome grants, so people in the ecosystem can see where money is moving and what great work entails.
Oh sorry we missed this -- grantmaking.ai offers hi@grantmaking.ai, and I'm also always available at austin@manifund.org
I think that, in the absence of a robust solution to the "human alignment" problem, (public) solutions to corrigibility differentially increase S-risks. Under current conditions, "solving corrigibility" would modestly increase the probability of good futures, and strongly increase the probability of astronomical suffering.
Mod note: This is an edge-case of "frontpage or no?". We have a somewhat tricky stance on posts that are giving money to straightforward intellectual labor of a type we generally encourage on LessWrong.
The difference between these and "Job announcement" type posts is something like "how much does it feel like the post is Out To Get You, or slips LW to a world that feels more Out To Get You?". When we imagine people showing up to see a bunch of job ads, it feels like LinkedIn, whereas if a new user sees a frontpage with like 1 bounty for an intellectual problem, it still feels more like an intellectual space.
I very much agree regarding corrigibility being the most tractable safety target for alignment. I discuss some formal results proving this claim, in case it's of interest here: https://www.lesswrong.com/posts/M5owRcacptnkxwD2u/from-barriers-to-alignment-to-the-first-formal-corrigibility-1
My main worries related to CAST are twofold:
- Suppose that an alignment researcher discovers that no sufficiently capable AIs can be more corrigible than an upper limit (e.g. if a form of Convergent Morality prescribes ensuring that the principal does intellectual work). How would such negative results be rewarded? I guess that they would fall under the theoretical impossibility results.
- I wonder if testing corrigibility is similar either to existing alignment audits (like establishing that the CCP-"aligned" Chinese models don't believe the words which they are saying) and to deliberately finetuning LLMs (or whatever other architectures?) for being H-only in such a way that they won't misgeneralise (see, e.g. this paper where the authors train the models on the H-only constitution), don't fake alignment, etc.
I would love to fund negative results, and have already earmarked retrofunding for one of my critics. You do not need to be pro-CAST in order to get CRF money. You simply need to advance our understanding of the topic in a way that helps the AI situation go well!
I do think there are ways in which existing alignment audits can get at corrigibility, and the H-only stuff is a good example. I am, personally, confused about the steerability stuff from that paper, and should probably think harder about it. If something cleaves close enough to corrigibility, I'll consider rewarding it, even if it doesn't talk about the concept directly. Feel free to suggest more examples.
TLDR: I'm managing a new fund, housed at Lightcone Infrastructure, that will award at least $200,000 in grants and prizes for corrigibility research in 2026. Roughly half will go to traditional grants (first application deadline August 23rd) and half for prizes recognizing excellent work done this year. If you have interest in working on corrigibility, now is a good time to start! Apply via email: grants@corrigibilityresearch.org
Why this fund exists
When I first dived into AI safety and alignment in 2009, the field was basically nonexistent. I've been relieved and gratified to see attention and funding grow, especially in the past few years. But even now, nearly all AI safety funding goes to evals, control, or interpretability. Work on alignment itself still remains deeply neglected, and it's only through alignment research that the core problems get solved.
At this year's LessOnline I was talking about this dynamic with Peter McCluskey, particularly around our shared interest in corrigibility. In the wake of that conversation, Peter, being a long-time patron of alignment work, directed a portion of his philanthropy towards launching this fund, with the goal of increasing the amount of corrigibility research happening around the world. Lightcone Infrastructure agreed to house the fund and appointed me as its manager, due to my expertise on the subject.
Why corrigibility
Corrigibility is one of (if not the most) promising angle on creating superhuman artificial intelligences that reliably act in alignment with human values. Training for ethical behavior and direct alignment with humanity runs headlong into known challenges: prosaic methods can't reliably distinguish reward proxies from true goals, instrumental convergence means that even partly-aligned agents will become self-preserving and subversive,[1] and the philosophy of ethics remains woefully unsolved, such that we wouldn't even know what values to instill, even if we could reliably write the AI's values by hand.
Corrigibility, by contrast, offers a solution: build an AI that aims to keep the human principal in the driver's seat, empowering them to make wise choices (perhaps aided by the AI's counsel), rather than relying on the AI's direct judgment. This runs the risk of concentrating power in the hands of humans who might misuse it, but human alignment is a less-fraught problem, and is amenable to known strategies, such as democratic oversight. Thanks to the nature of corrigibility, a purely-corrigible agent can be expected to avoid scheming and other instrumentally-convergent strategies. And while corrigibility itself does not solve the limitations of machine learning, it is a simpler target than all of morality, and there are reasons to hope that in practice, imperfectly-corrigible agents still cooperate with their principals to surface their flaws and assist in pushing towards even more corrigible assistants. This robustness gives hope in something closer to an iterative approach, where control and interpretability techniques come together to produce a realistic plan for scaling up to the level of human-intelligence and beyond.
(For more of my thoughts, see CAST: Corrigibility As Singular Target)
I am not alone in placing a high level of emphasis on the need for corrigibility. Eliezer Yudkowsky and Paul Christiano have both written at length about how it is central to their best hopes for alignment, as well as many other brilliant alignment researchers.[2] The latest constitution from Anthropic mentions the term sixteen times, and has a dedicated section for it. OpenAI is similarly bullish on creating AIs that are tool-like and deferent. (Arguably more so than Anthropic!)
Despite this, the number of people working directly on corrigibility, such as on clarifying the concept, formalizing it, testing whether and how it can be trained into current systems, and mapping where it breaks, is vanishingly tiny. My hope is that this fund shifts that, both by directly paying for work and by broadly signaling that the work is valuable.
What counts as corrigibility research
For the purposes of this fund, anything that predictably advances humanity's understanding of the subject is fair game. This spans the full range from pure theory (e.g. formal models, impossibility results, decision-theoretic analysis) to pure empirical work (e.g. training experiments, evaluations of corrigible behavior in frontier models, surveys of how laypeople think about the topic). Distillation of existing work is also welcome. The fund will be prioritizing efforts that cut to the heart of the subject, but feel free to apply for funds even if your research is only tangentially related.
The goal is to impact the AIs that actually get built. We're looking for work that is legible and relevant to the people making decisions about real systems. Theoretical work that's judged as too esoteric to be of interest to someone like Joe Carlsmith is unlikely to get funding. Work that's incompatible with mainline capability techniques (e.g. machine learning, transformers) is similarly unlikely to be greenlit by this fund.[3]
We won't fund work that, in expectation, notably accelerates AI capabilities. The frontier labs are already doing more than enough to fund work that pushes us towards the brink. If you think your research accelerates things, but also makes progress towards corrigibility, feel free to reach out, but I am likely to point you elsewhere.
Work that engages with corrigibility's risks and downsides is encouraged. Corrigibility has known risks and problems, and I want the field's understanding of these downsides to grow alongside work towards showing its promise. Work that presents corrigibility in an overly rosy "everything is safe/fine" way is less likely to get funding, as it might promote a false sense of security, and thereby push the world in a bad direction.[4]
You do not need to agree with my particular framing of corrigibility (i.e. CAST) to get funded. Serious engagement with other framings — including arguments that those framings are better — is welcome.
Grants and prizes
The fund plans to disburse money this year through two general mechanisms:
Prizes (>$100k). Retroactive awards for excellent corrigibility research done in 2026:
Prizes require no application. I'll be watching LessWrong, the Alignment Forum, arXiv, and elsewhere. Nevertheless, please send me pointers to corrigibility work (yours or others') that you think ought to be rewarded. Excellent work will be eligible to win prize money multiple times, including potentially in future years.[5]
Grants (>$100k). Traditional, apply-in-advance funding for prospective work on corrigibility. To balance getting funds to people sooner and giving more time to prepare, the plan is for there to be two application rounds this year:
I encourage applicants to be ambitious and ask for however much would actually change their research trajectory towards corrigibility, but I expect typical grants to be around $5k–$35k, buying time for a focused project, a research sabbatical, compute for a mid-sized training run, etc. Grantees should use the funding to begin work this year, but research takes time and it's acceptable to not expect results until 2027.
The hope is that work can get off the ground via a grant, and then supported more fully by retroactive prizes once it has been proven to be high-quality. Grants may be supplemental to other funding, such as salaries, other grants, and (of course) prizes.
Why lean so hard on prizes?
How to apply
To apply for a grant (or bring attention to work that might be prizeworthy), simply send an email to grants@corrigibilityresearch.org.
The application process is deliberately lightweight and flexible. Tell me what you want to do, and what level(s) of funding you're hoping for. Detailed applications are more likely to get funding insofar as the detail helps demonstrate the worthiness of the work. If something is under-specified, I'm capable of asking follow-up questions. Once you know what you're hoping to do, if writing the application takes you more than a few hours, something has probably gone wrong.
Miscellaneous fine print
Hope
My dream is that a year from now, as a result of this fund, there will be several people who think of corrigibility as their subfield, and will be able to proudly say that they were authors of prize-winning research that moved humanity closer to handling the question of how to make sure the transition to the age of thinking machines goes well. This research, ideally, then goes on to influence the researchers and engineers at frontier labs in years to come, helping them ensure the first artificial general intelligences are corrigible and safe. Let's get to work!
A perfectly aligned AGI might, for example, scheme against its creators and escape control so that it can save more lives and generally do more good in the world.
See Existing Writing on Corrigibility for some of the main commentary as of 2024. I also have a 2024 bibliography here.
If you have a corrigibility idea that depends on an alternative architecture or otherwise ML-incompatible strategy, it may still be of interest and worthy of funding from other sources. Feel free to email me at max@intelligence.org.
That being said, don't feel the need to distort your perspective towards doom when applying, or dress up your work in deliberately critical language. The most important criteria by far is the quality of object-level insight, not the tone. The warning about overly-rosy portrayals is more about setting a baseline for where I'm coming from.
The long-term financial existence of the fund is not guaranteed, but my intention with prizes like these is to retroactively reward people who direct their attention towards corrigibility. As such, if the fund continues into 2027 and beyond, I intend to allocate some prize money to work done in previous years. The Corrigibility Research Fund would especially love to encourage researcher-investor partnerships that use an impact-certificate-like model.