One of the main people involved with setting up tailwind here! I feel like there's some very spherical cow thinking going on when you say that funding research aimed at solving alignment is just itself an attempt to solve alignment, and that if I don't know how to solve alignment, then obviously funding research to solve alignment is doomed (I’m sorry if I’m mischaracterising your position).
To port that over to the perpetual motion analogy, that's like saying that it's a dumb strategy for the person who wants to solve perpetual motion to hire a bunch of the world's best scientists to try and build perpetual motion machines. Seems very reasonable that that strategy could unearth ideas for perpetual motion machines that the original person didn't have? Similarly, I'm hopeful that when we make grants, there'll be people who have ideas that I don't have, or come up with ideas that I haven't yet come up with.
I'm not expecting us to make any grants to teams of only AIs. You could call this a scalable oversight protocol if you like, but the key goal of this funding is to increase the number of humans involved in the safety work. You might be skeptical that it's possible to scale the number of people doing productive work on AI safety, but the problem of finding smart people and getting them the context so they work on the important problems is a different problem to the problem of aligning super intelligence, and it seems like you’re conflating them.
Your argument crucially relies on the claim that aligning human organizations and aligning AI systems is essentially the same problem, which could just as well be used to claim that it's impossible to design any large-scale organization which achieves the goals of its founders. The same analogy underlies "corporations are the real unaligned AI (so I cannot imagine AI being qualitatively worse)". In fact, if we grant the analogy this inference is better grounded than yours. It extrapolates from the structures that have appeared many times and we've observed extensively to the one which doesn't exist yet rather than the other way around.
To the contrary, I think funding the creation of a large-scale AI safety research community works in much the same way as funding any other kind of large research program, and is very much possible. It could very well fail, and indeed it could fail by being subject to principal-agent problems, but that too is still following the same patterns that all human organizations are subject to and are now and again overcome.
You also repeatedly emphasize that the researchers will be assisted by powerful AI. It's not clear to me what logical role this has in your argument, seeing as your opening claim
A scalable way to produce alignment progress (that is, one that produces progress arbitrarily far beyond what the organizer could produce through their own sweat and tears), that isn't a correct alignment solution in its own right, is another such obviously impossible thing.
as well as the first few paragraphs defending this claim, applies equally well to human-only research. That said, as I see it there are two possibilities: Either human beings are still substantially understand the research produced and meaningfully choose its direction, or they don't. If humans do have a meaningful say in the research, then they will be inclined to direct the research to being aligned to human values and the situation is meaningful different from a black-box AI. If they don't, then RSI is possible and we're already doomed, and it's unlikely the AI alignment research groups will be the first to foom. That an AI swarm has solved a Millenium Prize problem only says that we're closer to the second scenario.
In case you prefer the medium of heavy-handed metaphorical dialogues:
Alice: I'm going to create a lab with the biggest and most powerful machines, so that researchers could run the experiments they need to build a perpetual motion machine!
Bob: Ridiculous! How can you power this lab with a perpetual motion machine, seeing as even if the research does go as well you hope you won't have a perpetual motion machine until after the research is complete. You're using circular reasoning.
Alice: What? We'll be powering the lab with an ordinary power plant. We need to generate a lot of power, but I have no clue where you got the idea that we'll be powering it with a perpetual motion machine.
Bob: Then how come I don't see any coal plants in the specs for this lab?
Alice: People have recently invented other kinds of power plants that are more efficient and less polluting than coal plants. We use them because we think they're better power plants in general. This has nothing to do with our own energy research, we would recommend any other industry to also use these power plants. We're far from the first people to use these kinds of plants, they just haven't completely caught on yet.
Bob: Let's step back. You're the one trying to build a perpetual motion machine, how did argument end up being framed so that you sound like a reasonable person?
Alice: I admit my goal is far-fetched. If you had stuck to reasonable scientific objections to perpetual motion machines, perhaps you would have made me sound like a fool. But you insisted that the way we plan on powering our research machines is circular, when we're just powering it in the ordinary way that every other industry powers their machines. That's what's making you look bad.
I think this post raises an important objection specific to alignment research and talent pipelines that feed directly into it, but doesn't apply to other aspects of Project Tailwind that I think may be more robustly positive-impact.
Some such research directions from the website: AI for decision-makers, independent auditing/assessment, agreement verification technology, threat modeling institute, bio uplift demos.
There is a popular tradition on this website of clarifying complex ideas through somewhat heavy handed, metaphorical dialogues. TO WIT,
Alice says “I don’t have a good lead on how to build a perpetual motion machine, but I want to find one and am rich as all hell. Here is my plan for a research center that will study perpetual motion forever that won't have any power hookups or solar panels, because it's a clever research center that does not require any external fuel”
Bob says "One of your sentences is wrong. You either have a lead on perpetual motion design worth investigating or something stupid. I lean toward stupid."
Alice says "It sounds like you think perpetual motion is impossible due to some conservation law. However, there are holes, such as around dark energy, or the fact that energy isn't conserved in all large scale general relativity geometries"
Bob says "Your plan is either a perpetual motion machine, that we should investigate as a perpetual machine design and not a research facility, or stupid. This is unrelated to conservation laws or whether perpetual motion is possible, neither of which I mentioned. Your plan will not unfold as you expect unless you designed it with a perpetual motion solution in hand, which you say that you do not have. I know without looking at your plan that if I start at its HVAC compressor and start following electrical wires, I will find something incompatible either with reality or the claims you have made. You must be claiming to expect to be surprised by a solution to perpetual motion appearing during the facility's construction instead of during its operation"
Importantly, it is logically possible for this conversation to continue:
Alice says "Well I'll power my facility with this bowling ball sized rock I found that stays at 1000C no matter how I heat or cool it."
Bob says "Holy shit, lead with that! However, while I am surprised, I wasn't wrong: why did you claim that you did not have a lead on a perpetual motion machine design? Let's, um, go get you checked over for radiation sickness, but barring that we should absolutely put resources into investigating your rock- probably in a conventionally powered lab"
===============================================================
A scalable way to produce alignment progress (that is, one that produces progress arbitrarily far beyond what the organizer could produce through their own sweat and tears), that isn't a correct alignment solution in its own right, is another such obviously impossible thing.
The analogy is between a perpetual motion research organization that runs forever, and an alignment research organization that scales. The organization founder doesn't know how to set an intelligent system to stay focused on a specific goal as its mental capabilities snowball, so he builds a system of intelligences to ponder the question, with the full intention of that system snowballing.
I was pushed toward this line of reasoning as I noticed a repeating pattern: alignment research organizations are frequently trying for venture capital or effective altruism style scalability. Field building, mentoring, bootcamps, grants, benchmarking- these are considered exciting directions because people are faced with a big problem and want unbounded impact, and I fully cede that these are far more impactful than staring at a LateX document alone and thinking really hard. However, if you still need to do alignment research you are not capable of doing so while scaling! I picture a wildfire where the firefighters are only willing to fight it, not by hand digging trenches or wetting down brush, but exclusively by starting controlled burns, because it's so big that only such drastic measures are up to the task. I note that unlike the wildfires in our world, here the fire started as a controlled burn and no one has ever put out a controlled burn before.
Narrowing down from the larger view where I'm slandering mentorship and community (slander that I unfortunately have to stand by, since they have mostly built feel-good-flavored capabilities)-- Recently, a large number of people have put forth scalable plans specifically for converting money into alignment progress, and appear to be executing these plans backed by millions to billions of dollars. If you buy my argument that a scalable plan for converting money into alignment progress is an alignment scheme, then it should be judged as such: with intense scrutiny and a security mindset.
A specific instance of the problem is the recent launch of Coefficient Giving's project tailwind.[1] It is a remarkably scary instantiation of a classic plan:
1) Repeatedly put out calls for grant applications
2) Everyone choses whether to write an application, and whether to use AI (that can one-shot Navier-Stokes existence and uniqueness) to help them write.
3) Judge the applications by some criteria (optionally assisted by AIs that can one-shot Navier Stokes existence and uniqueness)
4) The winning grant recipients use the money to do alignment research, each applicant decides whether to be assisted by AIs (that can one-shot Navier-Stokes existence and uniqueness).
When viewed with clear eyes, this is obviously just a bootleg implementation of scalable oversight and I would argue it should be paused, in my impossible dreams indefinitely, or at least replaced with one of the vaguely plausible scalable oversight designs that have been examined though the lens of being risky and probably destructive. It immediately creates Amodei's country of geniuses, trying to get the money, who can think faster than you and try more approaches than you can think of. It's just that they're scaling-pilled human-AI centaurs winning mostly by parallelism instead of AIs in a datacenter winning by serial speed. Of course, if you do think science style grantmaking and publishing is already a solution to alignment (our miraculous hot rock), make that claim so we can debate that point instead of hoping that science as she is played will find a finished solution to alignment.
My object level prediction of what will go wrong with tailwind is pure extrapolation from history: when alignment research schemes seek to use ambient pools of itchy money to scale up their impact, they either fail to scale, or scale explosively while differentially producing capabilities. And then we impact-weighted average over a bunch of them. However, this is getting into the weeds of whether scaled grant funded/funding orgs are a good alignment solution, but all I'm asking for in this essay is to evaluate members of the broader class of scalable plans as alignment solutions before implementing them. I don't present ironclad evidence here on whether any specific scheme passes that evaluation. A positive prediction of what specifically will go wrong is not important compared to the diagonal argument that if you had any reason to suspect nothing would go wrong, that reason would be an alignment plan strong enough to obviate the need for the program.
Plausible plans should explicitly include why they don't scale their impact arbitrarily far past the plan-proposer’s individual effort, for the same reason that engine designs that don't claim to be perpetual motion machines always have limits to how long they can run without external inputs, limits that can be easily read off the schematics and design reasoning.
There is a tight analogy between a perpetual motion research organization that runs forever, and an alignment research organization that scales.
If I’ve gotten my point across, the answer to the following question will be clear (the thesis is a tool that answers this question among others) In my exposition, why is the reference class assistant AI used for writing grant proposals a swarm AI that cracked a Millennium Prize but costs million dollars per query, not an agent AI that can be subscribed to for $200 a month? What simple change could be made to the Coefficient program to force applicants to use the $200 AI? Why would other changes not have this effect?
Solidly on the "billion" side of "millions to billions of dollars." Don't let the rhyme fool you- a billion is rather significantly more money