I think you should have stated the point more forcefully. It's insane that we don't have alignment prediction markets (with high liquidity and real money) given
There should be a group of expert forecasters who make a (possibly subsidized) living on alignment prediction markets! Alignment researchers should routinely bet thousands on relevant questions!
There's a huge amount of low-hanging dignity here, I can vividly imagine the cringing in Daith Ilan right now!
I agree, but it’s literally illegal to have real money prediction markets in the US on anything but finance and maybe elections. The only realistic paths are getting it legalized, building an actually nice to use and not scammy crypto prediction market, or accepting legal risk like you’re early Uber
Super interesting, thanks for writing this!
I work on Manifold, and one of the motivations for building the site is to play a roll furthering AI safety. One simple path is using prediction markets to make it common knowledge that AI capabilities is advancing rapidly. They can help us plan or warn us about potentially bad outcomes coming up. This is roughly the AI timelines question though.
So I'm glad to also hear that you might find prediction markets useful for concrete research questions or judging alternative proposals within the AI field.
Let me know if you think of any way the Manifold team can help out here!
One simple path is using prediction markets to make it common knowledge that AI capabilities is advancing rapidly.
Yeah, I was wondering if someone was going to bring that up. That seems to me like a hammer-in-search-of-nail kind of mistake. In the context of today's society, prediction markets aren't the optimal tool for convincing people of things or building common knowledge or the like; most people don't trust them or even pay attention.
Within LW and the immediately adjacent community itself, there's nonzero value in aggregating peoples' timeline predictions and turning the aggregate into common knowledge, but given that only people near-adjacent to the community will pay attention the returns to that sort of thing are limited. (Also marginal returns for markets on the topic diminish rapidly; I expect they saturated some time ago.)
Let me know if you think of any way the Manifold team can help out here!
At the moment I think the ball is mostly in the users' court, and I'm hoping this post will spur people to create some more-useful questions. (Though of course the Manifold team could try to make that happen more proactively - e.g. you could talk to some of the MATS participants about questions that come up when they're thinking about what to work on, probably there's lots of potential material there.)
Thanks for the feedback! If I understand you correctly, these markets would be more helpful, is that right?
They're closer. Eliezer's is mildly decision-relevant: if I thought that we'd have that sort of capability advance (independent of my own work), then I might assume other people will figure out interpretability and focus on things orthogonal/complementary to it. Bionic's could be decision-relevant in principle but is not particularly informative for me right now; I already have a decent intuitive sense of how readily-available funding is for alignment work, and beyond that total spending forecasts are not particularly relevant to my strategic decision-making. (Availability of money is sometimes decision relevant, e.g. about 18 months ago I realized how abundant funding was for alignment research and decided to allocate more time to training people as a result.)
But neither of these currently makes me go "Oh, apparently I should do X!" or "Huh, people are really optimistic about Y, maybe I should read up on that" or "Hmm, maybe these forecasters are seeing a problem with Z that I haven't noticed". They don't make me update my research-effort allocations. Maybe there are people for which one of these markets would update effort allocations, but the questions don't really seem optimized for providing decision-relevant information.
This post only indirectly touches on Metaculus, but I found nostalgebraist's criticisms of that site pretty convincing. Do you think the general site (rather than their separate forecasting tournaments) can be useful for alignment work despite that?
Yes.
Those criticisms (like most criticisms of prediction markets) are ultimately about the incentive structure and people which produce answers to the questions. One way to frame the core claim of this post is: the main bottleneck right now to making predictions markets useful for alignment is not getting good answers, but rather asking relevant questions. Maybe answer quality will be a bottleneck later on, but I don't think it's the limiting factor right now.
Definitely agreed that the bottleneck is mostly having good questions! One way I often think about this is, a prediction market question conveys many bits of information about the world, while the answer tends to convey very few.
Part of the goal with Manifold is to encourage as many questions as possible, lowering the barrier to question creation by making it fast and easy and (basically) free. But sometimes this does lead to people asking questions that have wide appeal but are less useful (like the ones you identified above), whereas generating really good questions often requires deep subject-matter expertise. If you have eg a list of operationalized questions, we're always more than happy to promote them to our forecasters!
Yeah, I definitely think Manifold made the right tradeoffs (at least at current margins) in making question-creating as easy as possible.
If you have eg a list of operationalized questions, we're always more than happy to promote them to our forecasters!
My actual hope for this post was that a few other people would read it, write down a list of questions like "how will I rank the importance of X in 3 years?", precommit to giving their own best-guess answers to the questions in a few years, and then set up a market on each question. My guess is that a relatively new person who expects to do alignment research for the next few years would be the perfect person for this, or better yet a few such people, and it would save me the effort.
I'm up for doing that. Are there any important things I should take into account before doing it? My first draft would be something like:
Will tailcalled consider X alignment approach important in 4 years?
With description:
I have been following AI and alignment research on and off for years, and have a somewhat reasonable mathematical background to evaluate it. I tend to have an informal idea of the viability of various alignment proposals, though it's quite possible that idea might be wrong. In 4 years, I will evaluate X and decide whether there have been any important good results since today. I will probably ask some of the alignment researchers I most respect, such as John Wentworth or Steven Byrnes, for advice about the assessment, unless it is dead-obvious. At the time of making the market, I currently think <extremely brief summary>.
<link to core post for X>
List of approaches I would currently have in the evaluation:
Something roughly along those lines sounds right. You might consider e.g. a ranking of importance, or asking some narrower questions about each agenda - how promising will they seem, how tractable will they seem, how useful will they have been in hindsight for subsequent work produced in the intervening 4-year period, how much will their frames spread, etc - depending on what questions you think are most relevant to how you should allocate attention now. You might also consider importance of subproblems, in addition to (or instead of) agendas. Or if there's things which seem like they might be valuable to look into but would cost significant effort, and you're not sure whether it's worthwhile, those are great things for a market on your future judgement.
In general, "ask lots of questions" is a good heuristic here, analogous to "measure lots of stuff".
Markets up: https://www.lesswrong.com/posts/3KeT4uGygBw6YGJyP/ai-research-program-prediction-markets
You might consider e.g. a ranking of importance
I considered that, but unless I'm misunderstanding something about Manifold markets, they have to be either yes/no or open-ended.
or asking some narrower questions about each agenda - how promising will they seem, how tractable will they seem, how useful will they have been in hindsight for subsequent work produced in the intervening 4-year period, how much will their frames spread, etc - depending on what questions you think are most relevant to how you should allocate attention now [...]
In general, "ask lots of questions" is a good heuristic here, analogous to "measure lots of stuff".
I agree with measuring lots of stuff in principle, but Manifold Markets only allows me to open 5 free markets.
I think asking or finding/recognizing good questions rather than giving good answers mediocre questions is probably a highly valuable skill or effort pretty much anywhere we are looking to advance knowledge -- or I suppose advance pretty much anything. How well prediction markets will help in producing that . . . worth asking I suspect.
Clearly just predicting time lines does little to resolve a problem or help anyone prioritize their efforts. So as an uninformed outsider, can any of the existing prediction questions be recast into a set of question that do shift the focus on a specific problem and proposed approaches to resolve/address the risk?
Would an abstract search of recent (all?) AI alignment papers perhaps point to a collection of questions that might be then placed on the prediction markets? If so, seems like a great survey effort for an AI student to do some leg work on. (Though I suppose some primitive AI agent might be more fitting and quicker ;-)
For anyone who'd like to see questions of this type on Metaculus as well, there's this thread. For certain topics (alignment very much included), we'll often do the legwork of operationalizing suggested questions and posting them on the platform.
Side note: we're working on spinning up what is essentially an AI forecasting research program; part of that will involve predicting the level of resources allocated to, and the impact of, different approaches to alignment. I'd be very glad to hear ideas from alignment researchers as to how to best go about this, and how we can make its outputs as useful as possible. John, if you'd like to chat about this, please DM me and we can set up a call.
I'm trying to see if pol.is would be good for this, like so: https://pol.is/4fdjudd23d
pol.is is a tool for aggregating opinions on political subjects from among a lot of people - it takes agree/disagree votes, clusters opinions based on similarity of voting, and ultimately tries to find consensus opinions. It was used in Taiwan to help write ride-share legislation.
I'm hoping I can misuse it here for operationalizing prediction market questions. If the "manifold users" like to bet on understandable questions, the "forecasters" like to bet on precise questions, while the "researcher" likes questions about day-to-day work, then perhaps by getting enough people from each "party" to weigh in it will find "consensus" questions that they are simultaneously useful, precise, and popular (and therefore more accurate).
I am unsure if pol.is will actually work better at the 10-100 people level compared to a normal forum. Let's give it a try anyways!
So, I’m an alignment researcher. And I have a lot of prediction-flavored questions which steer my day-to-day efforts. Which recent interpretability papers will turn out, in hindsight a year or two from now, to have been basically correct and high-value? Will this infrabayes stuff turn out to be useful for real-world systems, or is it the kind of abstract math which lost contact with reality and will never reconnect? Is there any mathematical substance to Predictive Processing other than “variational inference is a thing” plus vigorous hand-waving? How about Shard Theory? What kinds of evidence will we see for/against the Natural Abstraction Hypothesis over the next few years? Will HCH-style amplification ever be able to usefully factor problems which the human operator/programmer doesn’t immediately see how to factor? Will some version of this conjecture be proven? Will it be proven by someone else if I don’t focus on it? Are there any recent papers/posts which lots of other people expect to have high value in hindsight, but which I haven’t paid attention to?
Or, to sum that all up in two abstract questions: what will I (or someone else whose judgement I at least find informative) think I should have paid more attention to, in hindsight? What will it turn out, in hindsight, that I should have ignored or moved on from sooner?
Hmm, I wonder if the prediction markets have anything useful to say here? Let’s go look for AI predictions on Manifold…
So, we’ve got about a gazillion different flavors of AI capabilities questions, with a little bit of dabbling into how-society-will-react-capabilities questions. On the upside, operationalizing things in lots of different ways is exactly what we want. On the downside, approximately-all of the effort is in operationalizing one thing (AI capabilities timelines), and that thing is just not particularly central to day-to-day research decisions. It’s certainly not on the list of questions which first jump to mind when I think of things which would be useful to know to steer my research. Sure, somebody is going to argue in the comments that timelines are relevant for some particular decision, like whether to buy up GPU companies or something, but there’s no way in hell that focusing virtually all alignment-related prediction-market effort on operationalizations of capabilities timelines is actually the value-maximizing strategy here.
(And while I happened to open up Manifold first, the situation is basically the same in other prediction markets. To a first approximation, the only alignment-adjacent question prediction markets ever weigh in on is timelines.)
So, here’s my main advice to someone who wants to use prediction markets to help alignment work: imagine that you are an alignment researcher/grantmaker/etc. Concretely imagine your day-to-day: probing weight matrices in nets, conjecturing, reading papers/posts, reviewing proposals, etc. Then, ask what kind of predictions have the highest information value for your work. If the answer is “yet another operationalization of timelines”, then you have probably fucked up somewhere.
Of course you might also try asking some researchers or grantmakers the same question, though keep in mind the standard user-interview caveat: users do not actually know what they want or would like.