EDIT: We've stopped answering questions for now, sorry if we didn't get to your question! We're still really interested in what kinds of questions people want forecasted, feedback on how useful it is to delegate forecasts, and Elicit as a tool, so feel free to keep commenting these thoughts. We also forecasted questions on the EA Forum version of this post.

Hi everyone! We, Ought, have been working on Elicit, a tool to express beliefs in probability distributions. This is an extension of our previous work on delegating reasoning. We’re experimenting with breaking down the reasoning process in forecasting into smaller steps and building tools that support and automate these steps.

In this specific post, we’re exploring the dynamics of Q&A with distributions by offering to make a forecast for a question you want answered. Our goal is to learn:

  1. Whether people would appreciate delegating predictions to a third party, and what types of predictions they want to delegate
  2. Whether a distribution can more efficiently convey information (or convey different types of information) than text-based interactions
  3. Whether conversing in distributions isolates disagreements or assumptions that may be obscured in text
  4. How to translate the questions people care about or think about naturally into more precise distributions (and what gets lost in that translation)

We also think that making forecasts is quite fun. In that spirit, you can ask us (mainly Amanda Ngo and Eli Lifland) to forecast any continuous question that you want answered. Just make a comment on this post with a question, and we’ll make a distribution to answer it.

Some examples of questions you could ask:

We’ll spend <=1 hour on each one, so you should expect about that much rigor and information density. If there’s context on you or the question that we won’t be able to find online, you can include it in the comment to help us out.

We’ll answer as many questions as we can from now until Monday 8/3. We expect to spend about 10-15 hours on this, so we may not get to all the questions. We’ll post our distributions in the comments below. If you disagree or think we missed something, you can respond with your own distribution for the question.

We’d love to hear people’s thoughts and feedback on outsourcing forecasts, providing beliefs in probability distribution, or Elicit generally as a tool. If you’re interested in more of what we’re working on, you can also check out the competition we’re currently running on LessWrong to amplify Rohin Shah’s forecast on when the majority of AGI researchers will agree with safety concerns.

New Comment
25 comments, sorted by Click to highlight new comments since: Today at 2:32 AM

If I held an outdoor event near Berkeley CA, in December, where ~250 people sat outdoors in social-bubble groups 30' from each other, would at least one person get Covid-19?

[edit: event is 3 hours, with lots of shouting and singing]

How long is the event? Is there perfect compliance with groups staying apart? Is everyone wearing masks? Are people singing/shouting?

Event is 3 hours with lots of shouting/singing.

I think the intention is that the forecasts are of continuous variables. Are you interested in the expected number of people who get covid?

Oh, yes that’s even better. I had assumed I was supposed to phrase it as a discrete question. 
(also interested in fractional people if it’s less than one)

I guess ‘number of microcovids generated by the event’ is what I actually want. Not sure what the requirements of the question are. 

Either expected number of people who get covid or number of microcovids generated by the event works as a question! My instinctive sense is that # of people who get covid will be easier to quickly reason about, but I'll see as I'm forecasting it.

Nod. My guess is that that number of people in expectation will likely be 0 (if it turns out to be 1 or more than that's a dealbreaker for the event), and "fractional chance at least one person gets it" would be decision relevant for me.

*I guess further consideration that's harder to reason about is "I'm expecting most of the attendees to be the sort of person who takes caution seriously most of the time, which lowers the rate of people who are likely to have in the first place to transmit it." But, not sure how to quantify that and don't really want to rely on it.

Here's my prediction, and here's a spreadsheet with more details (I predicted expected # of people who would get COVID). Some caveats/assumptions:

  • There's a lot of uncertainty in each of the variables that I didn't have time to research in-depth
  • I didn't adjust for this being outdoors, you can add a row and adjust for that if you have a good sense of how it would affect it.
  • I wasn't sure how to account for the time being 3 hours. My sense is that if you're singing loudly at people < 1m for 3 hours, this is going to be a pretty high infection rate. Also, I assumed they weren't wearing masks because of the singing. I'm most uncertain about this though
  • You didn't mention how big the pods are. I assumed 10 people in a pod, but it would change it if this were much smaller.


(fwiw: I was assuming the people in pods would already have infected each other or would be about to infect each other anyways (due to sharing a house and lots of airspace), so the intent was something more about how much additional spread would be between pods, outdoors)

Oh yeah that makes sense, I was slightly confused about the pod setup. The approach would've been different in that case (still would've estimated how many people in each pod were currently infected, but would've spent more time on the transmission rate for 30 feet outdoors). Curious what your current prediction for this is? (here is a blank distribution for the question if you want to use that)

I haven't yet attempted to seriously estimate it. I know of two other people who have risk calculators that I'm going to try to use at some point, and was interested in having a few different estimates to help triangulate things.

This is super self absorbed, and maybe not reasonably doable with the info you have, but... no harm in asking I guess.

I'd like to know my chances of winning the SSC book review contest (assuming Scott starts asking for entries again following this).

(In my favor: I think I'm a pretty good writer, reviewing a book that's totally on-brand. Against me: he had ~20 entries more than a month before the original deadline (and by Sturgeon's law, two of those might have been decent), and I haven't finished yet.)

By 2030 how many logical qubits will the leading commercially available quantum device be able to compute with?

My forecast is based on:

I don't have a background in quantum computing, so there's a chance I'm misinterpreting the question in some way, but I learned a lot doing the research for the forecast (like that there's a lot of controversy regarding whether quantum supremacy has been achieved yet).

Amusingly, during my research I stumbled upon this Metaculus question about when a >49 qubit quantum computer would be created which resolved ambiguously due to the issue of how well-controlled the qubits are. For the purposes of this forecast I assumed it would resolve based on the raw number of qubits, without adjusting for control.

When will air travel from New York to Hong Kong no longer require arriving passengers to self-quarantine?

I'm not sure if a probability counts as continuous?

If so, what's the probability that this paper would get into Nature (main journal) if submitted? Or even better, how much more likely is it to get into The Lancet Public Health vs Nature? I can give context by PM. https://doi.org/10.1101/2020.05.28.20116129

In what year would more than 30% of US adults own a device whose primary interaction mode is through augmented reality (such as google glass or the rumored apple AR glasses)?

Here’s my prediction for this! I predicted a median of March 1, 2029. Below are some of the data sources that informed my thinking.

Related Metaculus question: When will sales of a non-screen technology be greater than sales of a screen technology?

A house in the Seattle area sold for $1,000,000 in April 2019.

How much would the same house sell for in April 2021? Assume away "someone wanted to sell after only 2 years" considerations; maybe instead a better phrasing is "a house in the Seattle area had an extremely credible offer for $1,000,000 in April 2019, which was not acted upon and records of which are not publicly available".

My forecast is based on historical data from Zillow. I explained my reasoning in the notes. The summary is that housing prices haven't changed very much in Seattle since April 2019 (on the whole it's risen 1%). On the other hand, prices in more expensive areas have stayed the same or declined slightly. I settled on a boring median of the price staying the same. Due to how stable the prices have been recently, I think most of the variation will come from the individual house and which neighborhood it's in, with an outside chance of large Seattle home value fluctuations.

Most of the variance I've seen discussed comes from a severe change in the tech labor market producing a severe change in the housing market in the Bay Area, Seattle, etc. Did you intentionally or unintentionally leave out the effects of COVID? Or are they wrapped up in black swans?

I must admit I haven't followed the discussions you're referring to but if I were to spend more time forecasting this question I would look into them.

I didn't include effects of COVID in my forecast as it looks like the Zillow Home Value Index for Seattle has remained relatively steady since March (2% drop). I'm skeptical that there are likely to be large effects from COVID in the future when there hasn't been a large effect from COVID thus far,

A few reasons I could be wrong:

  • Zillow data is inaccurate or incomplete, or I'm interpreting it incorrectly.
  • COVID affects variation of individual housing prices much more than the trend of the city as a whole.
  • The COVID effects will be much bigger in the next half of a year than in the previous. Perhaps there will be a second wave much worse than the market is pricing in which produces large effects.

Cool deal. The reason I asked about black swans was specifically because of the potential permanent shift in companies' allowance or even desire for remote work. Chance seems low but significant, impact seems modest with a chance of large.

How much money will be raised by special purpose acquisition companies (SPACs) in initial public offerings in the year 2021?

To clarify, I'm looking for the number of new SPACs that issue equity times the average amount that they raised. I am NOT looking for the amount of money raised by the companies that were eventually the acquisition targets of these SPACs.


1. https://marker.medium.com/why-spacs-are-the-new-ipo-dcefe54b4bdd

2. https://www.bloomberg.com/news/newsletters/2020-06-23/money-stuff-bill-ackman-wants-a-mature-unicorn

3. https://en.wikipedia.org/wiki/Special-purpose_acquisition_company#:~:text=2018%3A%20%2410.7bn%20across%2046,bn%20across%2059%20SPAC%20IPOs

4. https://www.bloomberg.com/opinion/articles/2020-07-27/spacs-aren-t-cheaper-than-ipos-yet

5. https://www.spacresearch.com/

How many days after August 1st, 2020, will the FDA issue an Emergency Use Authorization for a product that performs some form of COVID testing directly in the home, meaning sample collection, molecular biology, and diagnostic result delivery can all be done without a user leaving their home?

Access to cloud computing is fine. Tele-guidance from a trained healthcare practitioner or anyone else is fine.


1. https://www.fda.gov/emergency-preparedness-and-response/mcm-legal-regulatory-and-policy-framework/emergency-use-authorization#covidinvitrodev