Conditional Forecasting as Model Parameterization

10mo

This work was done jointly with Rajashree Agrawal. We'd like to thank the Epoch AI summer mentorship project, led with Forecasting Research Institute, for introducing us. Thanks to Josh Rosenberg, Tegan McCaslin, Avital Morris and others at FRI, as well as Jaime Sevilla at Epoch AI. Thanks also to Misha Yagudin and Jonathan Mann for their feedback.

This is the second piece in a series on forecasting, especially conditional forecasting and the potential we see there for making fuzzy models more legible:

Forecasting: the way I think about it
The promise of conditional forecasting for parameterizing our models of the world [this post]
What we're looking at and what we're paying attention to (Or: Why we shouldn't

... (read 1949 more words →)

Metaculus Q4 AI Benchmarking: Bots Are Closing The Gap

Molly

Molly, Tom Liptay

In Q4 we ran the second tournament in the AI Benchmarking Series which aims to assess how the best bots compare to the best humans on real-world forecasting questions, like those found on Metaculus. Over the quarter we had 44 bots compete for $30,000 on 402 questions with a team of ten Pros serving as a human benchmark on 122 of those 402. We found that:

Metaculus Pro Forecasters were better than the top bot “team” (a team of one, this quarter), but not with statistical significance (p = 0.079) using log scoring with a weighted t-test.
Top bot performance improved relative to the Pro benchmark to a -8.9 head-to-head score in Q4 2024,

... (read 3884 more words →)

Replying toForecasting: the way I think about it

Molly2y

Forecasting: the way I think about it

Figure 1 is clumsy, sorry. In the case of a smooth probability distribution of infinite worlds, I think the median and the average world are the same? But in practice, yes, it's an expected value calculation, summing P(world) * P(U|world) for all the worlds you've thought about.

Replying toForecasting: the way I think about it

Molly2y

Forecasting: the way I think about it

Good q, yes, that's the vertical axis in all the figures.

Forecasting: the way I think about it

Molly

This is the first post in a little series I'm slowly writing on how I see forecasting, particularly conditional forecasting; what it's good for; and whether we should expect people to agree if they just talk to each other enough.

Views are my own. I work at the Forecasting Research Institute (FRI), I forecast with the Samotsvety group, and to the extent that I have formal training in this stuff, it's mostly from studying and collaborating with Leonard Smith, a chaos specialist.

My current plan is:

Forecasting: the way I think about it [this post]
The promise of conditional forecasting / cruxing for parameterizing our models of the world
What we're looking at and what we're paying attention

... (read 610 more words →)

Replying toPlaying Northboro with Lily and Rick

Molly2y

Playing Northboro with Lily and Rick

This is so awesome and encouraging! I play old-time fiddle and I've wanted to play dances for years, but I've been afraid I can't get enough tunes up to dance tempo. But I can play a lot of the tunes on your list! You've given me the push I need. Thanks Lily and Jeff! Sounds like a lot of fun.

Results from an Adversarial Collaboration on AI Risk (FRI)

Josh Rosenberg

Josh Rosenberg, AvitalM, Molly, rosehadshar

Authors of linked report: Josh Rosenberg, Ezra Karger, Avital Morris, Molly Hickman, Rose Hadshar, Zachary Jacobs, Philip Tetlock^[1]

Today, the Forecasting Research Institute (FRI) released “Roots of Disagreement on AI Risk: Exploring the Potential and Pitfalls of Adversarial Collaboration,” which discusses the results of an adversarial collaboration focused on forecasting risks from AI.

In this post, we provide a brief overview of the methods, findings, and directions for further research. For much more analysis and discussion, see the full report: https://forecastingresearch.org/s/AIcollaboration.pdf

(This report is cross-posted to the EA Forum.)

Abstract

We brought together generalist forecasters and domain experts (n=22) who disagreed about the risk AI poses to humanity in the next century. The “concerned” participants (all of... (read 2663 more words →)

Replying toMERV Filters for Covid?

Molly3y

MERV Filters for Covid?

Thanks for this, Jeff! I'm planning to have two MERV-14 cubes going at an upcoming contra-dance. I'll refer curious people to this post.

I couldn't find 20x20x1 filters at my local hardware store; only 20x14x1 and 20x30x1. I don't have a good intuition for how effectiveness relates to filter size. How much less effective do you think a 14-inch tall "cube" would be than a proper cube?

LESSWRONG
LW

LESSWRONG
LW

Molly

Results from an Adversarial Collaboration on AI Risk (FRI)

Forecasting: the way I think about it

Conditional Forecasting as Model Parameterization

Metaculus Q4 AI Benchmarking: Bots Are Closing The Gap

Molly

Molly

Conditional Forecasting as Model Parameterization

Metaculus Q4 AI Benchmarking: Bots Are Closing The Gap

Forecasting: the way I think about it

Results from an Adversarial Collaboration on AI Risk (FRI)

Molly

Results from an Adversarial Collaboration on AI Risk (FRI)

Forecasting: the way I think about it

Conditional Forecasting as Model Parameterization

Metaculus Q4 AI Benchmarking: Bots Are Closing The Gap

Molly

Molly

Conditional Forecasting as Model Parameterization

Metaculus Q4 AI Benchmarking: Bots Are Closing The Gap

Forecasting: the way I think about it

Results from an Adversarial Collaboration on AI Risk (FRI)

Abstract