LESSWRONG
LW

ChristianWilliams
3744680
Message
Dialogue
Subscribe

Posts

Sorted by New

Wikitag Contributions

Comments

Sorted by
Newest
No wikitag contributions to display.
2ChristianWilliams's Shortform
2y
1
ChristianWilliams's Shortform
ChristianWilliams2y30

Metaculus is conducting its first user survey in nearly three years. If you have read analyses, consumed forecasts, or made predictions on Metaculus, we want to hear from you! Your feedback helps us better meet the needs of the forecasting community and is incredibly important to us. 


Take the short survey here — we truly appreciate it! (We'll be sure to share what we learn.) 

Reply
Can AI Outpredict Humans? Results From Metaculus's Q3 AI Forecasting Benchmark
ChristianWilliams9mo40

Hi @gwern, we are currently in the process of combing through winners' documentation of their bots and which models they used. We haven't yet encountered anyone who claims to have used one of the base models. 

We will share here if we learn a participant did indeed use one. 

Reply
AI forecasting bots incoming
ChristianWilliams10mo50

Maybe "sidestep the data leakage issue" then. The series was designed with these issues in mind. (I work at Metaculus.)  

Reply
Metaculus Introduces New Forecast Scores, New Leaderboard & Medals
ChristianWilliams2y20

Hi @Odd anon, thanks for the feedback and questions. 

1. To your point about copying the Community Prediction: It's true that if you copy the CP at all times you would indeed receive a high Baseline Accuracy score. The CP is generally a great forecast! Now, CP hidden periods do mitigate this issue somewhat. We are monitoring user behavior on this front, and will address it if it becomes an issue. We do have some ideas in our scoring trade-offs doc for further ways to address CP copying, e.g.: 

We could have a leaderboard that only considers the last prediction made before the hidden period ends to calculate Peer scores. This largely achieves the goal above: it rewards judgement, it does not require updates or tracking the news constantly. It does not reward finding stale questions.

Have a look here, and let us know what you think! (We also have some ideas we're tinkering with that are not listed in that doc, like accuracy metrics that don't include forecasts that are on the CP or +/- some delta.)

2. On indicating confidence:  You'll see in the tradeoffs doc that we're also considering the idea of letting users exclude a particular forecast from their peer score (Idea # 3), which could somewhat address this. (Interestingly, indicating confidence was attempted at Good Judgment Project, but ultimately didn't work and was abandoned.) 

We're continuing to develop ideas on the above, and we'd definitely welcome further feedback!

Reply
Some Thoughts on Conditional Forecasts – Lessons from the 2020 Election
ChristianWilliams2y10

Interesting read, thanks for writing it up. FYI the link "The report on the 2022 results is now available" leads to a private Google Drive file. 

Reply
[Event] Join Metaculus Tomorrow, March 31st, for Forecast Friday!
ChristianWilliams2y10

Fair point. Noted for future event posts. 

Reply
Progress links and tweets, 2022-11-01
ChristianWilliams3y30

Prediction market Metaculus launches a “Forecasting Our World In Data” tournament (via @metaculus)

Hi, this link should be https://www.metaculus.com/tournament/forecasting-Our-World-in-Data/ instead, it's missing the www

Reply
4Market Pulse Challenge 25Q3, $7,500 Prize Pool
6d
0
20Forecast AI 2027
1mo
0
6Analysis of Automated Prompt Engineering for Forecasting
1mo
0
4The Metaculus Cup Series Is Live, $5,000 Prize Pool
2mo
0
15POTUS Predictions Tournament
3mo
0
9USAID Outlook: A Metaculus Forecasting Series
4mo
0
6Build a Metaculus Forecasting Bot in 30 Minutes: A Practical Guide
5mo
0
21The State of Metaculus
5mo
0
7AI Forecasting Benchmark: Congratulations to Q4 Winners + Q1 Practice Questions Open
6mo
0
10Bridgewater x Metaculus Forecasting Contest Goes Global — Feb 3, $25k, Opportunities
6mo
0
Load More