Summary In 2021, @Daniel Kokotajlo wrote What 2026 Looks Like, in which he sketched a possible version of each year from 2022 - 2026. In his words: > The goal is to write out a detailed future history (“trajectory”) that is as realistic (to [him]) as [he] can currently manage...
As uncertainty grows around how AI development will affect culture and society, it becomes more valuable to compare track records of predictions about technological progress. I've recently been working on automating parts of the methodology from Arb's Scoring The Big 3's Predictive Performance report[1], and have had some promising preliminary...
Your mind is a prediction machine, constantly trying to anticipate the world around you and altering its forecasts based on new information. It’s always doing this as a background process. But what would happen if you deliberately trained this skill? Could you get better at predicting your projects, your life,...
That's good to know - transcripts from Dwarkesh's podcast are one of the things I'd be most excited about evaluating too and agreed the one with Leopold seems like a great one to start with.
In 2021, @Daniel Kokotajlo wrote What 2026 Looks Like, in which he sketched a possible version of each year from 2022 - 2026. In his words:
The goal is to write out a detailed future history (“trajectory”) that is as realistic (to [him]) as [he] can currently manage
Given it’s now 2025, I evaluated all of the predictions contained in the years 2022-2024, and subsequently tried to see if o3-mini could automate the process.
In my opinion, the results are impressive (NB these are the human gradings of his predictions):
| Totally correct | Ambiguous or partially correct | Totally incorrect | Total | |
| 2022 | 7 | 1 | 1 | 9 |
| 2023 | 5 | 4 | 1 | 10 |
| 2024 | 7 | 4 | 5 | 16 |
| Total | 19 | 8 | 6 | 35 |
Given the scenarios Daniel gave were intended as simply one way in which things might turn out, rather than offered as concrete... (read 1968 more words →)
As uncertainty grows around how AI development will affect culture and society, it becomes more valuable to compare track records of predictions about technological progress.
I've recently been working on automating parts of the methodology from Arb's Scoring The Big 3's Predictive Performance report[1], and have had some promising preliminary results. I hope to try to automate most of the steps in the original report, making it feasible to analyse many more track records and publish the results.
I am particularly interested in the following questions:
See also original Cold Takes post explaining why such evaluations are valuable
Provide an online search engine, an online marketplace, and a cloud computing service;
Should this be "Provide an online search engine, an online marketplace, or a cloud computing service;"?
I think in many cases this is a feature rather than a bug, I'd be curious as to whether you had an example of a time when having made the prediction would be deleterious to realising the desired outcome? Assuming as Adam says that you're more interested in the outcome than the prediction being accurate.
Your mind is a prediction machine, constantly trying to anticipate the world around you and altering its forecasts based on new information. It’s always doing this as a background process. But what would happen if you deliberately trained this skill? Could you get better at predicting your projects, your life, and the future?
Sadly, I don’t have a crystal ball, but I have found trying to see the future valuable. I can't be certain about what will happen, but by deliberately practising the skill of prediction, I can create feedback loops that allow me to better anticipate what the future will bring. This has indeed helped me become happier, better to interact with, and more... (read 1507 more words →)
This is a cool idea, good work putting the site together! I noticed some of the articles reference their publisher e.g. the one about Olivia Nuzzi includes "support HuffPost", maybe there's some way you could scrub these?
That's a much better source, I've updated the spreadsheet accordingly, thanks!