As uncertainty grows around how AI development will affect culture and society, it becomes more valuable to compare track records of predictions about technological progress. 

I've recently been working on automating parts of the methodology from Arb's Scoring The Big 3's Predictive Performance report^[1], and have had some promising preliminary results. I hope to try to automate most of the steps in the original report, making it feasible to analyse many more track records and publish the results.

I am particularly interested in the following questions:

1. Which track record(s) would you find valuable to have evaluated in a similar way to Asimov, Clarke and Heinlein’s, as in the Arb report?
2. What would you want to see from an LLM-based evaluation that would give you confidence that the results are meaningful and accurate?

1. ^{^}

   See also original Cold Takes post explaining why such evaluations are valuable