Why no love for this project?
http://www.theuncertainfuture.com/
My perception as an outsider is that SI put a fair amount of manpower into it, finished it, submitted it to hacker news, and then folks largely forgot about it. Is it even linked from the SI website?
Norman Rasmussen's analysis of the safety of nuclear power plants, written before any nuclear accidents had occurred, correctly predicted several details of the Three Mile Island incident in ways that that previous experts had not (see McGrayne 2011, p. 180).
Is there any way that a policy maker could have known in advance to pay attention to Rasmussen rather than other experts? Is this a case of retroactively selecting the predictor who happened to be right out of a large group of varied, but roughly equally justified, predictors, or did Rasmussen use systematically better methods for making his predictions?
It's worth noting that stories of catastrophes that were successfully averted because someone listened to an expert may be hard to find.
If an expert tells you to add a safety mechanism, and you end up using that mechanism, you know that the expert helped you.
I mean a safety mechanism like a button that shuts down the assembly line. If someone gets caught in the machinery and you push the button to prevent them from getting (more) hurt, you will be happy the expert told you to install that button.
Is there any way that a policy maker could have known in advance to pay attention to Rasmussen rather than other experts?
Yes. Rasmussen used Bayes, while everyone else used the methods of (1) Frequentism or (2) Experts Must Have Great Intuitions.
All else being equal, I would put more trust in the report that uses Bayesian statistics than a report that uses Frequentist statistics, but I wouldn't expect that strong an effect from that alone. (I would expect a strong increase in accuracy for using any kind of statistics over intuition.)
Following your link, I notice that Rasmussen's report used a fault tree. I would expect that the consideration of failure modes of each component of a nuclear reactor played a huge role in his accuracy, and that Bayesian and Frequentist statistics would largely agree how to get individual failure rates from historical data and how to synthesize this information into a failure rate for the whole reactor. Assuming the other experts did not also use fault trees, I would credit the fault trees more than Bayes for Rasmussen's success. (And if they did, I would wonder where they went wrong.)
A model of AI risk currently being developed in MATLAB by Anna Salamon and others
I hadn't heard about this one. Is there a list somewhere of all the little projects that the SI is working on? (Or should there be)? Posts like this one (and the monthly status reports) are very useful, but since each post only lists some of the things going on, I'm worried that I'll miss something interesting. Or that there's something I thought the SI were working on but which had been quietly dropped.
Series: How to Purchase AI Risk Reduction
Norman Rasmussen's analysis of the safety of nuclear power plants, written before any nuclear accidents had occurred, correctly predicted several details of the Three Mile Island incident in ways that that previous experts had not (see McGrayne 2011, p. 180). Had Rasmussen's analysis been heeded, the Three Mile Island incident might not have occurred.
This is the kind of strategic analysis, risk analysis, and technological forecasting that could help us to pivot the world in important ways.
Our AI risk situation is very complicated. There are many uncertainties about the future, and many interacting strategic variables. Though it is often hard to see whether a strategic analysis will pay off, the alternative is to act blindly.
Here are some examples of strategic research that may help (or have already helped) to inform our attempts to shape the future:
Here are some additional projects of strategic research that could help inform x-risk decisions, if funding were available to perform them:
I'll note that for as long as FHI is working on AI risk, FHI probably has an advantage over SI in producing actionable strategic research, given past successes like the WBE roadmap and the GCR volume. But SI is also performing actionable strategic research, as described above.