"Often I compare my own Fermi estimates with those of other people, and that’s sort of cool, but what’s way more interesting is when they share what variables and models they used to get to the estimate."
– Oliver Habryka, at a model building workshop at FHI in 2016
One question that people in the AI x-risk community often ask is
"By what year do you assign a 50% probability of human-level AGI?"
We go back and forth with statements like "Well, I think you're not updating enough on AlphaGo Zero." "But did you know that person X has 50% in 30 years? You should weigh that heavily in your calculations."
However, 'timelines' is not the interesting question. The interesting parts are in the causal models behind the estimates. Some possibilities:
I had similar thoughts in the context of Aumann's agreement theorem.
Aumann's agreement theorem is like mining as much information as you can out of only the final probability estimates (and common knowledge of being honest and prefert Bayesians). Given such sparse info, all you can do is meta-update on how your partner disagrees with you, then meta-meta update on how they still disagree after meta-updating, etc.
Whereas in real conversations, people at best do one or two steps, and instead spend the time talking about their evidence and their reasoning (in... (read more)