This is a special post for quick takes by kman. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I was reading the "Singularity Summit 2011 Workshop Report" when I came across this:
After some discussion, participants gave their estimates for the probability of a good result (“win”) given various scenarios: e.g. prob(win | WBE or neuromorphic AI), prob(win | WBE), etc. Individuals’ estimates varied quite a bit. Zvi gave his personal estimate from a weighted average of others’ estimates, weighted by his estimate of each participant’s expertise. His given estimates were a roughly 14% chance of win if WBE or neuromorphic AI comes first and, coincidentally, a roughly 14% chance of win if de novo AI (Friendly AI or not) came first.
Or in modern lingo, P(doom | AGI) of 86%. These days Eliezer is the only participant that's given a P(doom | AGI) number at least that high, and Zvi is the only other that's even come close.
I have an impression that Eliezer was more optimistic than that back then (e.g. see here[1]), so I don't predict that these numbers are due to his views being heavily weighted in the calculation. @Zvi do you have any recollection of how/why these numbers ended up so low?
A very incomplete, somewhat uncharitable list of possible explanations:
People on some level were like "now that AGI doesn't feel like a distant abstract thing, I need to be optimistic in order to stay sane"
DL was somehow actually a huge positive update for many in a way that I've failed to understand
People such as Luke and Carl adopted Modest Epistemology and started deferring to those in the larger EA tent around them
The point of the exercise was to compare two counterfactual branches, so getting the absolute optimism of each branch right wasn't prioritized and ended up lower than those involved would have endorsed at the time
Not deep learning generally, but LLMs specifically. The superficially friendly-seeming assistant that obeys your natural language commands in typical usage may not be the kind of alignment that matters in the long run, but in 2011, it wasn't obvious that even that was possible.