This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
My main take on Bayesian epistemology being wrong is that I think to the extent it's useless in real life, it's because it focuses way too much on the ideal case, ala @Robert Miles's tweet here:
(The other problem I have with it is that even in the ideal case, it doesn't have a way to sensibly handle 0 probability events, or conditioning on probability 0 events, which can actually happen once we leave the world of finite sets and measures.)
That said, I don't think that people being wrong about epistemology is the cause of high p(Doom).
I'd agree more with @Algon in that the issues lie elsewhere (though a nitpick is that I wouldn't say that EU maximization is wrong for TAI/AGI/ASI, but rather that certain dangerous properties don't automatically hold, and that systems that EU maximize IRL like GPT-4 aren't actually nearly as dangerous as often assumed. Agree with the other points.)
[This comment is no longer endorsed by its author]Reply