To make sure I'm understanding you correctly, do you think the largest problem comes from (1) thinking of AGI development as a sequence of Bernoulli trials, or (2) each Bernoulli trial having constant probability, or (3) both?

It's not obvious to me that (1) is hugely problematic - isn't Laplace's rule of succession commonly applied to forecasting previously unseen events? Are you perhaps arguing that there's something particular to AGI development such that thinking of it as a series of Bernoulli trials is completely invalid?

(1) seems reasonable as a model at this level of abstraction, absent quibbles
about whether some outcome really is AGI or not, instead of some degree of
AGI-ness.
(2) seems utterly wrong, and I don't think it even makes sense to talk about a
"first trial" as being a clear-cut thing, let alone having a sensible
probability, and definitely not as something related to success of all future
trials. I contend that it is not even "semi-informative", it is useless.

To make sure I'm understanding you correctly, do you think the largest problem comes from (1) thinking of AGI development as a sequence of Bernoulli trials, or (2) each Bernoulli trial having constant probability, or (3) both?

It's not obvious to me that (1) is hugely problematic - isn't Laplace's rule of succession commonly applied to forecasting previously unseen events? Are you perhaps arguing that there's something particular to AGI development such that thinking of it as a series of Bernoulli trials is completely invalid?

I'm more sympathetic to your cr... (read more)