alexfeghhi

There might be some AI paradigm that doesn’t even exist today, and yet ten years from now it will have already been subjected to 100,000 person-years of R&D, and trillions of dollars of investment

I think it's important to identify the ability of a paradigm to scale with resources. For instance, the current DL paradigm can be scaled with data and compute in ways that have been predicted through scaling laws. The paradigm you're described seems to rely on algorithmic improvements to innately derive feedback from the environment and to learn efficiently f... (read more)

Doesn't the traditional lesswrong Bayesian Utilitarianism save you from Omega's bomb? Whatever decision theory you believe in, you must have to it in order to justify left boxing. Since there's no rigorous basis to assign probabilities to decision theories, your probability can't be that high and you end up right boxing.

I agree, but from a different angle.

There are several ways in which communicating probabilities offers useful information, and I don't see p(doom) in any of them.

For one, there are true probabilities, such as the probability of a radioactive atom decaying by it's half life. This satisfies the frequentist interpretation, where one could re-run the radioactive decay experiment infinitely and observe the true probabilities over time. This does not apply to p(doom); it is unclear how quantum uncertainty could lead to noticeably different outcomes were we to re... (read more)

AI is a poorly understood, speculative field, so people with different intuitions come to different conclusions. If your intuition is that AI is inherently dangerous, it makes sense to seriously investigate the AI safety questions, whereas if you don't, you would focus on the potential benefits and want to accelerate.

I remember when first reading about paperclip maximizer AIs I thought it was stupid, as in anything superintelligent would be capable of understanding instructions. I think this intuition hints at the stronger reason to be more optimistic abou... (read more)