Consider two future technologies.

For technology A:

  1. The underling physical laws governing it are fully understood.
  2. A blueprint for the technology is available and the universal consensus of scientists is that this blueprint will work.
  3. A worldwide consortium  of scientists has  undertaken to build the technology and  are funded tune of $10B's

For technology B, none of this is true:

  1. There is  no agreement on the underlying rules describing how and whether the system will work
  2. The general consensus is that existing blueprints will not work no matter how much they are scaled up
  3. The largest projects are in the $10M's of  dollars and are  frequently deemed "too expensive"

Which of  these technologies  would you expect to be developed first?

I  would  argue, based on this evidence, that AGI  is no closer (and  probably in fact much further) away than Commercial  Nuclear Fusion.  I furthermore suspect there is less  than  a  50/50 chance that  nuclear fusion will  achieve "positive energy" by ITER's 2035 target.

8 comments, sorted by Click to highlight new comments since: Today at 8:18 PM
New Comment

If that's all we knew these about two technologies, then I would agree that the second is likely to come later, if at all. But it's not.

Consider two technologies, in one, experts consistently underestimate how long progress will take and it always ends being far more difficult and complex than they expect. In the second, experts consistently overestimate how long progress will take, and it often ends up being far easier and simpler than they expected. In this case you would expect the second one (which is AI*) to come first.

I don't understand either field enough to predict which will come first, but to do that you'd need either a more detailed understanding, instead of a simple argument that takes only a few parts into account, or have one very strong argument, which I don't think this post makes.

*I know you talked about AGI and I also take narrow AI into account, I think that's reasonable given the field is younger and is seeing a huge explosion of progress lately.

In the second, experts consistently overestimate how long progress will take


This  doesn't  seem like a fair characterization of AI.  People  have  been predicting we could build machines that "think like humans"  at least  since Charles Babbage and they are all  pretty consistently overoptimistic.

but to do that you'd need either a more detailed understanding

My point is precisely that we  do have a detailed understanding of what it takes to build a fusion  reactor, and it is still (at  least) 15 years away.

All this argument actually says is that nuclear fusion is more promising than AI because nuclear fusion is more politically successful in getting funding. But (1) that routes through the opinions of politicians and laypeople, so it isn't informative about the underlying technology, and (2) on top of that you've grossly misrepresented the scientific consensuses and the funding situation.

There is definitely not a consensus that Tokomaks will work, in the sense of providing practically-useful energy. And the relevant comparison to ITER's construction cost is not the amount of compute spent to train a single model, it's the total spending on GPU/TPU hardware by all AI researchers, over a time scale comparable to ITER's construction.

There is definitely not a consensus that Tokomaks will work

Small quibble  here.  My point is that  we completely understand the underlying physical laws governing fusion.  There is no equivalent to "E=MC^2" (or the Standard  Model) for AGI.  

I'd also be really interested to see a quote along  the lines  of  "tokomaks won't work" or "ITER  will not produce more energy than it consumes  (Q>1)"  if they actually exist.  My current prior is that something like 99% of people who have studied nuclear fusion think it is possible with current technology to build a Tokomak with Q>1.

The physical laws allow us to get an idea about how hard nuclear fussion happens to be. It allows us to rule a lot of approaches as not having the chance to work. 

Standard railways have a track gauge of less than 1.5 meters. Back in the 1930s, Hitler planned the Breitspurbahn, broad-gauge railway with a track gauge of 3 meters. No dramatic new tech required, but it would seriously scale up transportation by rail of people and goods. Hitler planned to connect many European cities with them. 

None of that happened. We're still using the old track gauge, and European connections are relatively mediocre. But we've landed on the Moon and we've all got smartphones in our pockets. 

Planned developments with relatively straightforward tech regularly don't happen. Surprising new tech regularly disrupts all plans. 

I'm not saying the future is completely unknowable, but your three requirements don't seem too matter very much when you look for similar scenarios in history. 

I am no MIRIan, but I see an obvious difference: nuclear fusion is easy and has been around for about 70 years , it's controlled nuclear fusion that is hard. By contrast, there is no needle one has to thread with AGI, once it's achieved, it is self-sustaining, and, arguably, is more like a nuclear bomb than like a nuclear reactor. So that's an argument that AGI is not as hard.

However, there is an opposite argument: self-sustaining long-lasting nuclear fusion has been around for 13.8 billion years, and spontaneously arises in nature, while AGI has never been observed in nature, as far as we know, and "intelligence" in general artificial or natural, has not been observed outside the surface of this planet.

Nuclear fusion is a little over eight minutes away, so you're probably right. For now.