1. Of course, the answer could be "A kajillion years from now" or "Never."
2. See this section of "Forecasting TAI with Biological Anchors" (Cotra (2020)) for a more full definition of "transformative AI."
3. I'm sorry. But I do think the rest of the series will be slightly more fun to read this way.
4. The examples here are of course simplified. For example, both Deep Blue and AlphaGo incorporate substantial amounts of "tree search," a traditionally-programmed algorithm that has its own "trial and error" process.
5. And they can include simulating long chains of future game states.
6. Some AIs could be used to determine whether papers are original contributions based on how they are later cited; others could be used to determine whether papers are original contributions based only on the contents of the paper and on previous literature. The former could be used to train the latter, by providing a "That's correct" or "That's wrong" signal for judgments of originality. Similar methods could be used for training AIs to assess the correctness of papers.
7. E.g., https://openai.com/blog/improving-language-model-behavior/
8. Due to improvements in hardware and software.
9. It's even worse than spaghetti code.
10. More books: Human Compatible, Life 3.0, and The Alignment Problem.
1. If you have no idea what that means, try my short economic growth explainer.
2. Global real growth has generally ranged from slightly negative to ~7% per year.
3. I'm skipping over 2020 here since it was unusually different from past years, due to the global pandemic and other things.
4. For the historical data, see Modeling the Human Trajectory. The projections are rough and meant to be visually suggestive rather than using the best modeling approaches.
5. This refers to real GDP growth (adjusted for inflation). 2% is lower than the current world growth figure, and using the world growth figure would make my point stronger. But I think that 2% is a decent guess for "frontier growth" - growth occurring in the already-most-developed economies - as opposed to total world growth, which includes “catchup growth” (previously poor countries growing rapidly, such as China today).
To check my 2% guess, I downloaded this US data and looked at the annualized growth rate between 2000-2020, 2010-2020, and 2015-2020 (all using July since July was the latest 2020 point). These were 2.5%, 2.2% and 2.05% respectively.
6. 2% growth over 35 years is (1 + 2%)^35 = 2x growth.
7. Wikipedia's highest listed estimate for the Milky Way's mass is 4.5*10^12 solar masses, each of which is about 2*10^30 kg, each of which is estimated as the equivalent of about 1.67*10^-27 hydrogen atoms. (4.5*10^12 * 2*10^30)/(1.67*10^-27) =~ 5.4*10^69.
8. Wikipedia: "In March 2019, astronomers reported that the mass of the Milky Way galaxy is 1.5 trillion solar masses within a radius of about 129,000 light-years." I'm assuming we can't travel more than 129,000 light-years in the next 8200 years, because this would require far-faster-than-light travel.
9. This calculation isn't presented straightforwardly in the post. The key lines are "No matter what the technology, a sustained 2.3% energy growth rate would require us to produce as much energy as the entire sun within 1400 years" and "The Milky Way galaxy hosts about 100 billion stars. Lots of energy just spewing into space, there for the taking. Recall that each factor of ten takes us 100 years down the road. One-hundred billion is eleven factors of ten, so 1100 additional years." 1400 + 1100 = 2500, the figure I cite. This relies on the assumption that the average star in our galaxy offers about as much energy as the sun; I don't know whether that's the case.
10. There is an open debate on whether Modeling the Human Trajectory is fitting the right sort of shape to past historical data. I discuss how the debate could change my conclusions here.
11. 250 doublings would be a growth factor of about 1.8*10^75, over 10,000 times the number of atoms in our galaxy.
12. 20 years would be 240 months, so if each one saw a doubling in the world economy, that would be a growth factor of about 1.8*10^72, over 100 times the number of atoms in our galaxy.
13. That’s because of the above observation that today’s growth rate can’t last for more than another 8200 years (82 centuries) or so. So the only way we could have more than 82 more centuries with growth equal to today’s is if we also have a lot of centuries with negative growth, ala the zig-zag dotted line in the "This Can't Go On" chart.
14. This dataset assigns significance to historical figures based on how much they are covered in reference works. It has over 10x as many "Science" entries after 1500 as before; the data set starts in 800 BC. I don't endorse the book that this data set is from, as I think it draws many unwarranted conclusions from the data; here I am simply supporting my claim that most reference works will disproportionately cover years after 1500.
15. To be fair, reference works like this may be biased toward the recent past. But I think the big-picture impression they give on this point is accurate nonetheless. Really supporting this claim would be beyond the scope of this post, but the evidence I would point to is (a) the works I'm referencing - I think if you read or skim them yourselves you'll probably come out with a similar impression; (b) the fact that economic growth shows a similar pattern (although the explosion starts more recently; I think it makes intuitive sense that economic growth would follow scientific progress with a lag).
16. The papers cited in The Duplicator on this point specifically model an explosion in innovation as part of the dynamic driving explosive economic growth.
1.The best example I can think of, but surely not the only one.
2. The movie The Matrix gives a decent intuition for the idea with its fully-immersive virtual reality, but unlike the heroes of The Matrix, a digital person need not be connected to any physical person - they could exist as pure software.
The agents ("bad guys") are more like digital people than the heroes are. In fact, one extensively copies himself.
3. See Age of Em Chapter 6, starting with "Regarding the computation ..."
4. For example, when multiple teams of digital people need to coordinate on a project, they might speed up (or slow down) particular steps and teams in order to make sure that each piece of the project is completed just on time. This would allow more complex, "fragile" plans to work out. (This point is from Age of Em Chapter 17, "Preparation" section.)
5. See Age of Em Chapter 11, "Retirement" section.
6. Without human bodies - and depending on what kinds of robots were available - digital people might not be good substitutes for humans when it comes to jobs that rely heavily on human physical abilities, or jobs that require in-person interaction with biological humans.
However, digital people would likely be able to do everything needed to cause an explosive economic growth, even if they couldn't do everything. In particular, it seems they could do everything needed to increase the supply of computers, and thereby increase the population of digital peopl... (read more)
7. It is debatable whether the world is getting somewhat better at these things, somewhat worse, or neither. But it seems pretty clear that the progress isn't as impressive as in computing.
8. Why would the copy cooperate in the experiment? Perhaps because they simply were on board with the goal (I certainly would cooperate with a copy of myself trying to learn about meditation!). Perhaps because they were paid (in the form of a nice retirement after the experiment). Perhaps because they saw themselves and their copies (and/or original) as the same person (or at least cared a lot about these very similar people). A couple of factors that would facilitate this kind of experimentation: (a) digital people could examine their own state of mind to... (read more)
9. I'd also expect them to be able to try more radical things. For example, in today's world, it's unlikely that you could run a randomized experiment on what happens if people currently living in New York just decide to move to Chicago. It would be too hard to find people willing to be randomly assigned to stay in New York or move to Chicago. But in a world of digital people, experimenters could pay New Yorkers to make copies of themselves who move to Chicago. And after the experiment, each Chicago copy that wished it had stayed in New York could choose t... (read more)
10. See footnote from the first bullet point on why people's copies might cooperate with them.
11. And air for cooling.
12. See the estimates in Astronomical Waste for a rough sense of how big the numbers can get here (although these estimates are extremely speculative).
This comment is a container for our temporary "footnotes-as-comments" implementation that gives us hover-over-footnotes.
1. The agents ("bad guys") are more like digital people. In fact, one extensively copies himself.
2. These are all taken from this video, except for the last one.
3. Football video games have already expanded to simulate offseason tradings, signings and setting ticket prices.
4. It's also possible there could be conscious "digital people" who did not resemble today's humans, but I won't go into that here - I'll just focus on the concrete example of "digital people" that are virtual versions of humans.
5. According to the PhilPapers Surveys, 56.5% of philosophers endorse physicalism, vs. 27.1% who endorse non-physicalism and 16.4% "other." I expect the vast majority of philosophers who endorse physicalism to agree that a sufficiently detailed simulation of a human would be conscious. (My understanding is that biological naturalism is a fringe/unpopular position, and that physicalism + rejecting biological naturalism would imply believing that sufficiently detailed simulations of humans would be conscious.) I also expect that some philosophers who don't e... (read more)
6. From an email from a physicist friend: "I think a lot of people have the intuition that real neural activity, produced by real chemical reactions from real neurotransmitters, and real electrical activity that you can feel with your hand, somehow has some property that mere computer code can't have. But one of the overwhelming messages of modern physics has been that everything that exists -- particles, fields, atoms, etc, is best thought of in terms of information, and may simply *be* information. The universe may perhaps be best described as a mathemat... (read more)
7. I actually expect it would start off very expensive, but become cheaper very quickly due to a productivity explosion, discussed below.
8. For an illustration of this, see this report: How much computational power does it take to match the human brain? (Particularly the Uncertainty in neuroscience section.) Even estimating how many meaningful operations the human brain performs is, today, very difficult and fraught - let alone characterizing what those operations are.
9. This statement is based on my understanding of conventional wisdom plus the fact that recorded video and audio often seems quite realistic, implying that the camera/microphone didn't fail to record much important information about its source.