Here's a vastly longer and more detailed analysis of how accurate Ray Kurzweil's 2019 predictions were: https://www.militantfuturist.com/how-ray-kurzweils-2019-predictions-are-faring/
Also see this:
https://www.militantfuturist.com/will-kurzweils-2019-be-our-2029/
And this:
https://www.militantfuturist.com/the-kurzweil-predictions-that-dont-matter/
So the rise of AGI will mark the third phase shift to the global economy.
What will the FOURTH phase shift owe to, and according to your model, when will it happen?
Eliezer,
It looks like you put forth an alternate scenario to the destruction of the human race here:
"A human being runs on 100 watts. Without even compressing humanity at all, 800GW, a fraction of the sunlight falling on Earth alone, would suffice to go on operating our living flesh, if Something wanted to do that to us."
I did the math, and if a Dyson structure were positioned between the Earth and the Sun, only 0.0005% of electricity produced by the portion of the structure obscuring the Earth would be enough to support a population of eight billion humans "consuming" 100 Watts of electricity apiece. It sounds like you're envisioning an outcome where a "benevolent" AGI blocks out the Sun, but still provides humans with enough resources to survive on Earth.
Would it be as bad as it sounds? Wouldn't the infrared light radiated from the back of the Dyson structure still warm the Earth a bit? Couldn't the waste heat generated by machine activities on Earth (data centers, factories) be recaptured to heat sealed human habitats?
The loss of sunlight would not affect our ability to harness other energy sources here, like fossil fuels, nuclear, and, importantly, geothermal power. Those could support a human civilization on Earth for millions of years. We might even be able to replace the Sun with an orbiting fusion reactor designed to produce light and to focus it on the Earth's surface.