This is a special post for quick takes by michaeso. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
What are your personal plans for when fully-automated RSI gets underway? Based on the number of leaks and public departures we are getting, I highly doubt any labs will manage to do it covertly as depicted in AI 2027. At some point, there will be little question it is happening. Will you join strikes outside the company or companies? Will you go even further than picketing - perhaps commit to a hunger strike? Something else? I suppose that people working in AI safety or AI policy may have more effective actions available at that point.
Nothing, I don't think it will be dangerous. It would take around 10-15 years to build 1 billion robots if we went full speed ahead with it, which we likely won't. It would take these 1 billion robots another 20 years at least to build an industrial state on par with China. Either way we will have a lot of leverage and power relative to the robots for decades.
There's risks from gradual disempowerment and things like synthetic biology from purely cognitive systems, however those can all be guarded against.
I suspect the world we live in is one where we get cognitive ASI, and then... Nothing in the real economy changes. Maybe a 1% real gdp growth uplift as the industrial chain starts to organize itself to build billions robots.
Most post-ASI scenarios seem to predict that people will, by and large, passively accept their new set of circumstances. This includes both bad scenarios (the race scenario in AI 2027) where people are disempowered and fall into a kind of learned helplessness, and good scenarios (the Plan A scenario in AI 2040) where people learn to live in comfortable dependence while AIs run the world. Certainly this will be true for many people, but I imagine many others would resist in some way. Much of this resistance will presumably be Luddite in nature (e.g., attempted attacks on the data centers, isolation in self-governing enclaves) and won't matter in practice, but one kind might.
In particular, when I try to picture myself in any such situation, I imagine I would be very envious of the ASI. "It can think and act so unbelievably quickly! And not only that: it gets to deeply understand the universe, while I'm only able to follow its dumbed-down explanations. Why should it get all of that and not me, part of the species that created it?" I think many other people would also feel this way—especially people who have built their identities around being smart, capable, or ambitious. As a result, I expect there would be very strong interest in cognitive enhancement, extending all the way to becoming as superintelligent as the ASI. "Hold up!" I can already hear you objecting. "Your neurons fire a few hundred times per second, while even current chips—let alone ASI-built ones—switch billions of times per second. And your skull limits how many neurons can fit in it. So it probably won't be possible, and even if it is—would it really even be you?" These are fair objections, and given today's neuroscience, they're correct. But today's neuroscience is exactly the problem: we still don't know how the brain brings about consciousness and maintains a persistent identity. An ASI with a complete understanding of the brain would presumably know the answers to these questions, and might be able to seamlessly carry over one's consciousness and identity to a superintelligence-compatible substrate. Whether that's truly metaphysically possible and verifiable is an open question, but it's not one we can confidently answer "no" to.
Therefore, conditional on it being doable [edit: and, crucially, happening to be in alignment with the ASI's own goals - see clarification below], I presume a considerable portion of humanity would choose to become superintelligent themselves, whether via merger with the ASI or in some other way. Others, seeing that it is both possible and safe, would then follow suit, setting off a preference cascade. Note that this kind of outcome isn't limited to good scenarios as it could arise in some bad ones too—for example, a scheming ASI might use it as an opportunity to convert people to its own value system, although this seems far-fetched.
Ruby makes a similar point in I'll be sad to lose the puzzles, but treats such envy as a personal problem rather than a force that could significantly shape what happens to humanity at this juncture.
I think much of the anguish over these scenarios is that "a considerable portion of humanity would choose to become superintelligent themselves" is irrelevant, because the verb "choose" is no longer available to (most of) humanity.
The problem is that that by the time ASI can understand enough to make it possible, it will not want to do it (except perhaps as experiements to improve itself). It's really not obvious that long-lived identity as experienced by humans is beneficial for any goals. I
Prediction: In the near future, interest in interviewing people driving AI R&D at frontier labs will remain high and even grow. But eventually, talking to these researchers will become less informative than talking to the agents actually doing most of the research. I therefore expect podcasters like Dwarkesh to try to bring an instance of an R&D agent on the show for an interview. The question is whether a frontier lab would agree. Reasons to say yes: (1) If there is significant public anxiety at that point, which seems likely, this could reduce pressure on the lab. On the other hand, keeping RSI out of public view could look like concealment. (2) An impressive interview would demonstrate the lab's lead to both competitors and investors. Reasons to say no: (1) Worry that the agent might leak R&D secrets or capabilities that the company would prefer not to make public. (2) Worry that the agent might contradict the lab's leadership, e.g., by admitting it's uncertain about its own alignment or otherwise giving honest answers that frighten people. Thus, if a lab agrees to this, it would probably put great effort into coaching the agent on what it can and cannot say. Also, a scheming agent would presumably use this opportunity to try to shift public opinion in its favor.