This is a linkpost for https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4913167
Hi Seth--the other author of the paper here.
I think there are two things to say to your question. The first is that, in one sense, we agree. There are no guarantees here. Conditions could evolve such that there is no longer any positive-sum trade possible between humans and AGIs. Then, the economic interactions model is not going to provide humans any benefits.
BUT, we think that there will be scope for positive-sum trade substantially longer than is currently assumed. Most people thinking about this (including, I think, your question above) treat the most important question as: Can AI automate all tasks, and perform them more efficiently (with fewer inputs) than humans. This, we argue, following e.g., Noah Smith, isn't quite right. That is a question about who has the absolute advantage at a task. But for trade, what matters is who has the comparative advantage. Comparative advantage is not about who can do X most efficiently (in the simple sense), but instead who can do it at lowest opportunity cost.
AIs may face very high opportunity costs precisely because they are so capable at doing the things they value. We imagine, e.g., an AI whose ultimate goal is finding prime numbers. Suppose it is massively more efficient at this than humans--and also more efficient at all possible tasks. Suppose further that the AI is constrained at the margin in by compute. Thus, for each marginal A100 produced, the AI can either use it to find more primes (EXTREMELY HIGH VALUE TO AI) or use it to pilot a robot that maintains its own servers (low value to AI). Here, the AI may well prefer to use the A100 to find more primes and pay humans to maintain the server racks. Even better if it pay humans with something they value immensely but which is very cheap for the AI to produce. Maybe, e.g., a vaccine.
This is just a toy example, but I think it gives the idea. There are many quesitons here, especially about what resource will constrain AGI at the margin, and how rivalrous human consumption will be w/r/t that resource. If the AI is constrained at the margin, and blanketing the whole earth in solar panels is by far the cheapest way to get it, we may be doomed. If constrained to some degree by compute and power, and space-based fusion reactors are almost as cheap as solar, maybe we're fine. It's complicated!
Another thing worth mentioning here is that the existence of human-AI trade won't eliminate the human-human economy. Similarly, US-Korea trade didn't eliminate the intra-Korea economy. What it did do was help to push incomes up across Korea, including in sectors that don't export. This is for a bunch of reasons, including international trade's general productivity enancements via technology exhange, but also Baumol effects spilling over to purely domestic markets.
If we think of humans like the Asian Tiger economies, and the AIs like the US or EU economies, I think the world of long-run trade with AIs doesn't seem that bad. True, the US is much richer per capita than South Korea. But they are also very rich, compared with the globe and their own baseline. So we can imagine a world in which AIs do, indeed, have almost all of the property. But the total amount of property/consumption is just so vast that, even with a small share, humans are immensely wealthy by contemporary standards.
Just wanted to share a new paper on AI rights, co-authored with Peter Salib, that members of this community might be interested in. Here's the abstract:
AI companies are racing to create artificial general intelligence, or “AGI.” If they succeed, the result will be human-level AI systems that can independently pursue high-level goals by formulating and executing long-term plans in the real world. Leading AI researchers agree that some of these systems will likely be “misaligned”–pursuing goals that humans do not desire. This goal mismatch will put misaligned AIs and humans into strategic competition with one another. As with present-day strategic competition between nations with incompatible goals, the result could be violent and catastrophic conflict. Existing legal institutions are unprepared for the AGI world. New foundations for AGI governance are needed, and the time to begin laying them is now, before the critical moment arrives. This Article begins to lay those new legal foundations. It is the first to think systematically about the dynamics of strategic competition between humans and misaligned AGI. The Article begins by showing, using formal game-theoretic models, that, by default, humans and AIs will be trapped in a prisoner’s dilemma. Both parties’ dominant strategy will be to permanently disempower or destroy the other, even though the costs of such conflict would be high. The Article then argues that a surprising legal intervention could transform the game theoretic equilibrium and avoid conflict: AI rights. Not just any AI rights would promote human safety. Granting AIs the right not to be needlessly harmed–as humans have granted to certain non-human animals–would, for example, have little effect. Instead, to promote human safety, AIs should be given those basic private law rights–to make contracts, hold property, and bring tort claims–that law already extends to non-human corporations. Granting AIs these economic rights would enable long-run, small-scale, mutually-beneficial transactions between humans and AIs. This would, we show, facilitate a peaceful strategic equilibrium between humans and AIs for the same reasons economic interdependence tends to promote peace in international relations. Namely, the gains from trade far exceed those from war. Throughout, we argue that human safety, rather than AI welfare, provides the right framework for developing AI rights. This Article explores both the promise and the limits of AI rights as a legal tool for promoting human safety in an AGI world.