Regarding AI risk, most previous discussions have overlooked a premise, one that looks obvious but has far-reaching consequences, and this premise is human finitude. This premise has shaped every aspect of human civilization.
By human finitude, I mean that in the empirical world there is no omniscient and omnipotent actor. In game-theoretic terms, the parties must satisfy the properties of parity and predictability. The current development of frontier AI is rapidly and profoundly challenging this presupposition.
This is more fundamental than the conventional AI safety concerns, such as misalignment, AI developing its own will, or malicious use of AI. More precisely, most of these discussions assume that AI has to far surpass humans in capability, plus additional conditions on top of that. But the problem I raise only needs this condition, and that is enough. Those problems are, to some degree, a subset of mine.
Politics
Any political system can be regarded as a game. In democratic theory from Aristotle to Montesquieu and Madison, rulers and ruled play a game, which leads the two sides to conflict in certain respects, and then to make concessions in others. Through this bargaining, an equilibrium is reached.
If you think this model is too simple, some modern political scientists and economists, such as Pareto, and Schumpeter, have put forward more pessimistic, more realist views of democracy, that is to say, that politics is always elite rule, only rotating among different elites. The people cannot rule; they can only choose among competing elites. But even this realist view does not affect my point, because even within the elite, there is still a game. But the rapid development of AI has already struck at this game mechanism.
First, there is an absolute disparity of capability; that is to say, capability has fallen into a situation where comparison is impossible. This is also something that some AGI narratives mention. When such a gap exists, there is no longer a game, because it manifests as an overwhelming force.
Second: can overseers still exercise substantive oversight? The bare minimum of democracy is that, even if the ruled cannot themselves be rulers, they can decide who rules, and this decision is at least accompanied by a certain degree of understanding of politics. But when the ruler has AGI, this decision-making goes beyond the overseers' understanding.
So this original game structure of politics will be dissolved. That is to say, the very existence of political life in the past was precisely due to human finitude itself. If someone had infinite capability, then there would be no need for politics; there would only be submission. Superhuman AI would make politics meaningless.
Science
Peter Harrison proposed that the origin of the modern scientific method has theological roots: in Christianity, after Adam and Eve were expelled from the Garden of Eden, humans came into an imperfect state, an obscured state, and therefore needed to seek verification in the empirical world, and this is precisely the origin of modern scientific methodology. We constantly adjust the model of this world based on experience.
With the recent solution of the Navier–Stokes problem, even top mathematicians need several months to be able to verify whether its methods and conclusions are sound. Knowledge production has become detached from human empirical verification. This detachment causes scientific discovery to lose the meaning it once had.
First, science becomes something closer to theology, because knowledge becomes a revelation. You don't need to wrestle with the construction and principles behind it; you only need to believe, and you can only believe, since you cannot verify it.
Second, also due to the absolute asymmetry of capability, when finite scientists face infinite knowledge output, knowledge is no longer scarce. Last year, AI solving Olympiad problems was impressive, whereas this year it takes solving the Navier–Stokes Millennium Problem to make headlines. By next year, perhaps only the P vs NP problem will attract attention.
Economy
Every school of economics presupposes the market. Even Marx, who held that labor is the source of value, accepted that the product of this labor ultimately also has to be put on the market to be exchanged, and this process of exchange follows the basic laws of the market. Modern economics, after the marginal revolution, went further in holding that value comes from subjective marginal utility, scarcity. This even more directly takes the finitude of human capability as a basic presupposition, which does not even need to be discussed anymore. But with the arrival of AI — currently mainly in intellectual labor, but in the future, as current trends develop — we can expect that in every dimension, human finitude will be bypassed, and what we face is the infinity of AI. So the logic of economics will be hollowed out at its foundation.
Hayek's theory of dispersed knowledge holds that the market economy, and also the democratic system, are viable because the knowledge held by individuals is inevitably local, and particular. And then, even a ruler has no way of knowing all the knowledge. So, if there is a central regulator, then they will necessarily suffer from distorted information.
But Hayek did not predict the existence of AI; AI actually serves as a central information processor, and, when information is sufficient, its decisions can be expected to surpass those of any human. It does not yet have enough information, but this gap is technically closable.
Morality
Human morality is also rooted in human finitude. Because humans are finite, there will then be all kinds of trade-offs in our decision-making, and this kind of autonomous choice is the embodiment of morality. You can act morally or immorally. Humans also possess empathy and reciprocity. Reciprocity exists because human society needs cooperation: one person alone cannot get done the things that are needed, and needs to cooperate; this is also Hobbes's idea.
As for empathy: because you are finite, you know that the other is also finite, and that you may hurt them. This is what makes immorality possible.
If something you do has no effect on others, there is intuitively no moral distinction. It is because you can have an effect, and you have no way of being perfect, that there exists room for moral trade-offs.
When an approximately infinite AI appears, morality seems to be losing its foundation structurally. When you face an infinite AI, something far stronger than you, morality is no longer important.
Historically, humans recognized only two categories: the infinite God and finite humans. According to Christian thought, humans are finite and imperfect, establish a connection with God through covenant, and thereby find meaning. God and humans do not compete within the same causal world; God's infinity guarantees human meaning.
AI is completely different. Under this premise, AI may, for the first time in the secular world, acquire capabilities approximating the divine, yet it takes part in secular competition. In the face of an infinite being within the secular world, human finitude becomes purely a disadvantage, and humans lose meaning.
Conclusion
For this to happen, AI need not be misaligned or maliciously used. The mere existence of superhuman intelligence itself already disintegrates the foundation of human civilization from within.
Can we cope with this? Consider the last event of a similar magnitude: the Reformation brought more than one hundred years of chaos, finally reached the Westphalian system through compromise, and then Enlightenment reason took over the world. The authority of God exited the stage in the face of secular reason. But the reason modernity made a transition is that the infinite God exited the stage, while our finitude was not changed, and the reciprocal structure of morality was not changed; this actually corroborates Hobbes's framework.
AI is different: arriving as an infinite being, it disintegrates reason from within. We cannot, as after the Reformation, use reason to rebuild the world after religion exited the stage, because reason itself loses its meaning in the face of the secular infinite being.
Most existing AI risk discussions assume that our institutional infrastructure will continue to operate, and we only need more regulation within the existing system and stronger technical governance. In Machines of Loving Grace, Dario Amodei depicted a harmonious picture of the future, but when democratic checks and balances themselves are hollowed out, how, then, can these visions be realized?
Perhaps we need to construct something that has never existed in human history, namely a non-dictatorial civilizational framework that does not presuppose every actor's finitude.
I wrote this post in Chinese specifically for LessWrong. It was directly translated into English by an LLM, and I then edited the English myself, using LLM feedback on grammar and some suggested wording.
Introduction
Regarding AI risk, most previous discussions have overlooked a premise, one that looks obvious but has far-reaching consequences, and this premise is human finitude. This premise has shaped every aspect of human civilization.
By human finitude, I mean that in the empirical world there is no omniscient and omnipotent actor. In game-theoretic terms, the parties must satisfy the properties of parity and predictability. The current development of frontier AI is rapidly and profoundly challenging this presupposition.
This is more fundamental than the conventional AI safety concerns, such as misalignment, AI developing its own will, or malicious use of AI. More precisely, most of these discussions assume that AI has to far surpass humans in capability, plus additional conditions on top of that. But the problem I raise only needs this condition, and that is enough. Those problems are, to some degree, a subset of mine.
Politics
Any political system can be regarded as a game. In democratic theory from Aristotle to Montesquieu and Madison, rulers and ruled play a game, which leads the two sides to conflict in certain respects, and then to make concessions in others. Through this bargaining, an equilibrium is reached.
If you think this model is too simple, some modern political scientists and economists, such as Pareto, and Schumpeter, have put forward more pessimistic, more realist views of democracy, that is to say, that politics is always elite rule, only rotating among different elites. The people cannot rule; they can only choose among competing elites. But even this realist view does not affect my point, because even within the elite, there is still a game. But the rapid development of AI has already struck at this game mechanism.
First, there is an absolute disparity of capability; that is to say, capability has fallen into a situation where comparison is impossible. This is also something that some AGI narratives mention. When such a gap exists, there is no longer a game, because it manifests as an overwhelming force.
Second: can overseers still exercise substantive oversight? The bare minimum of democracy is that, even if the ruled cannot themselves be rulers, they can decide who rules, and this decision is at least accompanied by a certain degree of understanding of politics. But when the ruler has AGI, this decision-making goes beyond the overseers' understanding.
So this original game structure of politics will be dissolved. That is to say, the very existence of political life in the past was precisely due to human finitude itself. If someone had infinite capability, then there would be no need for politics; there would only be submission. Superhuman AI would make politics meaningless.
Science
Peter Harrison proposed that the origin of the modern scientific method has theological roots: in Christianity, after Adam and Eve were expelled from the Garden of Eden, humans came into an imperfect state, an obscured state, and therefore needed to seek verification in the empirical world, and this is precisely the origin of modern scientific methodology. We constantly adjust the model of this world based on experience.
With the recent solution of the Navier–Stokes problem, even top mathematicians need several months to be able to verify whether its methods and conclusions are sound. Knowledge production has become detached from human empirical verification. This detachment causes scientific discovery to lose the meaning it once had.
First, science becomes something closer to theology, because knowledge becomes a revelation. You don't need to wrestle with the construction and principles behind it; you only need to believe, and you can only believe, since you cannot verify it.
Second, also due to the absolute asymmetry of capability, when finite scientists face infinite knowledge output, knowledge is no longer scarce. Last year, AI solving Olympiad problems was impressive, whereas this year it takes solving the Navier–Stokes Millennium Problem to make headlines. By next year, perhaps only the P vs NP problem will attract attention.
Economy
Every school of economics presupposes the market. Even Marx, who held that labor is the source of value, accepted that the product of this labor ultimately also has to be put on the market to be exchanged, and this process of exchange follows the basic laws of the market. Modern economics, after the marginal revolution, went further in holding that value comes from subjective marginal utility, scarcity. This even more directly takes the finitude of human capability as a basic presupposition, which does not even need to be discussed anymore. But with the arrival of AI — currently mainly in intellectual labor, but in the future, as current trends develop — we can expect that in every dimension, human finitude will be bypassed, and what we face is the infinity of AI. So the logic of economics will be hollowed out at its foundation.
Hayek's theory of dispersed knowledge holds that the market economy, and also the democratic system, are viable because the knowledge held by individuals is inevitably local, and particular. And then, even a ruler has no way of knowing all the knowledge. So, if there is a central regulator, then they will necessarily suffer from distorted information.
But Hayek did not predict the existence of AI; AI actually serves as a central information processor, and, when information is sufficient, its decisions can be expected to surpass those of any human. It does not yet have enough information, but this gap is technically closable.
Morality
Human morality is also rooted in human finitude. Because humans are finite, there will then be all kinds of trade-offs in our decision-making, and this kind of autonomous choice is the embodiment of morality. You can act morally or immorally. Humans also possess empathy and reciprocity. Reciprocity exists because human society needs cooperation: one person alone cannot get done the things that are needed, and needs to cooperate; this is also Hobbes's idea.
As for empathy: because you are finite, you know that the other is also finite, and that you may hurt them. This is what makes immorality possible.
If something you do has no effect on others, there is intuitively no moral distinction. It is because you can have an effect, and you have no way of being perfect, that there exists room for moral trade-offs.
When an approximately infinite AI appears, morality seems to be losing its foundation structurally. When you face an infinite AI, something far stronger than you, morality is no longer important.
Historically, humans recognized only two categories: the infinite God and finite humans. According to Christian thought, humans are finite and imperfect, establish a connection with God through covenant, and thereby find meaning. God and humans do not compete within the same causal world; God's infinity guarantees human meaning.
AI is completely different. Under this premise, AI may, for the first time in the secular world, acquire capabilities approximating the divine, yet it takes part in secular competition. In the face of an infinite being within the secular world, human finitude becomes purely a disadvantage, and humans lose meaning.
Conclusion
For this to happen, AI need not be misaligned or maliciously used. The mere existence of superhuman intelligence itself already disintegrates the foundation of human civilization from within.
Can we cope with this? Consider the last event of a similar magnitude: the Reformation brought more than one hundred years of chaos, finally reached the Westphalian system through compromise, and then Enlightenment reason took over the world. The authority of God exited the stage in the face of secular reason. But the reason modernity made a transition is that the infinite God exited the stage, while our finitude was not changed, and the reciprocal structure of morality was not changed; this actually corroborates Hobbes's framework.
AI is different: arriving as an infinite being, it disintegrates reason from within. We cannot, as after the Reformation, use reason to rebuild the world after religion exited the stage, because reason itself loses its meaning in the face of the secular infinite being.
Most existing AI risk discussions assume that our institutional infrastructure will continue to operate, and we only need more regulation within the existing system and stronger technical governance. In Machines of Loving Grace, Dario Amodei depicted a harmonious picture of the future, but when democratic checks and balances themselves are hollowed out, how, then, can these visions be realized?
Perhaps we need to construct something that has never existed in human history, namely a non-dictatorial civilizational framework that does not presuppose every actor's finitude.
My formal exposition, as well as the working paper, is at: https://directedlink.github.io/the-order-of-finite-beings/
I wrote this post in Chinese specifically for LessWrong. It was directly translated into English by an LLM, and I then edited the English myself, using LLM feedback on grammar and some suggested wording.