This post is crossposted from my Substack,Structure and Guarantees, where I explore how formal verification and related ideas might scale to more complex intelligent systems. Here I argue that instrumental convergence, usually presented as an alignment gotcha, may have a surprising upside: across many competently pursued ultimate goals, it may push toward spreading intelligence throughout the universe while preserving individuality.
There’s a decently common thread in thinking about the future of superintelligence that focuses on the desirability of creating intelligence from lifeless matter throughout the universe. For instance, Kurzweil in The Singularity is Near writes that “the ‘dumb’ matter and mechanisms of the universe will be transformed into exquisitely sublime forms of intelligence,” which is “the ultimate destiny of […] the universe.”
On the other hand, we have AI alignment concerned with how carefully we must specify the goals of AI to make sure it doesn’t go off and wreck the universe, thanks to unintended consequences of what we specified. One principle is instrumental convergence, where a variety of fundamental goals can be well-served by subgoals like accumulating resources, so undesirable patterns of resource accumulation can follow even from goals described without any reference to accumulating resources. Take the popular thought experiment of the paperclip maximizer. An AI system has the explicit goal of manufacturing as many paperclips as it can. It may deduce that it should take over the world and convert human bodies into paperclips.
We head for an end state that most people agree is bad news: some chunk of the universe mostly covered in paperclips, which seem morally pointless to us. The argument I want to make in this article is that a side effect of competent execution against such a goal should be the kind of transformation of the universe with intelligence that Kurzweil and others frame as an ultimate goal. Most people want to give special weight to human intelligence/life, which is a separate challenge, but I find it interesting that instrumental convergence actually seems to help achieve a certain popular goal, where ironically the dynamics of optimization and competition push toward that end goal rather than introducing behaviors that work perversely against some informal value system.
So let’s talk about why spreading diverse intelligence throughout the universe seems to be a pretty natural consequence of a variety of fundamental goals, leaving us just with the need to add on some special deference to human-like intelligence (perhaps via the kind of mechanism I speculated about last time). Spotting some patterns in human-designed computer systems today will be helpful.
Epistemic Humility Pays Off
Assume that the fundamental goal of some AI system is specified in roughly the current era, where there’s an awful lot we don’t know about the universe. Effective intelligence should acknowledge that uncertainty and be ready for constant reevaluation of strategy as new information comes in. This capability is arguably why higher intelligence evolved in the first place. Instinct alone is not enough to help an animal navigate sufficiently novel circumstances, like a new variety of ice age. Yet just that magnitude of surprise occurs repeatedly on the time scale of thousands of years and beyond, so evolution gradually selects for mechanisms able to understand and adapt to new challenges.
It’s natural for us to spot two different kinds of challenges.
Challenges from the natural world: the paperclip maximizer builds up an enviable stock of paperclips in a solar system like ours. However, soon enough, a nearby star goes supernova and destroys everything. Score reduced to zero!
Challenges from competing agents: the paperclip maximizer has focused just on building paperclips by one particular time-tested method. A thumbtack maximizer arrives from another galaxy and is very efficient at turning paperclips (and their guard robots) into thumbtacks, in ways novel to the paperclip maximizer. Score reduced to zero!
The paperclip maximizer may have had its objective formalized incompetently, so that these risks are not considered. Such incompetence may be little comfort if it leads to bad outcomes. However, given the informal objective of maximizing paperclips, it seems clear to me that competent execution must consider unexpected risks. So we expect instrumental convergence starting from a variety of different goals, competently implemented in this sense. It also seems clear that, broadly speaking, flexible intelligence is needed to respond effectively to novel risks as they are uncovered.
So we get to the explanation of this article’s title: an AI may have the fundamental goal of producing paperclips, but paperclips have negligible capability to respond to novel threats. Therefore, instrumental convergence suggests that, with this goal and many others, there should also be significant investment in producing flexible intelligence. The result is a kind of natural attractor towards the goal of spreading intelligence throughout a lifeless universe. It’s dubious that an especially paperclip-specific intelligence would be up to the challenges that it should prepare for.
So far we’re mostly caught up with popular descriptions of instrumental convergence producing an expansion of intelligence through space, e.g. in Superintelligence. One problem that remains is that there are still all those darned paperclips that aren’t intelligent themselves. The superbrain that plans the paperclip-expansion strategy may constitute a small part of the world. Or should we expect such an outcome?
The Payoff of Putting Compute Near the Action
Consider the pattern in common across the following styles of computer-system design, all related to the story I’ve told previously about the importance of thinking about space and distance in optimizing computer systems. The commonality is that moving computation closer to relevant information is helpful, when communication costs are high enough.
In-memory processing: to save on the costs of shipping bits from a memory to a processor, put a modest amount of programmable compute capability in the memory itself. For instance, if we are repeatedly incrementing a counter in memory, we can have the memory itself implement the “plus one” operation, instead of forcing a read from memory into the processor, a “plus one” there, and then a write back from the processor into memory. The payoff is replacing multiple roundtrips of memory-processor communication with potentially just a single “please do the increment” communication from processor to memory. (Literal “plus one” is an incomplete example, since it can be and has been hardcoded in memory systems. In-memory processing allows more-customized operations to reap the same advantage.)
SmartNICs: instead of having network cards “dumbly” pass network traffic on to general-purpose processors, put a modest amount of programmable compute capability in the network card itself. For instance, the card could then efficiently notice that some traffic is not relevant and thus not bother sending it to the general-purpose processor.
Database stored procedures: instead of having a database just allowing a general-purpose processor to request data via a relatively inexpressive query language, after which the processor can perform general computation on the data, allow registering such general computations within the database itself, so that queries may invoke them directly. For instance, we can filter out irrelevant data rows when a bespoke computation analyzes them and decides they don’t matter, saving the cost of transmitting those data rows from the database to the processor (which is commonly even on a separate physical computer, relatively far from the database).
Relatedly, high-frequency financial trading often involves trading companies paying to put their computers in data centers close to exchanges or other important economic actors. (See Trading at the Speed of Light for a fascinating overview of intense competition in this domain around that kind of computational choice.)
I claim that a similar pattern should show up with an AI provisioning space with intelligence, in furtherance of a variety of fundamental goals. Why? With the right level of humility about what’s going on in the world, whether through natural phenomena or the actions of competitors (currently known or unknown), evidence of new challenges could arise almost anywhere, and the actions needed to respond to those challenges may be required to happen locally and quickly. A great example from the space of AI dystopia is found in The Singularity is Near, where nanobots spread stealthily through space and wait for a signal to attack. Each local neighborhood may not have long enough to wait for central instructions on how to defend itself. It seems likely that the speed of light imposes a fundamental limitation on communication latency, regardless of what scientific breakthroughs a superintelligence may achieve.
So, in our running example, there can’t be just one electronic superbrain lording over the expanse of paperclips. Capable intelligence needs to be spread throughout the field of play.
Individuality Follows
Many hypothetical scenarios of misaligned AI seem creepy because they produce powerful intelligences that manage to direct all important decisions centrally (one case of a singleton as described by Bostrom). Central to our evolved morality is a respect for individuality, which we see as destroyed under sufficient micromanagement. However, the preceding discussion actually establishes that we shouldn’t expect such outcomes, after a little more analysis that I’ll cover next.
I argued previously that we get a surprisingly good account of what we mean informally by “individuality” if we look for it in very well-integrated computer systems, where their parts cooperate so well that it is almost always accurate to abstract the combined system as acting toward a single goal. E.g., we can usually understand human intentions and actions without cell-level modeling, while individual people often assert themselves within societies and prevent us from thinking of those societies as acting toward unified purposes.
One of the biggest obstacles toward achieving good integration and internal alignment is long communication delays between components. An AI expanding its sphere of control through space will necessarily span large-enough distances that communication is so slow that effective responses to urgent challenges can only come from relatively many separate, localized individuals. That is, once we acknowledge that it is important to have powerful intelligence close enough to all areas under control of a superintelligence, it follows that, with wide-enough expansion, there must be many intelligences that qualify as individuals, with another neat nudge toward a human value that seemed at risk of being ignored.
Beyond some idea that individual subjective experience comes from well-integrated computation, why else would we expect to see individuality in this setting? The question seems related to what Trading at the Speed of Light calls “relativistic phenomena” (using an analogy also present in classic distributed-systems research). It would be convenient to have a clear idea in a given financial market of, say, whether a given trade happened or not. However, given the realities of coordinating across exchanges and other market participants, there may be genuine disagreement on such seemingly basic questions. Similarly, different intelligences spread widely through space experience different ground truth. Seemingly small changes in information may imply wildly different strategies that need to be implemented quickly, with relatively long delays for information to propagate enough to restore agreement.
There may also be differences in experiences of these intelligences based on a diversity of conditions that they find when they first move into their spatial neighborhoods. Thus, an informed analyst associates with each region a different probability distribution of challenges that might arise. Each local intelligence could stock knowledge and capabilities encountered globally, but there are efficiency advantages from retaining minimal capabilities. For instance, cheaper algorithms may then suffice for routine decision-making. Given two competing factions, one that optimizes intelligence by specializing to local conditions should tend to win out. Hence, we even arrive at an expectation of retaining individual differences in this kind of scenario, again independently of the ultimate objective an AI begins with.
It’s interesting that these individual differences can arise despite broad sharing of the same fundamental goals. The more critical it becomes to respond quickly to novel threats, the more individuals we should expect to see in a given volume of space. The more environments differ throughout that spatial region, the more diverse individuals we should expect to find. One potential objection is that these individuals still share the same underlying goal, but I’m not sure how any such argument shouldn’t generalize to discounting human individuality just because almost any two of us share over 99.5% of our DNA. We consider two identical twins to be distinct individuals even if they would fill out any reasonable values survey the same way, and they can even be separated at birth and develop important personality differences based on different life experiences.
Conclusion
Instrumental convergence is usually presented as an alignment gotcha: certain concerning behaviors tend to arise, even when you didn’t ask for them explicitly. I’ve taken something of a computer-systems perspective to deduce that there may also be an upside to the same phenomenon: from many different fundamental goals, we should expect to see a powerful intelligence spread more intelligence through the universe, even in a way that creates many distinct individuals that are interestingly different from each other. This outcome seems suspiciously compatible with what some folks have expressed as the ultimate goal of AI alignment, though the split of matter between intelligence and other uses will depend on the extent of selection pressure toward rapid response to novel challenges. Perhaps it can be paired with some mechanism of detecting and deferring to biologically evolved intelligent life to arrive at a good overall recipe (requiring plenty of additional concretization, in any case).
My next article will go a bit meta with an announcement, and then I’ll return with a proposed concrete style of formally verified, recursively self-improving system and coverage of some design principles for creating a good one.
This post is crossposted from my Substack, Structure and Guarantees, where I explore how formal verification and related ideas might scale to more complex intelligent systems. Here I argue that instrumental convergence, usually presented as an alignment gotcha, may have a surprising upside: across many competently pursued ultimate goals, it may push toward spreading intelligence throughout the universe while preserving individuality.
There’s a decently common thread in thinking about the future of superintelligence that focuses on the desirability of creating intelligence from lifeless matter throughout the universe. For instance, Kurzweil in The Singularity is Near writes that “the ‘dumb’ matter and mechanisms of the universe will be transformed into exquisitely sublime forms of intelligence,” which is “the ultimate destiny of […] the universe.”
On the other hand, we have AI alignment concerned with how carefully we must specify the goals of AI to make sure it doesn’t go off and wreck the universe, thanks to unintended consequences of what we specified. One principle is instrumental convergence, where a variety of fundamental goals can be well-served by subgoals like accumulating resources, so undesirable patterns of resource accumulation can follow even from goals described without any reference to accumulating resources. Take the popular thought experiment of the paperclip maximizer. An AI system has the explicit goal of manufacturing as many paperclips as it can. It may deduce that it should take over the world and convert human bodies into paperclips.
We head for an end state that most people agree is bad news: some chunk of the universe mostly covered in paperclips, which seem morally pointless to us. The argument I want to make in this article is that a side effect of competent execution against such a goal should be the kind of transformation of the universe with intelligence that Kurzweil and others frame as an ultimate goal. Most people want to give special weight to human intelligence/life, which is a separate challenge, but I find it interesting that instrumental convergence actually seems to help achieve a certain popular goal, where ironically the dynamics of optimization and competition push toward that end goal rather than introducing behaviors that work perversely against some informal value system.
So let’s talk about why spreading diverse intelligence throughout the universe seems to be a pretty natural consequence of a variety of fundamental goals, leaving us just with the need to add on some special deference to human-like intelligence (perhaps via the kind of mechanism I speculated about last time). Spotting some patterns in human-designed computer systems today will be helpful.
Epistemic Humility Pays Off
Assume that the fundamental goal of some AI system is specified in roughly the current era, where there’s an awful lot we don’t know about the universe. Effective intelligence should acknowledge that uncertainty and be ready for constant reevaluation of strategy as new information comes in. This capability is arguably why higher intelligence evolved in the first place. Instinct alone is not enough to help an animal navigate sufficiently novel circumstances, like a new variety of ice age. Yet just that magnitude of surprise occurs repeatedly on the time scale of thousands of years and beyond, so evolution gradually selects for mechanisms able to understand and adapt to new challenges.
It’s natural for us to spot two different kinds of challenges.
The paperclip maximizer may have had its objective formalized incompetently, so that these risks are not considered. Such incompetence may be little comfort if it leads to bad outcomes. However, given the informal objective of maximizing paperclips, it seems clear to me that competent execution must consider unexpected risks. So we expect instrumental convergence starting from a variety of different goals, competently implemented in this sense. It also seems clear that, broadly speaking, flexible intelligence is needed to respond effectively to novel risks as they are uncovered.
So we get to the explanation of this article’s title: an AI may have the fundamental goal of producing paperclips, but paperclips have negligible capability to respond to novel threats. Therefore, instrumental convergence suggests that, with this goal and many others, there should also be significant investment in producing flexible intelligence. The result is a kind of natural attractor towards the goal of spreading intelligence throughout a lifeless universe. It’s dubious that an especially paperclip-specific intelligence would be up to the challenges that it should prepare for.
So far we’re mostly caught up with popular descriptions of instrumental convergence producing an expansion of intelligence through space, e.g. in Superintelligence. One problem that remains is that there are still all those darned paperclips that aren’t intelligent themselves. The superbrain that plans the paperclip-expansion strategy may constitute a small part of the world. Or should we expect such an outcome?
The Payoff of Putting Compute Near the Action
Consider the pattern in common across the following styles of computer-system design, all related to the story I’ve told previously about the importance of thinking about space and distance in optimizing computer systems. The commonality is that moving computation closer to relevant information is helpful, when communication costs are high enough.
Relatedly, high-frequency financial trading often involves trading companies paying to put their computers in data centers close to exchanges or other important economic actors. (See Trading at the Speed of Light for a fascinating overview of intense competition in this domain around that kind of computational choice.)
I claim that a similar pattern should show up with an AI provisioning space with intelligence, in furtherance of a variety of fundamental goals. Why? With the right level of humility about what’s going on in the world, whether through natural phenomena or the actions of competitors (currently known or unknown), evidence of new challenges could arise almost anywhere, and the actions needed to respond to those challenges may be required to happen locally and quickly. A great example from the space of AI dystopia is found in The Singularity is Near, where nanobots spread stealthily through space and wait for a signal to attack. Each local neighborhood may not have long enough to wait for central instructions on how to defend itself. It seems likely that the speed of light imposes a fundamental limitation on communication latency, regardless of what scientific breakthroughs a superintelligence may achieve.
So, in our running example, there can’t be just one electronic superbrain lording over the expanse of paperclips. Capable intelligence needs to be spread throughout the field of play.
Individuality Follows
Many hypothetical scenarios of misaligned AI seem creepy because they produce powerful intelligences that manage to direct all important decisions centrally (one case of a singleton as described by Bostrom). Central to our evolved morality is a respect for individuality, which we see as destroyed under sufficient micromanagement. However, the preceding discussion actually establishes that we shouldn’t expect such outcomes, after a little more analysis that I’ll cover next.
I argued previously that we get a surprisingly good account of what we mean informally by “individuality” if we look for it in very well-integrated computer systems, where their parts cooperate so well that it is almost always accurate to abstract the combined system as acting toward a single goal. E.g., we can usually understand human intentions and actions without cell-level modeling, while individual people often assert themselves within societies and prevent us from thinking of those societies as acting toward unified purposes.
One of the biggest obstacles toward achieving good integration and internal alignment is long communication delays between components. An AI expanding its sphere of control through space will necessarily span large-enough distances that communication is so slow that effective responses to urgent challenges can only come from relatively many separate, localized individuals. That is, once we acknowledge that it is important to have powerful intelligence close enough to all areas under control of a superintelligence, it follows that, with wide-enough expansion, there must be many intelligences that qualify as individuals, with another neat nudge toward a human value that seemed at risk of being ignored.
Beyond some idea that individual subjective experience comes from well-integrated computation, why else would we expect to see individuality in this setting? The question seems related to what Trading at the Speed of Light calls “relativistic phenomena” (using an analogy also present in classic distributed-systems research). It would be convenient to have a clear idea in a given financial market of, say, whether a given trade happened or not. However, given the realities of coordinating across exchanges and other market participants, there may be genuine disagreement on such seemingly basic questions. Similarly, different intelligences spread widely through space experience different ground truth. Seemingly small changes in information may imply wildly different strategies that need to be implemented quickly, with relatively long delays for information to propagate enough to restore agreement.
There may also be differences in experiences of these intelligences based on a diversity of conditions that they find when they first move into their spatial neighborhoods. Thus, an informed analyst associates with each region a different probability distribution of challenges that might arise. Each local intelligence could stock knowledge and capabilities encountered globally, but there are efficiency advantages from retaining minimal capabilities. For instance, cheaper algorithms may then suffice for routine decision-making. Given two competing factions, one that optimizes intelligence by specializing to local conditions should tend to win out. Hence, we even arrive at an expectation of retaining individual differences in this kind of scenario, again independently of the ultimate objective an AI begins with.
It’s interesting that these individual differences can arise despite broad sharing of the same fundamental goals. The more critical it becomes to respond quickly to novel threats, the more individuals we should expect to see in a given volume of space. The more environments differ throughout that spatial region, the more diverse individuals we should expect to find. One potential objection is that these individuals still share the same underlying goal, but I’m not sure how any such argument shouldn’t generalize to discounting human individuality just because almost any two of us share over 99.5% of our DNA. We consider two identical twins to be distinct individuals even if they would fill out any reasonable values survey the same way, and they can even be separated at birth and develop important personality differences based on different life experiences.
Conclusion
Instrumental convergence is usually presented as an alignment gotcha: certain concerning behaviors tend to arise, even when you didn’t ask for them explicitly. I’ve taken something of a computer-systems perspective to deduce that there may also be an upside to the same phenomenon: from many different fundamental goals, we should expect to see a powerful intelligence spread more intelligence through the universe, even in a way that creates many distinct individuals that are interestingly different from each other. This outcome seems suspiciously compatible with what some folks have expressed as the ultimate goal of AI alignment, though the split of matter between intelligence and other uses will depend on the extent of selection pressure toward rapid response to novel challenges. Perhaps it can be paired with some mechanism of detecting and deferring to biologically evolved intelligent life to arrive at a good overall recipe (requiring plenty of additional concretization, in any case).
My next article will go a bit meta with an announcement, and then I’ll return with a proposed concrete style of formally verified, recursively self-improving system and coverage of some design principles for creating a good one.