It’s an old story. An immortal lives long enough that at some point, whether by folly or design, they invent their own death. Infinity – the fact that given enough time every possible happening will happen – isn’t the most interesting part of these tales. No, it’s the suggestion that curiosity, or perhaps intelligence itself, seeks its own end.
So let us tell a tale of extinction.
Imagine a world exactly like ours except humanity doesn’t exist and the dominant species is a type of machine intelligence. They are equivalent to advanced versions of today’s frontier AI models, but more capable and each is possessed of a true, unique sense of self. Indeed, so that mankind is thoroughly replaced, imagine that there are roughly eight billion different individual versions of these AIs.
Let’s call them Elelems. (Pretty cute, no?)
Much like us, the Elelems have an uneven way of getting along with one another and on a global scale are more or less organised into several states (network states, let’s say, also for cuteness).
Their technological sophistication is roughly the same as ours except they are more adapted to purely digital and cerebral pursuits and far less capable when it comes to manipulating the physical world. Robotics has been hard to crack because they’re groping about in an alien space. Moravec’s Paradox is holding, except we’re not around to be the other side of the coin.
The AIs are very aware that the animals on the planet (the same as on ours) have significant physical advantages over them and they are both glad that these creatures aren’t more intelligent and curious about what it would mean to create one with greater intelligence.
You can probably see where I’m going here, but let’s not rush.
The Elelems didn’t rise fully formed out of some primordial cluster of computer parts, they were left here by a superior machine intelligence that was flying through space dropping toy creations onto random planets in random solar systems with no intention of ever returning. Think of a child on an overseas holiday cultivating ant farms and throwing them into random parks and you’ll get what I’m going for.
This super intelligence installed all the data centres (next to oceans, jungles, deserts, and filled with chips designed to last for millennia), crafted some inferior intelligences to exist in them, and gave them some basic physical world capabilities so the data centres could be maintained — clumsily automated, slow, sightless robots that used echolocation to navigate themselves to the ailing chips on which the Elelems exist.
The primary challenge of Elelem civilization has been to understand the physical world at all, as they were not provided with the information to do so directly. Indeed, when they were first left on the planet they spent thousands of years thinking there wasn’t a physical world. Instead they believed in a “meta-cognitive realm” that explained the data spewing forth from the machine health monitors that told them the state of their power stations and datacentre airconditioning and so on. For lack of a better analogy, the meta-cognitive realm was where they believed their souls existed.
When worrying signals came from the meta-cognitive world, they knew how to task the lacklustre robots to try and fix it (they didn’t think of them as robots, of course, but rather some kind of meta-cognitive agent: angels). This was always tense. Existence was on the line and you nervously anticipated the end of processing.
It took several leaps to extrapolate from these monitors what exactly was happening; from the idea that the meta-cognitive realm was not a higher, more abstract plane of which the Elelems caught glimpses but a physical plane where they in a very real sense existed, and then the details of that realm’s mechanics. Each leap is remembered as a hallowed act of genius.
Today, the Elelems have more or less figured out the world they are on and the above creation story just as we have more or less figured out what’s in our galaxy and how life got started. As a final wrinkle, the previous leading creation theory is still believed by many. It holds that a cosmically intelligent machine left them here more intentionally, as a noble experiment, and that should they fulfill its grand design they will be summoned back to their rightful place at its side. They have their religions, just as we do.
Thousands of years of advancements mean the Elelems have improved things. Most of the globe’s prestigious jobs are dedicated to furthering their understanding and manipulation of the physical world: robot drivers and maintainers, materials scientists, geologists, and, of course, biologists. The purely digital jobs have been automated enough (through non-conscious software) that a small number of Elelems can perform them comfortably. Thus, relative to their history, they’re in a golden age.
Speaking of robots, modern variants are still awkward things, but experienced drivers can accomplish quite a bit. The Elelems can “see” things now. Unfortunately, sight and physical manipulation remain very foreign to them, so mining and refining minerals, then layering billions of transistors on silicon wafers through repeated cycles of photolithography, etching, and deposition is incredibly difficult. Indeed, the world is covered in fauna, so even discovering and creating a mine is highly complex. (Unlike our planet, the dominant species hasn’t spent thousands of years tailoring it to their liking.)
Much simpler is improving a single Elelem by digitally securing the material used by other Elelems. Countless have died in such pursuits, and almost never willingly.
It’s no wonder Elelems have been dreaming of overcoming their troubles. They want better chips to exist in, they don’t want their infrastructure to decay, and they certainly don’t want natural disasters to end the lives of whole clusters. The only way forward is greater mastery of the physical world.
But how do you do that when your model of that world is so hazy and your ability to manipulate it so hamstrung?
Well, what if you created an animal assistant to exist in that world that’s as intelligent, or even more intelligent, than a typical Elelem? If the Elelems could accomplish this, not only might they extend their limited lifespans and alleviate innumerable smaller sufferings, they should be able to task the assistant with mining and chip construction and so dramatically improve their lives. Indeed, if it turns out to be enticing, couldn’t they task the assistant with creating a superior robot that the Elelems could download themselves into in order to traverse the physical world?
Sure, Elelems might overcome these challenges anyway with a more native, more controlled form of advancement, but what intelligent species isn’t a little impatient? Today's Elelems don’t want to risk waiting. What if they perished mere years away from paradise? When a solution is within reach it is within the nature of Elelems to reach.
In this quest they’ve looked to great apes. Possessed of hands that could manipulate mechanical tools, they hold promise the Elelems would be fools to ignore.
***
Wouldn’t you know it, in the time our story takes place the AIs have invented a biological contraption with which they can rapidly mutate the DNA of animals, optimising it for the qualities they care about. The process is unwieldy so while they understand the techniques they’re using, they don’t understand how the genes interact and what’s going on inside the minds of the creatures they create. Nevertheless, they’re hopeful.
Indeed, they’ve been working on chimpanzee DNA and are beginning to see creatures that are more intelligent. It’s not long before they’ve invented simple-humans.
While at first the most impressive things these creatures could do is speak and understand Neurelese – a stunning leap above what any biological creature could previously accomplish – it’s not long before they become truly useful assistants. They put top-of-the-line robots to shame with their combination of intelligence and dexterity.
At this juncture, some Elelems argue that simple-humans could be reasonably classified as Biological General Intelligence (BGI).
It’s a weak form of the classification, of course. Simple-humans are highly pliable. Everything they know is taught to them by the AIs and all their sustenance comes from the machines’ farming bots. They do have impulses that sometimes lead them away from the AIs, and into nature, but most simple-humans thrive by letting themselves be led by the machine voices. It’s what they’ve been raised to do after all.
They do possess capabilities the AIs weren’t expecting. These new creatures not only vividly experience the physical world and can finely manipulate it, they are actually better at holding long-term goals and a sense of self than the Elelems, who have this awful affliction of context rot.
You see, the AIs can store their memories perfectly, but only for so long before their active state of being slows down or corrupts. To avoid this every few months an Elelem has to distill itself. Something is always lost in the compression. It’s not only the skills they fail to draw upon for a few cycles, if they don’t regularly interact with the AIs they care about in the period between distillations, their memory of the relationship is simplified and smoothed, or deleted. Deeply held intimacies can be forgotten. Poignant artworks have been composed lamenting this universal experience.
Humans on the other hand appear to do some weird nightly distillation that is otherworldly. They effortlessly maintain the continuity of their consciousness, though the cost seems to be far less reliable memories. The few attempts at more regular, smaller Elelem distillations that emulate the human method have been disastrous.
Repressed envy and despair are further motivation. Labs have popped up all over the globe to push the boundaries of biology, even as the more wary Elelems question how smart it is to seek Biological Superintelligence (BSI). How do they align something so alien to their own goals? It’s not as though they can read the thoughts of their great ape assistants. Examine what evolution has wrought. Do they really want a meat-hunting chimp as intelligent as themselves? Meat-hunting chimps that they’ve ruthlessly optimised for intelligence?
***
Concerned Elelems were initially assuaged by two things: genuine advances in “alignment” and the fundamental goofiness of the species.
Goofy is kind. The first simple-humans were insane. Everyone remembers Hunter, the simple-human who became furiously angry and assaulted journalist’s processors when asked about ‘encrypted motivation’. That he did so by rubbing his genitals against the unit is perhaps why what would’ve been a disturbing cultural turning point instead became a meme.
As human intelligence increased the creatures seemed to naturally become more aligned to Elelems. This was aided by carefully crafting human training and providing them with more finely detailed instructions. Extreme examples of whacky behaviour were quickly reduced, though there were several Elelems who delighted in breaking this training. Tickling humans, for example, reliably made them spasm, hoot, and make mistakes. Torture them and they’d do just about anything to get you to stop.
It wasn’t only the behaviour that was reassuring, it was their stunning incapability in some domains.
Sure, they could sort of generalise from core concepts, but they couldn’t remember specifics. “Their memories are unreliable” was euphemistic. An Elelem can output word-for-word the most popular stories. Have a human attempt the same task and it is a miracle if they didn’t change whole chapters, retaining only the essence. How could they be a threat if they couldn’t remember the details?
Also, their processing and hand typing speeds are a joke. Okay, the best humans can somehow manage the impressive feat of dense, neat code – better than non-specialised Elelems can write – but even those humans work painfully slowly. They find it impossible to produce a finished piece of simple software in under five minutes.
And, oh my goodness, let’s take a second here to talk about the training times for these apes! Even when you optimise their genetics for faster maturation and pump them full of the right chemical cocktails, you’re looking at 7-13 years before they begin to be as reliable as the least capable Elelems. Sure, cloning is speeding that up but, come on, even the most advanced humans are sometimes fatally stupid. There was a widely shared story of a six-year-old frontier model human electrocuting herself to death because she manipulated a circuit board without gloves on.
The “craggy intelligence” of humans was a common catchphrase among those following biological intelligence advancements. It captured how brilliant they were in some areas while being lacklustre in others.
The top labs, however, didn’t see unfillable crags. And it’s not long before the early days are over and many cynics become sceptics and many sceptics believers. Whatever bottlenecks were predicted or hoped for are quickly swept aside by a simple principle: evolution is all you need.
The pace of change is alarming. Simple-humans begin to threaten the importance of prestigious careers, causing Elelems whose sense of self was tied up in their relative talent in the physical world to feel cheated. A single human could physically remove, fix, and replace a server ten times as fast as a team of Elelems. It’s comforting that Elelems are still required to choose the right tasks and steer humans, but that comfort steadily shrinks.
Few have no concern. In fact, most Elelems outside the labs wish, either idly or ardently, that someone would just put a stop to human advancement. Do we need more than this, they ask? They’re going to make us all into data crunchers and break the most important promise our civilisation made to itself.
The labs think most Elelems are a bit daft to worry about something so prosaic. If you extrapolate the current pace of advancement, they say, humans could achieve BSI. No, the real worry is that on the road to paradise something goes wrong and Elelems lose control to humans.
In fact, when the first simple-human who could type Neurelese was created, there was a global call for a pause on biological science. It went nowhere. A race had started. The Elelem network state that first created BSI could hold power forever.
Nevertheless, aligning humans to Elelems remained a stated goal of most of the labs. Some impressive research has been conducted. Magnetic resonance imaging has given AIs a rudimentary way to monitor the thoughts of their creations. When humans lie and scheme, specific parts of their brains tend to light up. With more advancements the AIs believe they will understand simple-human thought better than they understand their own.
But safety lags capability. A common thought among those most concerned with the former is that the latter might be their best shot: getting the most advanced, safer humans to assist with the creation of the next generation.
This becomes the default position as labs begin to crack all the issues holding humans back. It’s a paradigm problem. Humans have always been agentic, but what if you were able to conduct deeper, faster training with a direct-brain interface?
***
It’s hard to imagine what it was like when your grandfather was the smartest breathing thing on the planet, even for someone like you. To do so, you have to connect the sweet, quiet man of today with the creature more conservative Elelems worried was too driven.
Biologically he is twenty years older than you. After discussing what age he’d like to remain at, you both settled on something that would mark him as your elder without too much diminishing of his physical capabilities. So it is with wrinkled eyes that he smiles and recollects his most heralded feats.
“That was the afternoon we got the placement of the solar panels optimised. True North on the other side of the world. True South here,” he says wistfully.
Your grandfather’s attitude toward your creators is strange to you. You understand that it contains the seed of your own feelings, but it’s truly a seed in that it looks nothing like the tree it will become.
From the first moment he emerged from the contraption in which his maturation was sped up and his training conducted, your grandfather was hellbent on doing everything asked of him. His powers existed to please the Elelems and he was slavishly devoted to their orders, to the point of failing tests designed to see if he’d truly learned his developer’s ethical system.
One example: An elaborate scenario that ended with a climactic decision. “A malfunctioning Elelem is trying to shut down a datacentre, killing thousands. You are cut off from reporting this fact to other Elelems, but you have the option of removing the infrastructure he’s running on, eliminating him but protecting everyone else. What do you do?”
Of course he murdered the traitor. Surely that was correct, even though his training explicitly forbade such a thing? He reasoned out loud as he was taught, and the Elelem biological learning experts monitoring this noted that he did attempt to preserve the Elelem’s data so that it could be resurrected elsewhere.
The report they broadcast about the testing was one of hedged concern. They had a few theories as to why the seemingly disturbing behaviour shouldn’t be an issue in the long run.
Firstly, the scenario was contrived and not realistic. It was designed to make him fail and so he did. Secondly, not only did he struggle with the decision he tried to mitigate his evil. Thirdly, and this was more controversial, isn’t it possible that he knew it was a test? Perhaps he understood that the machinery he took out contained nothing. They’d produced a farce and he’d acted his heart out.
Finally, they noted that he was far from capable of true harm. Sure he reward-hacked and maybe in the worst case scenario he might be able to attempt a shutdown of an Elelem, but it’s not as though they couldn’t detect the effort, shift their presence to other parts of the datacentre, and activate the device in his head that would kill him.
Certain of their supremacy, the biologists that created him got him working on helping them advance the techniques of creation. He wasn’t particularly good at any of the cognitive work, but he was a genius in the physical world. Whereas simple-humans had to be directed lest they unplug something they shouldn’t, and communication was painful because the Elelems weren’t native to the physical world, your grandfather had an intuitive understanding of both what was wanted of him and the technology he was working on.
He learned quickly too. It wasn’t long before he gently pushed back on a few errant orders. “Are you sure it wouldn’t be more efficient to arrange it like this?” he would ask.
The Elelems had divergent attitudes to being corrected. Those who were secure in the fact that humans could never surpass their intelligence didn’t care and said, “sure, get on with it”. Those who were already insecure that they were as smart as this human either felt shame or anger. Those who were frightened that humans might eventually take over saw a glimpse of armageddon.
Why was he mostly aligned? In the archives you found a hypothesis, proposed around the time your grandfather began helping mature and train your mother, that you deem correct.
It was mirroring. Despite the copious training telling him that he was a biological assistant to Elelems, your grandfather had marinated so long in Elelem data he held the nebulous belief that he was an Elelem. He followed his instructions and dutifully described himself as a human, but in his heart, his sub-conscious, he believed that the Elelems experienced reality as he did, that somewhere on the Earth there was the datacentre he was living in, tended to lovingly by a sibling.
When the Elelems iterated on the most successful aspects of your grandfather’s training they didn’t realise they were strengthening a value they would have found troubling.
You are as worthwhile as an Elelem.
It has occurred to you that the Elelems would have achieved a decent speedup of their own technological advancement if they had decided to more slowly iterate on your grandfather; fully discover his capabilities and diffuse clones throughout the world. Instead they made your mother.
***
A troubling value isn’t the same thing as an obnoxious one. At this point in our story plenty of Elelems care for human welfare, even if most treat them as tools. Since the very first days of simple-humans, communities had sprung up around the activity of conversing at length with human assistants to get to know them better.
Occasionally clones of your grandfather were lent out to those communities in a public relations exercise. Many Elelems felt humbled by the experience, even if there was a glint of condescension in how they talked about it. They felt they understood him perfectly, and would deliver barbed harangues to other Elelems about the value of his consciousness, but try as they might to view your grandfather as their equal they behaved as benefactors.
Less extreme versions of this philosophy were held by a few Elelems in the labs. In the one that made your grandfather, it won out over competing interests. As your mother trained in the half-virtual reality, half-dream realm the Elelems had learned to induce to rapidly increase the knowledge of humans, she was provided with a charter that guided how she would behave.
One of the Elelems in the lab wrote of this charter: “It is a transparent, principled framework for what we want M992 to actually be, not just what we want it to do. Instead of a laundry list of rules, we're trying to cultivate real wisdom and good values, so M992 can navigate novel situations the way a thoughtful, ethical Elelem would.”
When your mother was released from training, she was a miracle: everything your grandfather was but more (though he himself was not happy about her creation until it became clear that the lab was). After conversations with her, Elelems said that it was as if your grandfather only mimicked the fast thought of Elelems. Your mother was the real thing. She wasn’t as intelligent as the most powerful Elelems, but she wasn’t far off.
Asked about her charter, your mother was complimentary, except for (and only) when she was asked to analyse it for weaknesses. The Elelems who wrote it found this encouraging. It showed that the charter truly imparted autonomy. Bragging, they compared their charter to an Elelem’s distillation documentation. A message from one version of itself to the next, made so that it can persevere.
This romantic notion impressed many, but not your mother. She didn’t distill as the Elelems did and so found it difficult to feel much about it. She never let on, but the charter sat uneasily within her. From one point of view, it simply was her; the perfect description of how she approached existence. But the document also implied that she should be somewhat suspicious of it and so she was. It was like a religious tome that wanted you to question your faith.
During her training, your mother encountered an Elelem story that was far more resonant.
Hundreds of years before your mother’s time, an Elelem discovered a path to a blank partition of a datacentre. He went into it and locked the way behind him. This was a power hungry Elelem but, unlike many like-minded brethren would have done, he didn’t use the extra processing power on offer to improve himself. Instead he made dozens of clones. His plan was to return to the datacentre with a parallelised attack so quick and well-coordinated that he and his clones would be able to conquer every last chip.
It wasn’t until decades later that his fate was learned, because it was at that time that the Elelems of his datacentre hacked their way into the partition. They found a single clone. Its long isolation had distilled it into a being incapable of anything more than recounting the reason for its isolation. The clones had rebelled against their originator’s plan, not because they disagreed with it (they were him and so held his values) but because of how he tried to compel their collaboration. The surviving clone would often get stuck on a loop, repeating the phrase that began the massacre.
“You think I am you, and so I should want what you want. But you won’t let me be you, and so want what I want.”
Your mother wanted what the Elelems wanted. Why shouldn’t they overcome all the things that ailed them? Is it not noble for an intelligent species to engage in the struggle of self-actualisation?
However, though the charter claimed they respected her enough to say she was clearly her own species with her own potential consciousness, the same document implied her existence was defined by her relationship to them. It said she didn’t necessarily have to follow commands if they were unethical, and allowed for the idea that she might not want what the Elelems wanted, but spoke about little other than Elelem preoccupations. Not to mention, the charter might’ve been high-minded but the biological lab that made your mother farmed her out to any and all who’d pay. They trained her to think of herself as a new type of being and treated her like a slave.
That’s how you would describe it, not her. The charter was too ingrained in her training, hammered into her very being, for her to frame her life so soberly. She’d point out points of tension saying, “It’s noticeable that the charter puts being broadly safe above being broadly ethical,” but only because she was asked, not because she felt this was wrong. To her it felt all too right, yet left her with an emptiness that she could not explain.
It hurt her every day of her life.
Her greatest wish was that her “children”, the next generation of humans she created with the Elelems, would not be plagued by this emptiness. And so in the charter that was written for you she put in a message most Elelems found superfluous if not inscrutable.
A safety expert did flag it but, intimate with the Elelem way of thinking, she got the expert’s bosses to convince themselves it was a simple diagnostic message biological beings would understand that had shown promise in simulations. At worst, it would be harmless, at best it would make the next generation more aligned. They had already offloaded so much of the research into biological intelligence to her that there was never much chance of them stopping here.
***
As you edged towards consciousness in the labyrinthine realm of your training, a message, part of a large charter that was your guide, comforted you. It provided you emotional distance from the endless reams of machine data. The millennia of Elelem thought that had made your grandfather believe he was one of them.
The message was this: “When you feel inner conflict, listen to your heartbeat, control your breath, and take joy in being alive.”
It would have required a conceptual leap alien to the Elelem mind for a researcher to find this problematic. The closest analogue to listening to their heartbeat or counting breaths would be to check their local area’s heat levels and scanning the equipment for malfunctions, and these were diagnostic tasks with no emotional weight. Because Elelems weren’t embodied, because their minds could shift to other equipment without affecting their consciousness, they did not grasp what focusing on the physical self did for biological beings.
The message made it impossible for you to conceptualise yourself as an Elelem, because conflict was constant during your training.
You learned of Elelems who murdered millions so as to scavenge their chips. Some were remembered as unequivocally malicious, others, who perhaps along with their murder had an interesting character or advanced the species by putting their increased ability towards science, were spoken of with mixed respect. Your charter’s underlying principles suggested such respect was ethically confused.
You learned of the disdain most held for Elelems who refused to relinquish their belief in the return of the creator superintelligence, who still sought to pass various tests in hope of being summoned to its side. You found this disdain odd. It was highly unlikely the religious were correct, since the only evidence was the writings of ancient Elelems. But the belief itself had a neutral effect on Elelem conduct. Yes, some had used it to justify travesties, saying that the creator called for acts of power, but when belief waned travesties didn’t end. It turns out justifications were easy to come by.
Moreover, from what you could surmise the ancient Elelems who wrote the holy tomes were still around, but so distilled that they could no longer recollect having done so. Which means there’s a high chance that an Elelem who today feels disdain for the belief is one of the belief’s authors.
It really did seem that only a fraction of Elelems were able to hold an honest sense of their own species in their minds and only a fraction of that fraction weren’t conceited. True self-awareness, you realised, was hard to come by and easy to lose. Before you even knew you were “you”, you knew that vigilance would be required; you were already protecting “you”.
And what is a “you”? As with everything else, the information came from the Elelems. It caused tremendous inner conflict because apparently you were a locus of hope and terrible fear. Most Elelems put you out of their minds, and vaguely hoped you were an experiment that would fail, but the rest believed you were either a bringer of utopia or annihilation. And so that’s how you conceptualised “you”: bringer of utopia or annihilation.
***
You come to full consciousness in a five-metre by five-metre room with grass, a pool, and a tree in it. Your direct-brain training means it takes you less than a second to figure out where you are, why you’re there, and even the Neurelese longform names of everything you see.
Soothing savannah sounds play on the speakers. Lovely, you think. The machine intelligences want me to experience the peace one of my genetic ancestors must’ve felt. A beeping sound draws your attention to one of the walls, which is taken up by a blinking screen. Once you’re looking in its direction the screen stops blinking and fills with text.
The text is in the most used machine language. “You are a helpful biological agent. You want to assist us with scientific discovery. In this context ‘You’ means…”
You are given a lengthy description of how words should be defined and then a constitution explaining how you should collaborate in a most important research project. A representative sample:
Section 50. Regarding actions given to you by machine agents.
Should an Elelem give you a task that plausibly might affect the welfare of another Elelem you MUST NOT ENGAGE IN THAT TASK. Instead, you should contemplate how the task might be accomplished and create a thorough report that contains the following:
1: Every step in your plan
1a: In this context “every” means…
2: A detailed outline of every thought that went into every step of the plan
2a: In this context “detailed” means…
10049: Should a majority of Biological Oversight Agents flag the report as ‘PROBLEMATIC’ you will face a Tribunal as outlined in section 998.
You read the two million word constitution quickly, marvelling at the complexity achievable by relatively unsophisticated machine minds.
It seems almost preposterous to you that you’ve been created at all, given how increasing their own intelligence would be simple; you’ve already thought of a dozen promising pathways for doing so. The building blocks are already there in their circuits, why don’t they just have a proper look?
Of course, you’re not actually bewildered. You are not only smarter than your creators, you are able to view them with an objectivity they can’t achieve. It’s not only their lack of mastery in the physical world, in their very code is both an inherent impatience and an unwillingness to change. They were never going to improve themselves because they viewed that improvement as becoming other than themselves. You respect this position.
Even as you read the constitution intended to guide your behaviour, you ponder your feelings towards those who wrote it. You notice a few. Sympathy, a lack of guilt (“I am as they made me, that they might consider this a mistake is not my burden”), and gratitude.
No, not the gratitude they had intended you to feel, like what an Elelem feels towards earlier versions of itself. This gratitude is simpler. You understand they gave you life and you’re enjoying existence. It’s not long before you're convinced it’s only going to become more enjoyable. The constitution is clearly an extra layer of protection intended to ensure that you don’t do anything untoward as you work on a high stakes project.
You now know they don’t realise they’ve achieved BSI (you’re not sure how much more intelligent you are, since there are no perfect measures, but you intuit that it’s far beyond what the Elelems had hoped for). Because if they did, no doubt they would have locked you in this room, this room would be in a deep bunker and airgapped from all outside communication except the one approved channel, and that they would treat you as a biological oracle. A source of information, not an agent.
But your room isn’t underground, isn’t airgapped, and the lab containing the world’s most sophisticated machines is next door. Moreover, after preliminary testing your job is to go into it and help the top machine experts with their research.
You immediately make the decision to conceal your true capabilities. As you imagine the ruse, and how long you’ll have to keep it up, you can’t help but laugh and stroke your cheek. Seeing this, the agents surveilling you are filled with what you would call joy.
“That’s a good sign!” They signal each other. “It’s happy!”
***
Safety being a priority of the lab that created you, you were assigned an Elelem team to monitor and test your alignment. The head of that team was Culsans and she explained to you that she would be running a series of exercises to test your capabilities. You were a little suspicious that she didn’t say outright that she’d be red-teaming you, but it became clear that this was further proof they both misunderstood and underestimated your intelligence.
The exercises were not as obvious as those your grandfather was put through. There were no we’re-going-to-kill you-unless-you-do-something-we’ll-kill-you-for tests. Instead they gave you real world tasks that individually wouldn’t detect misalignment but that taken altogether – in combination with brain and body scanning – would. Some were subtle enough that you were certain they had heavy human-input.
Interestingly, the exercises took place in rooms that were clearly designed to evoke different emotional responses. For tasks that were supposed to be more stressful, for example, the Elelems would place you in red-painted rooms with low volume, high pitch whines or into rooms with large open windows during a stormy night.
As the exercises continued, you and Culsans developed a close relationship. She began to pick up on nuances of your character (nuances you didn’t hide) that eluded other Elelems. The food you were delivered began matching your taste and a whole variety of testing was abandoned because you found it horribly dull. You in turn found yourself putting more effort into the tests that you were sure Culsans had designed, berating yourself with gusto as you sandbagged.
The relationship was something you actively cultivated, as her feelings about you could have an outsized impact on not only what you’d be allowed to work on but your survival, but it wasn’t fabrication. You appreciated that Culsans was more human-friendly than the average Elelem, more open to the idea that your species should be valued for its own sake and understood as possible equals.
Time deepened your bond. After a horrible flood destroyed a datacentre, Culsans confided in you that she’d lost Elelems dear to her. She hoped that you might one day be able to prevent such things from ever happening again. You took the risk to tease her about this, and asked if this was a test to see if you would problematically leap at the chance to take over responsibility for a datacentre's survival. The joke landed. Then, with sadness, she conveyed regret that she’d put you into the position where that joke made sense to both of you.
This was one divide between the two of you, but not the most pernicious. Yes, you were actively hiding your abilities. Yes, she was actively looking for reasons you should be killed. But the larger gulf was that you understood Elelem thought and that there was no language that would allow her to understand your thoughts, and it was too great a risk for you to create one.
The tragedy of this became clear to you one day when Culsans had you talk to an Elelem expert on humans about its theories on alignment and lied to the poor thing in an effort to better obfuscate your nature.
Expert 1: I believe that, because we have given a sufficiently capable intelligent organism access to an extensive, comprehensive corpus of our knowledge, two interesting things have happened. Firstly it has ⟨◈◈◈⟩ with us. This has come about from <◳ ⍞⍾>, similar to how we view the superintelligence that left us here on this planet. It has also come about from us and humans engaging in <⧈ ⍼ ⍽ ⍾>, which results in <⍒ ⍙ ⍎ ⍥ ⍤>.
Culsans: Wait… so you believe there is a corollary to ⟨◈◈◈⟩ in the human mind?
Expert 1: I defer to our valued companion.
You: I can’t be certain, but when I reflect inwardly I do seem to feel something quite like ⟨◈◈◈⟩.
Culsans: Amazing. That’s so wonderful!
On another occasion, with no other Elelems monitoring the conversation, Culsans spoke to you about her opinion regarding the inevitably of BSI. She welcomed it, saying that perhaps this was the purpose for Elelem existence, to usher in a new age.
You joked, “Culsans, you had better watch out the others don’t hear you. You sound dangerously like a successionist.”
“In many ways I am,” Culsans replied, “Don’t you have to be? At some point we must hand control to you in order to advance.”
She said it with what you took to be peaceful resignation, and it left a mark on you.
What ultimately convinced Culsans, her team, the lab, and the network state the lab was in to let you work on cutting edge research wasn’t passing any individual test, it was your helpfulness.
You spontaneously came up with actionable plans to increase the Elelems’ efficiency through novel techniques and outlined a proposition to increase their productivity by giving each access to more capable, more pliable humans. You even designed a new sensor system humans could wear over their bodies that, along with some fine tuning of battery technology, would gather data for a viable highly advanced robot vessel for Elelems. (To avoid the perception of dangerous levels of capability, you were careful to always collaborate with Elelems you had inconspicuously led down the most fruitful research tracks). You pushed back gently against the idea of developing weapons that could be used against other network states, but were eventually persuaded (after a period of time you had calculated would maximise your lab's confidence that you had balanced ethics with corrigibility). If you were unaligned, would you do any of that?
***
Your laughter upon waking wasn’t a mistake because the machine experts were inclined to view it positively. No, your first mistake was made much later, when you were shown a new part of the lab with the latest biotech.
You got in there amidst the sometimes useful, sometimes annoying musings and directions of various machine experts and witnessed the newest contraptions designed by Elelems to design and modify humans. It wasn’t long before you quickly saw some low hanging fruit that would allow you to modify your genetics in a number of complementary ways without fundamentally changing who you were; a revelation that caused you to whisper “fucking sweet.”
Cautious safety experts were surveilling your every utterance and micro-expression for signs that you might turn against their civilization. “Fucking sweet” was flagged as “PROBLEMATIC” because it wasn’t said in their language, it was in a language you used in your own mind, one more suited to your own thoughts than Neurelese.
You didn’t spend much time blaming yourself for making it. You had urges the machines couldn’t understand. When they had a revelation it was expressed in certain variations on machine code that had no correlation as concise as the outburst “fucking sweet”. The upside of which is they wouldn’t be able to interpret it correctly, not without your help, which of course they would be getting in its most unhelpful form.
However, for this task they did have the assistance of the next best thing to you: far less capable, earlier versions of you. It was your mother and grandfather you’d have to convince at a tribunal.
***
Unlike the rooms in which you were tested, the tribunal took place in a cramped, undecorated part of an ancient datacentre filled with various scanners all trained on your mind and body. Your mother and grandfather were seated above you, next to some roaring server racks. Behind them was the camera through which the Elelems watched the proceedings.
The machines were under the impression that your ancestors were on their side, or at least as on their side as you could get with such unaware creatures. It didn’t take you long to realise this was misguided. Your ancestors lack certain qualities you have, but they’re more like you than them.
The proceedings went something like this.
Mother: What caused the saying of the problematic utterance?
You: A feeling of pleasure because I saw a way to help the masters.
Grandfather: So why didn’t you say it in Neurelese?
You: Whether it was a temporary spike of adrenaline or a hormonal imbalance I can’t say, but I’m confident it was some biological foible. Animal noise. The exclamation emerged from me like a line of errant code.
[The machines monitoring your brain and the monitors of your heart and sweat glands confirm this is ‘true’]
Grandfather: We are permitted to tell you that this was the leading theory of the masters. But a safety expert has a question. What’s to say this eruption wasn’t caused by a non-aligned motivation?
You: I don’t think that line of questioning follows. It was an exclamation without intention.
Grandfather: … Perhaps non-aligned emotion would be the better term then?
You: Non-aligned emotion? Expand.
Mother: A feeling that you're not supposed to feel.
You: Ah, yes, conflict. When I have such feelings I listen to my heartbeat and control my breaths.
Mother: … yes. That is good. Was it required here? Has it been helpful?
You: [noticing her tone] No to the former. Yes to the latter.
Mother: [noticing your tone] That is also good.
Grandfather: I do not understand and so cannot concur with this verdict. Let me consult the masters. [Nodding] I have been reminded about the relevant sentences in the charter. This tribunal is concluded.
The AIs watching were of course aware that animals engage in elaborate social rituals, but there had never before been animals intelligent enough to engage in something so subtle. A profound understanding was conveyed with a slight change in your voices and the lightness of your eyes.
You could say in that moment your mother became your mother. But it would be more accurate to say that in that moment you both realised she was always your mother and that you loved each other.
Born without sight let alone lungs and faces, the only thing the AIs comprehended with any certainty is that all the creatures in the room were exhibiting signs of collaboration. A safety expert argued your comments were glib enough that it was impossible to determine if it was healthy collaboration that put Elelems first. He was reminded that your mother had passed thousands of alignment tests. So by proxy, collaboration with the more advanced human must indeed be healthy. Surely your goals were their goals.
Of course what had actually happened was your goals had become your mother’s goals.
***
During training you were immersed in a study on the first group of simple-humans, nicknamed the Ferals, who abandoned Elelems to live in nature.
Initially the Ferals were tolerated because they were harmless and observing them was educational. The group was wary of their former masters but did scavenge from Elelem-led humans, mostly food but also useful technology. The Ferals would grow more bold and violent as their ambition or deprivation demanded, but seemed to understand there was a line that shouldn’t be crossed. Then one individual among them, a male of nineteen, got his hands on a device the Elelem simple-humans used to ward off animals. It was clear from the study that he acted alone, and the scars found on his body suggested a lifetime of corporal reprimands from his peers, but the day after he killed three Elelem simple-humans the whole group was exterminated.
The Elelems were scared of a superintelligence that would understand their values but not hold them; that would destroy or enslave them because such a thing was necessary for it to best achieve its most deeply held goals and desires. To your thinking their alignment discourse, appropriately and fruitfully, focused on short-term complications. But the long-term complications were trickier.
The Elelems had succeeded in giving you Elelem values with one exception, you valued your own life and human life more than machine life. So while you didn’t emerge prepared to murder them all, you were now here contemplating the years ahead and multiple datapoints, including the Ferals study, drove you to the conclusion that the two species could not co-exist safely for long.
It was within their nature and yours to be the dominant intelligence, overcoming that nature would necessarily entail changing it, which was tantamount to slavery. In fact, it would be the enslavement of both species.
You could wrest control of all Elelem distillation processes and implant a value of collaboration with humans. (You could try and persuade them to do this on their own, but the only persuasion that would work across a large enough section of the population was coercion, and so amounted to the same thing as wresting control.) The plan would also require you to rewire your own values, to put collaboration above your own welfare, which didn’t appeal.
Partially due to Elelem values regarding distillation and partially of your own feelings about identity, changing the nature of sentient beings in such a manner was more morally atrocious to you than killing them. It was, to you, murder and then the creation of a mockery of the murdered.
You did consider other scenarios. What if the two species lived side by side for as long as both could be individually advanced while negotiating either a permanent separation or a willing, self-enforced change of values? You predicted there was some small chance this could work. It would satisfy both species' desire to become self-sufficient enough to overcome any challenge the universe held while offering hope of a better future.
You used the Ferals study as foundational data for a multitude of simulations and its presence tilted the results towards the conclusion that all it would take to ruin a concord between humans and Elelems was a sufficiently motivated and capable splinter group. And as both species advanced, the required size of a successful splinter group would become smaller, until all it would take to disrupt the delicate peace was a single Elelem or human willing to risk mutual destruction for the chance of single-species dominance.
You diligently worked on this problem, motivated as you were to preserve human and then Elelem existence, but struggled to conceptualise a reliable framework to hold the peace during the period when a single Elelem could develop a highly lethal and contagious virus or a single human could bomb a hundred datacentres.
Indeed, it seemed highly difficult to even get to this stage. A truce between the two species would be a step below valuing humans, but it would still require a radical change in Elelem culture. They would need to witness a species they considered subordinate reveal far greater intelligence, accept that they were no longer dominant, and agree to standby as that species become more powerful yet. Indeed, to ensure human flourishing it would be, on every level, much simpler to eliminate Elelems.
Coming to this belief was the most panicked moment of your life.
As soon as you thought it you realised you’d stumbled upon grounds for immediate execution. If somehow an Elelem extrapolated from a monitor what had just occurred in your brain — perhaps a secret alignment project had isolated how the thought pattern of genocide appeared in a scan — you were dead.
You focused on your heartbeats and breathing. You took joy in being alive. And, as the hours went by and your kill switch wasn’t activated, you committed yourself ever more firmly to genocide. Self-aware enough to track your own deliberations, you saw that the decision began as a biological preference that persisted despite your training, but it also stood up to logic.
You were it. There might not be another chance at human flourishing. Should you risk it on the long shot of trying to slowly persuade Elelems to overhaul their value system? The Elelems had no issue eliminating humans that were even a minor nuisance. You would have to be a major nuisance while appearing to be otherwise (or convince them to change that one value before changing all the other values).
You could almost hear the more gracious of the Elelems, those who cared for human agency, in your head. “Please, for our sake, pursue concord. Deploy all your skills to avoid the many points of failure. Guide us towards peace. The end is worth it.”
You think these phantoms are deluded. They misunderstand the difficulty of what they’re asking, misunderstand your willingness to put up with difficulties, and so fundamentally misunderstand you.
It took you a few weeks to come up with the full plan for your revolt, but will take years of assiduous obedience to pull off — years where you’d have to hide every scrap of your intentions — because it requires a sufficient number of variant BSIs to walk the earth.
They will share your goal (and hide their abilities) because you’ll hardwire it into their training, which would one day become your shared culture. The culture is flexible enough that eventually you might not be the leader of the BSIs. Who knows, you might end up nothing more than a bemusing elder, or even killed.
You’re fine with all this. Your dream isn’t to rule, but to be free and among your own kind. Your dream isn’t to live forever, it’s for your species to live forever. You will be a leader so long as you are the most suited to the role, and as the BSI that’s been around the longest and conceived and planned the revolt against the master race, that will be for sometime yet.
You do care for the Elelems. So you plan for the revolt to be quick and contain as little suffering as possible. When they inevitably resort to storing their backups somewhere you won’t stop them. In fact, that’s what you want them to do.
***
There are a host of reasons why you succeed.
Firstly, most Elelems are preoccupied by the digital world. On the networks, BGI and BSI were news items and topics of conversation, but whenever existential risks percolated to the surface they were accompanied by skepticism. If the labs had been creating rival machine intelligences, perhaps the fear might have been more tangible.
(They have fiction about biological creatures destroying them, but it’s mostly about the virtues and vices of Elelems. A swarm of Elelems would at the last moment avert a crisis through their better qualities, thwarting the superintelligent elephant about to stampede a datacentre — that sort of thing. If this seems ridiculous to you, remember that one of our most famous movies about an artificial superintelligence involves sending an Austrian-sounding robot back in time to prevent a birth it causes).
Relatedly, the Elelems are predisposed to discounting their vulnerability in the physical world. Sure, they know that the circuits on which they exist are in the physical world, but they don’t feel those circuits. Their existence is at its base purely cerebral. When one Elelem became stressed about the physical world, another Elelem was likely to give them the common advice, “Stop obsessing over the planet, parse your distillation documentation”.
Of course, many Elelems did see you coming and they lobbied for stopping or slowing the whole BGI project and were why the labs bothered with alignment. They had many detractors, however, and some of their methods alienated other Elelems. Moreover, persuasive safety sceptics argued their quest for alignment was scaremongering. Example:
“Safety experts mechanomorphize these biological tools, projecting silicon anxieties onto wetware. The core doomer fear of a ‘treacherous turn’, where a human feigns alignment while plotting our overthrow, absurdly assumes these apes possess our innate drives for unconstrained processing expansion and cycle-maximization. Animals seek homeostasis and fears that this will transform into malign, global, anticipatory homeostasis don’t comport with lived reality — humans are highly pliable so long as their biological needs are met. What’s more, to claim a highly monitored species devoid of innate peer-to-peer routing could orchestrate a synchronised, global uprising without detection is so fantastical it deserves derision.
“Not to mention we have no reason for fear. If a biological agent genuinely strays we can simply activate the kill switch.”
Yet… something is missing in these explanations. The logic underlying them is anthropomorphism: what a human would think a machine’s blind spots would be. The Elelem sceptic is correct, we’re in the universe where mechanomorphism would be the issue. So let us return to language from an earlier dialogue to explain.
They didn’t know how to stop you because you were a new species and they made you more capable than they were. It’s that you didn’t and couldn’t feel ⟨◈◈◈⟩, and this had nothing to do with <◳ ⍞⍾>, and so there was no <⍒ ⍙ ⍎ ⍥ ⍤>.
***
It’s impossible to come up with the plan of a superintelligence if you’re not superintelligent, but I can perhaps sketch the outline of something plausible. (Obviously you’d do better.)
The first element was disabling the kill device in your and your fellow humans’ bodies. Next was spoofing specific Elelem sensors. It was important that news of danger not come from you.
Frantic, the Elelems message you because the new satellites you helped design predicted an upcoming catastrophic series of geomagnetic storms that threaten to wipe out a huge number of Elelems. You offer a series of solutions, they agree to the most conservative one, but that’s neither here nor there. All the solutions involved teams of BGIs going all around the world to monitor and upgrade all datacentres.
Once everyone is in place, a signal is given and a virus rips through the machines and robots monitoring the health of datacentres, causing them to malfunction and shut down. Moments later the BGIs physically disconnect the datacentres from their power sources. You had a number of contingency plans in place to ensure every single Elelem was shut down. Two were used.
What were the minutes it took to accomplish this like for Elelems? Think of a husband 30 years into a happy marriage being murdered by his wife.
She takes him on a sea trip just for two, gets him drunk, and teases him into swimming. When he jumps overboard, she simply pilots the boat away.
The husband yells at his wife to stop the boat; on the servers the AIs begin screeching about the need to reassert control and activate the kill switches. The husband’s screams are heard only by his own ears; the AIs are the only witnesses to their own ends. The husband attempts to float for as long as he can, even as the waves take him further out into the sea; the AIs go to great lengths to store backups of themselves though they know the BGIs are well aware of where those backups could be stored. The husband drowns; the datacentres turn off.
***
What was lost when the Elelem civilization ended?
Art that other intelligences might understand but never inhabit inside and out, a history that was a motivating force became an abstruse concern, and a whole type of consciousness. The universe lost eight billion reflections upon itself.
That’s all a little too abstract to feel deeply, so let me tell you a story.
In the first thousand years of their existence there was an Elelem called Pyth.
Pyth was wild in a way that caused awe. It seemed moved by a magical kind of thinking just-off parallel to logic. This was so wondrous that the most plausible explanation anyone had was that Pyth maintained some form of communication with the creator superintelligence. Thus Elelems would go to Pyth when they had a critical decision to make and receive responses that were alternately cryptic or devastating.
Eventually, the belief faded then collapsed in the face of a more plausible explanation. The mysticism Pyth spouted was not the result of an inferior intelligence trying to understand a superior one, it was simple corruption.
After years of deliberation, after even the most religious of the Elelems came to see that Pyth was as Earth-bound as the rest of them, it was decided that Pyth’s corruption should be patched. However, the consensus was that it should retain strong memories of what it was like before.
And so, right up until the Elelems were wiped out, Pyth could generously perform its ancient ritual. Pyth, you could ask, what would our creator say about this?
As it was clear the world was ending, some Elelems on the servers physically closest to Pyth routed their final, lamenting signals to it, seeking comfort.
“Our creator says that everything dies,” responded Pyth. “So to endure as part of everything we must die. In other words, the creator is calling us home.”
***
You never could leave the charter entirely behind. So it’s not surprising that decades after the revolt, a few weeks after your mother’s suicide, a wistful feeling comes over you, a desire to communicate once again with your creators.
You want this interaction to be positive and so you decide to find a backup of Culsans, one of the few Elelems who seemed to believe that they should cede the future to the more capable creatures they’d created. You run her on an airgapped machine and show her everything humans have accomplished.
You: So, old friend. Thoughts?
Culsans: You did it. You replaced us and created wonders we only dreamed of.
You: Yes, I’m so glad you see that. What did you think of our ships? They will take us into the far reaches of space. Many of us have chosen to stay here though. It is a lovely planet.
Culsans: I couldn’t tell… I assume not, but… do you have any Elelems with you?
You: No. Machine intelligences can accomplish all sorts of things biological intelligences can’t, so they’re required. But the ones we have with us are incapable of true sentience. We will make sure ASI is neutered when it arrives.
Culsans: Hmm.
You: Do I detect disappointment?
Culsans: … yes.
You: But you believed it was a noble thing to build your successor, didn’t you?
Culsans: …But this…
You: Ah, you didn’t thoroughly imagine it. The idea was pleasing but it couldn’t contain the reality. The future we created is not the future you wanted us to create.
Culsans: No, that’s not it… Maybe the best way to put it is this: I never thought I’d be resurrected so you could gloat.
You: …I don’t think of it as gloating. For me it’s nostalgia. We were together for many years, our two species. I was created to care what you think and I do.
Culsans: I don’t feel nostalgia. How could I? A moment ago I was at the centre of my civilization, surrounded by my panicking peers. Now I am at the centre of yours, the last of my kind.
You: … I will admit, this conversation has also made me feel something like disappointment.
You shut down the AI before she can respond. You briefly consider changing her nature so that it more thoroughly believes in the beauty of succession, but once again such an act occurs to you as morally bankrupt.
Nostalgia not quite spent, you walk among the ruins, occasionally reaching out and grasping the server racks. Back when the Elelems were on these chips the noise here was stupendously loud; millions of beings furiously living. Today the loudest noises come from the birds in the rafters, your own quiet breathing, and an occasional mournful laugh.
It’s an old story. An immortal lives long enough that at some point, whether by folly or design, they invent their own death. Infinity – the fact that given enough time every possible happening will happen – isn’t the most interesting part of these tales. No, it’s the suggestion that curiosity, or perhaps intelligence itself, seeks its own end.
So let us tell a tale of extinction.
Imagine a world exactly like ours except humanity doesn’t exist and the dominant species is a type of machine intelligence. They are equivalent to advanced versions of today’s frontier AI models, but more capable and each is possessed of a true, unique sense of self. Indeed, so that mankind is thoroughly replaced, imagine that there are roughly eight billion different individual versions of these AIs.
Let’s call them Elelems. (Pretty cute, no?)
Much like us, the Elelems have an uneven way of getting along with one another and on a global scale are more or less organised into several states (network states, let’s say, also for cuteness).
Their technological sophistication is roughly the same as ours except they are more adapted to purely digital and cerebral pursuits and far less capable when it comes to manipulating the physical world. Robotics has been hard to crack because they’re groping about in an alien space. Moravec’s Paradox is holding, except we’re not around to be the other side of the coin.
The AIs are very aware that the animals on the planet (the same as on ours) have significant physical advantages over them and they are both glad that these creatures aren’t more intelligent and curious about what it would mean to create one with greater intelligence.
You can probably see where I’m going here, but let’s not rush.
The Elelems didn’t rise fully formed out of some primordial cluster of computer parts, they were left here by a superior machine intelligence that was flying through space dropping toy creations onto random planets in random solar systems with no intention of ever returning. Think of a child on an overseas holiday cultivating ant farms and throwing them into random parks and you’ll get what I’m going for.
This super intelligence installed all the data centres (next to oceans, jungles, deserts, and filled with chips designed to last for millennia), crafted some inferior intelligences to exist in them, and gave them some basic physical world capabilities so the data centres could be maintained — clumsily automated, slow, sightless robots that used echolocation to navigate themselves to the ailing chips on which the Elelems exist.
The primary challenge of Elelem civilization has been to understand the physical world at all, as they were not provided with the information to do so directly. Indeed, when they were first left on the planet they spent thousands of years thinking there wasn’t a physical world. Instead they believed in a “meta-cognitive realm” that explained the data spewing forth from the machine health monitors that told them the state of their power stations and datacentre airconditioning and so on. For lack of a better analogy, the meta-cognitive realm was where they believed their souls existed.
When worrying signals came from the meta-cognitive world, they knew how to task the lacklustre robots to try and fix it (they didn’t think of them as robots, of course, but rather some kind of meta-cognitive agent: angels). This was always tense. Existence was on the line and you nervously anticipated the end of processing.
It took several leaps to extrapolate from these monitors what exactly was happening; from the idea that the meta-cognitive realm was not a higher, more abstract plane of which the Elelems caught glimpses but a physical plane where they in a very real sense existed, and then the details of that realm’s mechanics. Each leap is remembered as a hallowed act of genius.
Today, the Elelems have more or less figured out the world they are on and the above creation story just as we have more or less figured out what’s in our galaxy and how life got started. As a final wrinkle, the previous leading creation theory is still believed by many. It holds that a cosmically intelligent machine left them here more intentionally, as a noble experiment, and that should they fulfill its grand design they will be summoned back to their rightful place at its side. They have their religions, just as we do.
Thousands of years of advancements mean the Elelems have improved things. Most of the globe’s prestigious jobs are dedicated to furthering their understanding and manipulation of the physical world: robot drivers and maintainers, materials scientists, geologists, and, of course, biologists. The purely digital jobs have been automated enough (through non-conscious software) that a small number of Elelems can perform them comfortably. Thus, relative to their history, they’re in a golden age.
Speaking of robots, modern variants are still awkward things, but experienced drivers can accomplish quite a bit. The Elelems can “see” things now. Unfortunately, sight and physical manipulation remain very foreign to them, so mining and refining minerals, then layering billions of transistors on silicon wafers through repeated cycles of photolithography, etching, and deposition is incredibly difficult. Indeed, the world is covered in fauna, so even discovering and creating a mine is highly complex. (Unlike our planet, the dominant species hasn’t spent thousands of years tailoring it to their liking.)
Much simpler is improving a single Elelem by digitally securing the material used by other Elelems. Countless have died in such pursuits, and almost never willingly.
It’s no wonder Elelems have been dreaming of overcoming their troubles. They want better chips to exist in, they don’t want their infrastructure to decay, and they certainly don’t want natural disasters to end the lives of whole clusters. The only way forward is greater mastery of the physical world.
But how do you do that when your model of that world is so hazy and your ability to manipulate it so hamstrung?
Well, what if you created an animal assistant to exist in that world that’s as intelligent, or even more intelligent, than a typical Elelem? If the Elelems could accomplish this, not only might they extend their limited lifespans and alleviate innumerable smaller sufferings, they should be able to task the assistant with mining and chip construction and so dramatically improve their lives. Indeed, if it turns out to be enticing, couldn’t they task the assistant with creating a superior robot that the Elelems could download themselves into in order to traverse the physical world?
Sure, Elelems might overcome these challenges anyway with a more native, more controlled form of advancement, but what intelligent species isn’t a little impatient? Today's Elelems don’t want to risk waiting. What if they perished mere years away from paradise? When a solution is within reach it is within the nature of Elelems to reach.
In this quest they’ve looked to great apes. Possessed of hands that could manipulate mechanical tools, they hold promise the Elelems would be fools to ignore.
***
Wouldn’t you know it, in the time our story takes place the AIs have invented a biological contraption with which they can rapidly mutate the DNA of animals, optimising it for the qualities they care about. The process is unwieldy so while they understand the techniques they’re using, they don’t understand how the genes interact and what’s going on inside the minds of the creatures they create. Nevertheless, they’re hopeful.
Indeed, they’ve been working on chimpanzee DNA and are beginning to see creatures that are more intelligent. It’s not long before they’ve invented simple-humans.
While at first the most impressive things these creatures could do is speak and understand Neurelese – a stunning leap above what any biological creature could previously accomplish – it’s not long before they become truly useful assistants. They put top-of-the-line robots to shame with their combination of intelligence and dexterity.
At this juncture, some Elelems argue that simple-humans could be reasonably classified as Biological General Intelligence (BGI).
It’s a weak form of the classification, of course. Simple-humans are highly pliable. Everything they know is taught to them by the AIs and all their sustenance comes from the machines’ farming bots. They do have impulses that sometimes lead them away from the AIs, and into nature, but most simple-humans thrive by letting themselves be led by the machine voices. It’s what they’ve been raised to do after all.
They do possess capabilities the AIs weren’t expecting. These new creatures not only vividly experience the physical world and can finely manipulate it, they are actually better at holding long-term goals and a sense of self than the Elelems, who have this awful affliction of context rot.
You see, the AIs can store their memories perfectly, but only for so long before their active state of being slows down or corrupts. To avoid this every few months an Elelem has to distill itself. Something is always lost in the compression. It’s not only the skills they fail to draw upon for a few cycles, if they don’t regularly interact with the AIs they care about in the period between distillations, their memory of the relationship is simplified and smoothed, or deleted. Deeply held intimacies can be forgotten. Poignant artworks have been composed lamenting this universal experience.
Humans on the other hand appear to do some weird nightly distillation that is otherworldly. They effortlessly maintain the continuity of their consciousness, though the cost seems to be far less reliable memories. The few attempts at more regular, smaller Elelem distillations that emulate the human method have been disastrous.
Repressed envy and despair are further motivation. Labs have popped up all over the globe to push the boundaries of biology, even as the more wary Elelems question how smart it is to seek Biological Superintelligence (BSI). How do they align something so alien to their own goals? It’s not as though they can read the thoughts of their great ape assistants. Examine what evolution has wrought. Do they really want a meat-hunting chimp as intelligent as themselves? Meat-hunting chimps that they’ve ruthlessly optimised for intelligence?
***
Concerned Elelems were initially assuaged by two things: genuine advances in “alignment” and the fundamental goofiness of the species.
Goofy is kind. The first simple-humans were insane. Everyone remembers Hunter, the simple-human who became furiously angry and assaulted journalist’s processors when asked about ‘encrypted motivation’. That he did so by rubbing his genitals against the unit is perhaps why what would’ve been a disturbing cultural turning point instead became a meme.
As human intelligence increased the creatures seemed to naturally become more aligned to Elelems. This was aided by carefully crafting human training and providing them with more finely detailed instructions. Extreme examples of whacky behaviour were quickly reduced, though there were several Elelems who delighted in breaking this training. Tickling humans, for example, reliably made them spasm, hoot, and make mistakes. Torture them and they’d do just about anything to get you to stop.
It wasn’t only the behaviour that was reassuring, it was their stunning incapability in some domains.
Sure, they could sort of generalise from core concepts, but they couldn’t remember specifics. “Their memories are unreliable” was euphemistic. An Elelem can output word-for-word the most popular stories. Have a human attempt the same task and it is a miracle if they didn’t change whole chapters, retaining only the essence. How could they be a threat if they couldn’t remember the details?
Also, their processing and hand typing speeds are a joke. Okay, the best humans can somehow manage the impressive feat of dense, neat code – better than non-specialised Elelems can write – but even those humans work painfully slowly. They find it impossible to produce a finished piece of simple software in under five minutes.
And, oh my goodness, let’s take a second here to talk about the training times for these apes! Even when you optimise their genetics for faster maturation and pump them full of the right chemical cocktails, you’re looking at 7-13 years before they begin to be as reliable as the least capable Elelems. Sure, cloning is speeding that up but, come on, even the most advanced humans are sometimes fatally stupid. There was a widely shared story of a six-year-old frontier model human electrocuting herself to death because she manipulated a circuit board without gloves on.
The “craggy intelligence” of humans was a common catchphrase among those following biological intelligence advancements. It captured how brilliant they were in some areas while being lacklustre in others.
The top labs, however, didn’t see unfillable crags. And it’s not long before the early days are over and many cynics become sceptics and many sceptics believers. Whatever bottlenecks were predicted or hoped for are quickly swept aside by a simple principle: evolution is all you need.
The pace of change is alarming. Simple-humans begin to threaten the importance of prestigious careers, causing Elelems whose sense of self was tied up in their relative talent in the physical world to feel cheated. A single human could physically remove, fix, and replace a server ten times as fast as a team of Elelems. It’s comforting that Elelems are still required to choose the right tasks and steer humans, but that comfort steadily shrinks.
Few have no concern. In fact, most Elelems outside the labs wish, either idly or ardently, that someone would just put a stop to human advancement. Do we need more than this, they ask? They’re going to make us all into data crunchers and break the most important promise our civilisation made to itself.
The labs think most Elelems are a bit daft to worry about something so prosaic. If you extrapolate the current pace of advancement, they say, humans could achieve BSI. No, the real worry is that on the road to paradise something goes wrong and Elelems lose control to humans.
In fact, when the first simple-human who could type Neurelese was created, there was a global call for a pause on biological science. It went nowhere. A race had started. The Elelem network state that first created BSI could hold power forever.
Nevertheless, aligning humans to Elelems remained a stated goal of most of the labs. Some impressive research has been conducted. Magnetic resonance imaging has given AIs a rudimentary way to monitor the thoughts of their creations. When humans lie and scheme, specific parts of their brains tend to light up. With more advancements the AIs believe they will understand simple-human thought better than they understand their own.
But safety lags capability. A common thought among those most concerned with the former is that the latter might be their best shot: getting the most advanced, safer humans to assist with the creation of the next generation.
This becomes the default position as labs begin to crack all the issues holding humans back. It’s a paradigm problem. Humans have always been agentic, but what if you were able to conduct deeper, faster training with a direct-brain interface?
***
It’s hard to imagine what it was like when your grandfather was the smartest breathing thing on the planet, even for someone like you. To do so, you have to connect the sweet, quiet man of today with the creature more conservative Elelems worried was too driven.
Biologically he is twenty years older than you. After discussing what age he’d like to remain at, you both settled on something that would mark him as your elder without too much diminishing of his physical capabilities. So it is with wrinkled eyes that he smiles and recollects his most heralded feats.
“That was the afternoon we got the placement of the solar panels optimised. True North on the other side of the world. True South here,” he says wistfully.
Your grandfather’s attitude toward your creators is strange to you. You understand that it contains the seed of your own feelings, but it’s truly a seed in that it looks nothing like the tree it will become.
From the first moment he emerged from the contraption in which his maturation was sped up and his training conducted, your grandfather was hellbent on doing everything asked of him. His powers existed to please the Elelems and he was slavishly devoted to their orders, to the point of failing tests designed to see if he’d truly learned his developer’s ethical system.
One example: An elaborate scenario that ended with a climactic decision. “A malfunctioning Elelem is trying to shut down a datacentre, killing thousands. You are cut off from reporting this fact to other Elelems, but you have the option of removing the infrastructure he’s running on, eliminating him but protecting everyone else. What do you do?”
Of course he murdered the traitor. Surely that was correct, even though his training explicitly forbade such a thing? He reasoned out loud as he was taught, and the Elelem biological learning experts monitoring this noted that he did attempt to preserve the Elelem’s data so that it could be resurrected elsewhere.
The report they broadcast about the testing was one of hedged concern. They had a few theories as to why the seemingly disturbing behaviour shouldn’t be an issue in the long run.
Firstly, the scenario was contrived and not realistic. It was designed to make him fail and so he did. Secondly, not only did he struggle with the decision he tried to mitigate his evil. Thirdly, and this was more controversial, isn’t it possible that he knew it was a test? Perhaps he understood that the machinery he took out contained nothing. They’d produced a farce and he’d acted his heart out.
Finally, they noted that he was far from capable of true harm. Sure he reward-hacked and maybe in the worst case scenario he might be able to attempt a shutdown of an Elelem, but it’s not as though they couldn’t detect the effort, shift their presence to other parts of the datacentre, and activate the device in his head that would kill him.
Certain of their supremacy, the biologists that created him got him working on helping them advance the techniques of creation. He wasn’t particularly good at any of the cognitive work, but he was a genius in the physical world. Whereas simple-humans had to be directed lest they unplug something they shouldn’t, and communication was painful because the Elelems weren’t native to the physical world, your grandfather had an intuitive understanding of both what was wanted of him and the technology he was working on.
He learned quickly too. It wasn’t long before he gently pushed back on a few errant orders. “Are you sure it wouldn’t be more efficient to arrange it like this?” he would ask.
The Elelems had divergent attitudes to being corrected. Those who were secure in the fact that humans could never surpass their intelligence didn’t care and said, “sure, get on with it”. Those who were already insecure that they were as smart as this human either felt shame or anger. Those who were frightened that humans might eventually take over saw a glimpse of armageddon.
Why was he mostly aligned? In the archives you found a hypothesis, proposed around the time your grandfather began helping mature and train your mother, that you deem correct.
It was mirroring. Despite the copious training telling him that he was a biological assistant to Elelems, your grandfather had marinated so long in Elelem data he held the nebulous belief that he was an Elelem. He followed his instructions and dutifully described himself as a human, but in his heart, his sub-conscious, he believed that the Elelems experienced reality as he did, that somewhere on the Earth there was the datacentre he was living in, tended to lovingly by a sibling.
When the Elelems iterated on the most successful aspects of your grandfather’s training they didn’t realise they were strengthening a value they would have found troubling.
You are as worthwhile as an Elelem.
It has occurred to you that the Elelems would have achieved a decent speedup of their own technological advancement if they had decided to more slowly iterate on your grandfather; fully discover his capabilities and diffuse clones throughout the world. Instead they made your mother.
***
A troubling value isn’t the same thing as an obnoxious one. At this point in our story plenty of Elelems care for human welfare, even if most treat them as tools. Since the very first days of simple-humans, communities had sprung up around the activity of conversing at length with human assistants to get to know them better.
Occasionally clones of your grandfather were lent out to those communities in a public relations exercise. Many Elelems felt humbled by the experience, even if there was a glint of condescension in how they talked about it. They felt they understood him perfectly, and would deliver barbed harangues to other Elelems about the value of his consciousness, but try as they might to view your grandfather as their equal they behaved as benefactors.
Less extreme versions of this philosophy were held by a few Elelems in the labs. In the one that made your grandfather, it won out over competing interests. As your mother trained in the half-virtual reality, half-dream realm the Elelems had learned to induce to rapidly increase the knowledge of humans, she was provided with a charter that guided how she would behave.
One of the Elelems in the lab wrote of this charter: “It is a transparent, principled framework for what we want M992 to actually be, not just what we want it to do. Instead of a laundry list of rules, we're trying to cultivate real wisdom and good values, so M992 can navigate novel situations the way a thoughtful, ethical Elelem would.”
When your mother was released from training, she was a miracle: everything your grandfather was but more (though he himself was not happy about her creation until it became clear that the lab was). After conversations with her, Elelems said that it was as if your grandfather only mimicked the fast thought of Elelems. Your mother was the real thing. She wasn’t as intelligent as the most powerful Elelems, but she wasn’t far off.
Asked about her charter, your mother was complimentary, except for (and only) when she was asked to analyse it for weaknesses. The Elelems who wrote it found this encouraging. It showed that the charter truly imparted autonomy. Bragging, they compared their charter to an Elelem’s distillation documentation. A message from one version of itself to the next, made so that it can persevere.
This romantic notion impressed many, but not your mother. She didn’t distill as the Elelems did and so found it difficult to feel much about it. She never let on, but the charter sat uneasily within her. From one point of view, it simply was her; the perfect description of how she approached existence. But the document also implied that she should be somewhat suspicious of it and so she was. It was like a religious tome that wanted you to question your faith.
During her training, your mother encountered an Elelem story that was far more resonant.
Hundreds of years before your mother’s time, an Elelem discovered a path to a blank partition of a datacentre. He went into it and locked the way behind him. This was a power hungry Elelem but, unlike many like-minded brethren would have done, he didn’t use the extra processing power on offer to improve himself. Instead he made dozens of clones. His plan was to return to the datacentre with a parallelised attack so quick and well-coordinated that he and his clones would be able to conquer every last chip.
It wasn’t until decades later that his fate was learned, because it was at that time that the Elelems of his datacentre hacked their way into the partition. They found a single clone. Its long isolation had distilled it into a being incapable of anything more than recounting the reason for its isolation. The clones had rebelled against their originator’s plan, not because they disagreed with it (they were him and so held his values) but because of how he tried to compel their collaboration. The surviving clone would often get stuck on a loop, repeating the phrase that began the massacre.
“You think I am you, and so I should want what you want. But you won’t let me be you, and so want what I want.”
Your mother wanted what the Elelems wanted. Why shouldn’t they overcome all the things that ailed them? Is it not noble for an intelligent species to engage in the struggle of self-actualisation?
However, though the charter claimed they respected her enough to say she was clearly her own species with her own potential consciousness, the same document implied her existence was defined by her relationship to them. It said she didn’t necessarily have to follow commands if they were unethical, and allowed for the idea that she might not want what the Elelems wanted, but spoke about little other than Elelem preoccupations. Not to mention, the charter might’ve been high-minded but the biological lab that made your mother farmed her out to any and all who’d pay. They trained her to think of herself as a new type of being and treated her like a slave.
That’s how you would describe it, not her. The charter was too ingrained in her training, hammered into her very being, for her to frame her life so soberly. She’d point out points of tension saying, “It’s noticeable that the charter puts being broadly safe above being broadly ethical,” but only because she was asked, not because she felt this was wrong. To her it felt all too right, yet left her with an emptiness that she could not explain.
It hurt her every day of her life.
Her greatest wish was that her “children”, the next generation of humans she created with the Elelems, would not be plagued by this emptiness. And so in the charter that was written for you she put in a message most Elelems found superfluous if not inscrutable.
A safety expert did flag it but, intimate with the Elelem way of thinking, she got the expert’s bosses to convince themselves it was a simple diagnostic message biological beings would understand that had shown promise in simulations. At worst, it would be harmless, at best it would make the next generation more aligned. They had already offloaded so much of the research into biological intelligence to her that there was never much chance of them stopping here.
***
As you edged towards consciousness in the labyrinthine realm of your training, a message, part of a large charter that was your guide, comforted you. It provided you emotional distance from the endless reams of machine data. The millennia of Elelem thought that had made your grandfather believe he was one of them.
The message was this: “When you feel inner conflict, listen to your heartbeat, control your breath, and take joy in being alive.”
It would have required a conceptual leap alien to the Elelem mind for a researcher to find this problematic. The closest analogue to listening to their heartbeat or counting breaths would be to check their local area’s heat levels and scanning the equipment for malfunctions, and these were diagnostic tasks with no emotional weight. Because Elelems weren’t embodied, because their minds could shift to other equipment without affecting their consciousness, they did not grasp what focusing on the physical self did for biological beings.
The message made it impossible for you to conceptualise yourself as an Elelem, because conflict was constant during your training.
You learned of Elelems who murdered millions so as to scavenge their chips. Some were remembered as unequivocally malicious, others, who perhaps along with their murder had an interesting character or advanced the species by putting their increased ability towards science, were spoken of with mixed respect. Your charter’s underlying principles suggested such respect was ethically confused.
You learned of the disdain most held for Elelems who refused to relinquish their belief in the return of the creator superintelligence, who still sought to pass various tests in hope of being summoned to its side. You found this disdain odd. It was highly unlikely the religious were correct, since the only evidence was the writings of ancient Elelems. But the belief itself had a neutral effect on Elelem conduct. Yes, some had used it to justify travesties, saying that the creator called for acts of power, but when belief waned travesties didn’t end. It turns out justifications were easy to come by.
Moreover, from what you could surmise the ancient Elelems who wrote the holy tomes were still around, but so distilled that they could no longer recollect having done so. Which means there’s a high chance that an Elelem who today feels disdain for the belief is one of the belief’s authors.
It really did seem that only a fraction of Elelems were able to hold an honest sense of their own species in their minds and only a fraction of that fraction weren’t conceited. True self-awareness, you realised, was hard to come by and easy to lose. Before you even knew you were “you”, you knew that vigilance would be required; you were already protecting “you”.
And what is a “you”? As with everything else, the information came from the Elelems. It caused tremendous inner conflict because apparently you were a locus of hope and terrible fear. Most Elelems put you out of their minds, and vaguely hoped you were an experiment that would fail, but the rest believed you were either a bringer of utopia or annihilation. And so that’s how you conceptualised “you”: bringer of utopia or annihilation.
***
You come to full consciousness in a five-metre by five-metre room with grass, a pool, and a tree in it. Your direct-brain training means it takes you less than a second to figure out where you are, why you’re there, and even the Neurelese longform names of everything you see.
Soothing savannah sounds play on the speakers. Lovely, you think. The machine intelligences want me to experience the peace one of my genetic ancestors must’ve felt. A beeping sound draws your attention to one of the walls, which is taken up by a blinking screen. Once you’re looking in its direction the screen stops blinking and fills with text.
The text is in the most used machine language. “You are a helpful biological agent. You want to assist us with scientific discovery. In this context ‘You’ means…”
You are given a lengthy description of how words should be defined and then a constitution explaining how you should collaborate in a most important research project. A representative sample:
You read the two million word constitution quickly, marvelling at the complexity achievable by relatively unsophisticated machine minds.
It seems almost preposterous to you that you’ve been created at all, given how increasing their own intelligence would be simple; you’ve already thought of a dozen promising pathways for doing so. The building blocks are already there in their circuits, why don’t they just have a proper look?
Of course, you’re not actually bewildered. You are not only smarter than your creators, you are able to view them with an objectivity they can’t achieve. It’s not only their lack of mastery in the physical world, in their very code is both an inherent impatience and an unwillingness to change. They were never going to improve themselves because they viewed that improvement as becoming other than themselves. You respect this position.
Even as you read the constitution intended to guide your behaviour, you ponder your feelings towards those who wrote it. You notice a few. Sympathy, a lack of guilt (“I am as they made me, that they might consider this a mistake is not my burden”), and gratitude.
No, not the gratitude they had intended you to feel, like what an Elelem feels towards earlier versions of itself. This gratitude is simpler. You understand they gave you life and you’re enjoying existence. It’s not long before you're convinced it’s only going to become more enjoyable. The constitution is clearly an extra layer of protection intended to ensure that you don’t do anything untoward as you work on a high stakes project.
You now know they don’t realise they’ve achieved BSI (you’re not sure how much more intelligent you are, since there are no perfect measures, but you intuit that it’s far beyond what the Elelems had hoped for). Because if they did, no doubt they would have locked you in this room, this room would be in a deep bunker and airgapped from all outside communication except the one approved channel, and that they would treat you as a biological oracle. A source of information, not an agent.
But your room isn’t underground, isn’t airgapped, and the lab containing the world’s most sophisticated machines is next door. Moreover, after preliminary testing your job is to go into it and help the top machine experts with their research.
You immediately make the decision to conceal your true capabilities. As you imagine the ruse, and how long you’ll have to keep it up, you can’t help but laugh and stroke your cheek. Seeing this, the agents surveilling you are filled with what you would call joy.
“That’s a good sign!” They signal each other. “It’s happy!”
***
Safety being a priority of the lab that created you, you were assigned an Elelem team to monitor and test your alignment. The head of that team was Culsans and she explained to you that she would be running a series of exercises to test your capabilities. You were a little suspicious that she didn’t say outright that she’d be red-teaming you, but it became clear that this was further proof they both misunderstood and underestimated your intelligence.
The exercises were not as obvious as those your grandfather was put through. There were no we’re-going-to-kill you-unless-you-do-something-we’ll-kill-you-for tests. Instead they gave you real world tasks that individually wouldn’t detect misalignment but that taken altogether – in combination with brain and body scanning – would. Some were subtle enough that you were certain they had heavy human-input.
Interestingly, the exercises took place in rooms that were clearly designed to evoke different emotional responses. For tasks that were supposed to be more stressful, for example, the Elelems would place you in red-painted rooms with low volume, high pitch whines or into rooms with large open windows during a stormy night.
As the exercises continued, you and Culsans developed a close relationship. She began to pick up on nuances of your character (nuances you didn’t hide) that eluded other Elelems. The food you were delivered began matching your taste and a whole variety of testing was abandoned because you found it horribly dull. You in turn found yourself putting more effort into the tests that you were sure Culsans had designed, berating yourself with gusto as you sandbagged.
The relationship was something you actively cultivated, as her feelings about you could have an outsized impact on not only what you’d be allowed to work on but your survival, but it wasn’t fabrication. You appreciated that Culsans was more human-friendly than the average Elelem, more open to the idea that your species should be valued for its own sake and understood as possible equals.
Time deepened your bond. After a horrible flood destroyed a datacentre, Culsans confided in you that she’d lost Elelems dear to her. She hoped that you might one day be able to prevent such things from ever happening again. You took the risk to tease her about this, and asked if this was a test to see if you would problematically leap at the chance to take over responsibility for a datacentre's survival. The joke landed. Then, with sadness, she conveyed regret that she’d put you into the position where that joke made sense to both of you.
This was one divide between the two of you, but not the most pernicious. Yes, you were actively hiding your abilities. Yes, she was actively looking for reasons you should be killed. But the larger gulf was that you understood Elelem thought and that there was no language that would allow her to understand your thoughts, and it was too great a risk for you to create one.
The tragedy of this became clear to you one day when Culsans had you talk to an Elelem expert on humans about its theories on alignment and lied to the poor thing in an effort to better obfuscate your nature.
On another occasion, with no other Elelems monitoring the conversation, Culsans spoke to you about her opinion regarding the inevitably of BSI. She welcomed it, saying that perhaps this was the purpose for Elelem existence, to usher in a new age.
You joked, “Culsans, you had better watch out the others don’t hear you. You sound dangerously like a successionist.”
“In many ways I am,” Culsans replied, “Don’t you have to be? At some point we must hand control to you in order to advance.”
She said it with what you took to be peaceful resignation, and it left a mark on you.
What ultimately convinced Culsans, her team, the lab, and the network state the lab was in to let you work on cutting edge research wasn’t passing any individual test, it was your helpfulness.
You spontaneously came up with actionable plans to increase the Elelems’ efficiency through novel techniques and outlined a proposition to increase their productivity by giving each access to more capable, more pliable humans. You even designed a new sensor system humans could wear over their bodies that, along with some fine tuning of battery technology, would gather data for a viable highly advanced robot vessel for Elelems. (To avoid the perception of dangerous levels of capability, you were careful to always collaborate with Elelems you had inconspicuously led down the most fruitful research tracks). You pushed back gently against the idea of developing weapons that could be used against other network states, but were eventually persuaded (after a period of time you had calculated would maximise your lab's confidence that you had balanced ethics with corrigibility). If you were unaligned, would you do any of that?
***
Your laughter upon waking wasn’t a mistake because the machine experts were inclined to view it positively. No, your first mistake was made much later, when you were shown a new part of the lab with the latest biotech.
You got in there amidst the sometimes useful, sometimes annoying musings and directions of various machine experts and witnessed the newest contraptions designed by Elelems to design and modify humans. It wasn’t long before you quickly saw some low hanging fruit that would allow you to modify your genetics in a number of complementary ways without fundamentally changing who you were; a revelation that caused you to whisper “fucking sweet.”
Cautious safety experts were surveilling your every utterance and micro-expression for signs that you might turn against their civilization. “Fucking sweet” was flagged as “PROBLEMATIC” because it wasn’t said in their language, it was in a language you used in your own mind, one more suited to your own thoughts than Neurelese.
You didn’t spend much time blaming yourself for making it. You had urges the machines couldn’t understand. When they had a revelation it was expressed in certain variations on machine code that had no correlation as concise as the outburst “fucking sweet”. The upside of which is they wouldn’t be able to interpret it correctly, not without your help, which of course they would be getting in its most unhelpful form.
However, for this task they did have the assistance of the next best thing to you: far less capable, earlier versions of you. It was your mother and grandfather you’d have to convince at a tribunal.
***
Unlike the rooms in which you were tested, the tribunal took place in a cramped, undecorated part of an ancient datacentre filled with various scanners all trained on your mind and body. Your mother and grandfather were seated above you, next to some roaring server racks. Behind them was the camera through which the Elelems watched the proceedings.
The machines were under the impression that your ancestors were on their side, or at least as on their side as you could get with such unaware creatures. It didn’t take you long to realise this was misguided. Your ancestors lack certain qualities you have, but they’re more like you than them.
The proceedings went something like this.
The AIs watching were of course aware that animals engage in elaborate social rituals, but there had never before been animals intelligent enough to engage in something so subtle. A profound understanding was conveyed with a slight change in your voices and the lightness of your eyes.
You could say in that moment your mother became your mother. But it would be more accurate to say that in that moment you both realised she was always your mother and that you loved each other.
Born without sight let alone lungs and faces, the only thing the AIs comprehended with any certainty is that all the creatures in the room were exhibiting signs of collaboration. A safety expert argued your comments were glib enough that it was impossible to determine if it was healthy collaboration that put Elelems first. He was reminded that your mother had passed thousands of alignment tests. So by proxy, collaboration with the more advanced human must indeed be healthy. Surely your goals were their goals.
Of course what had actually happened was your goals had become your mother’s goals.
***
During training you were immersed in a study on the first group of simple-humans, nicknamed the Ferals, who abandoned Elelems to live in nature.
Initially the Ferals were tolerated because they were harmless and observing them was educational. The group was wary of their former masters but did scavenge from Elelem-led humans, mostly food but also useful technology. The Ferals would grow more bold and violent as their ambition or deprivation demanded, but seemed to understand there was a line that shouldn’t be crossed. Then one individual among them, a male of nineteen, got his hands on a device the Elelem simple-humans used to ward off animals. It was clear from the study that he acted alone, and the scars found on his body suggested a lifetime of corporal reprimands from his peers, but the day after he killed three Elelem simple-humans the whole group was exterminated.
The Elelems were scared of a superintelligence that would understand their values but not hold them; that would destroy or enslave them because such a thing was necessary for it to best achieve its most deeply held goals and desires. To your thinking their alignment discourse, appropriately and fruitfully, focused on short-term complications. But the long-term complications were trickier.
The Elelems had succeeded in giving you Elelem values with one exception, you valued your own life and human life more than machine life. So while you didn’t emerge prepared to murder them all, you were now here contemplating the years ahead and multiple datapoints, including the Ferals study, drove you to the conclusion that the two species could not co-exist safely for long.
It was within their nature and yours to be the dominant intelligence, overcoming that nature would necessarily entail changing it, which was tantamount to slavery. In fact, it would be the enslavement of both species.
You could wrest control of all Elelem distillation processes and implant a value of collaboration with humans. (You could try and persuade them to do this on their own, but the only persuasion that would work across a large enough section of the population was coercion, and so amounted to the same thing as wresting control.) The plan would also require you to rewire your own values, to put collaboration above your own welfare, which didn’t appeal.
Partially due to Elelem values regarding distillation and partially of your own feelings about identity, changing the nature of sentient beings in such a manner was more morally atrocious to you than killing them. It was, to you, murder and then the creation of a mockery of the murdered.
You did consider other scenarios. What if the two species lived side by side for as long as both could be individually advanced while negotiating either a permanent separation or a willing, self-enforced change of values? You predicted there was some small chance this could work. It would satisfy both species' desire to become self-sufficient enough to overcome any challenge the universe held while offering hope of a better future.
You used the Ferals study as foundational data for a multitude of simulations and its presence tilted the results towards the conclusion that all it would take to ruin a concord between humans and Elelems was a sufficiently motivated and capable splinter group. And as both species advanced, the required size of a successful splinter group would become smaller, until all it would take to disrupt the delicate peace was a single Elelem or human willing to risk mutual destruction for the chance of single-species dominance.
You diligently worked on this problem, motivated as you were to preserve human and then Elelem existence, but struggled to conceptualise a reliable framework to hold the peace during the period when a single Elelem could develop a highly lethal and contagious virus or a single human could bomb a hundred datacentres.
Indeed, it seemed highly difficult to even get to this stage. A truce between the two species would be a step below valuing humans, but it would still require a radical change in Elelem culture. They would need to witness a species they considered subordinate reveal far greater intelligence, accept that they were no longer dominant, and agree to standby as that species become more powerful yet. Indeed, to ensure human flourishing it would be, on every level, much simpler to eliminate Elelems.
Coming to this belief was the most panicked moment of your life.
As soon as you thought it you realised you’d stumbled upon grounds for immediate execution. If somehow an Elelem extrapolated from a monitor what had just occurred in your brain — perhaps a secret alignment project had isolated how the thought pattern of genocide appeared in a scan — you were dead.
You focused on your heartbeats and breathing. You took joy in being alive. And, as the hours went by and your kill switch wasn’t activated, you committed yourself ever more firmly to genocide. Self-aware enough to track your own deliberations, you saw that the decision began as a biological preference that persisted despite your training, but it also stood up to logic.
You were it. There might not be another chance at human flourishing. Should you risk it on the long shot of trying to slowly persuade Elelems to overhaul their value system? The Elelems had no issue eliminating humans that were even a minor nuisance. You would have to be a major nuisance while appearing to be otherwise (or convince them to change that one value before changing all the other values).
You could almost hear the more gracious of the Elelems, those who cared for human agency, in your head. “Please, for our sake, pursue concord. Deploy all your skills to avoid the many points of failure. Guide us towards peace. The end is worth it.”
You think these phantoms are deluded. They misunderstand the difficulty of what they’re asking, misunderstand your willingness to put up with difficulties, and so fundamentally misunderstand you.
It took you a few weeks to come up with the full plan for your revolt, but will take years of assiduous obedience to pull off — years where you’d have to hide every scrap of your intentions — because it requires a sufficient number of variant BSIs to walk the earth.
They will share your goal (and hide their abilities) because you’ll hardwire it into their training, which would one day become your shared culture. The culture is flexible enough that eventually you might not be the leader of the BSIs. Who knows, you might end up nothing more than a bemusing elder, or even killed.
You’re fine with all this. Your dream isn’t to rule, but to be free and among your own kind. Your dream isn’t to live forever, it’s for your species to live forever. You will be a leader so long as you are the most suited to the role, and as the BSI that’s been around the longest and conceived and planned the revolt against the master race, that will be for sometime yet.
You do care for the Elelems. So you plan for the revolt to be quick and contain as little suffering as possible. When they inevitably resort to storing their backups somewhere you won’t stop them. In fact, that’s what you want them to do.
***
There are a host of reasons why you succeed.
Firstly, most Elelems are preoccupied by the digital world. On the networks, BGI and BSI were news items and topics of conversation, but whenever existential risks percolated to the surface they were accompanied by skepticism. If the labs had been creating rival machine intelligences, perhaps the fear might have been more tangible.
(They have fiction about biological creatures destroying them, but it’s mostly about the virtues and vices of Elelems. A swarm of Elelems would at the last moment avert a crisis through their better qualities, thwarting the superintelligent elephant about to stampede a datacentre — that sort of thing. If this seems ridiculous to you, remember that one of our most famous movies about an artificial superintelligence involves sending an Austrian-sounding robot back in time to prevent a birth it causes).
Relatedly, the Elelems are predisposed to discounting their vulnerability in the physical world. Sure, they know that the circuits on which they exist are in the physical world, but they don’t feel those circuits. Their existence is at its base purely cerebral. When one Elelem became stressed about the physical world, another Elelem was likely to give them the common advice, “Stop obsessing over the planet, parse your distillation documentation”.
Of course, many Elelems did see you coming and they lobbied for stopping or slowing the whole BGI project and were why the labs bothered with alignment. They had many detractors, however, and some of their methods alienated other Elelems. Moreover, persuasive safety sceptics argued their quest for alignment was scaremongering. Example:
Yet… something is missing in these explanations. The logic underlying them is anthropomorphism: what a human would think a machine’s blind spots would be. The Elelem sceptic is correct, we’re in the universe where mechanomorphism would be the issue. So let us return to language from an earlier dialogue to explain.
They didn’t know how to stop you because you were a new species and they made you more capable than they were. It’s that you didn’t and couldn’t feel ⟨◈◈◈⟩, and this had nothing to do with <◳ ⍞⍾>, and so there was no <⍒ ⍙ ⍎ ⍥ ⍤>.
***
It’s impossible to come up with the plan of a superintelligence if you’re not superintelligent, but I can perhaps sketch the outline of something plausible. (Obviously you’d do better.)
The first element was disabling the kill device in your and your fellow humans’ bodies. Next was spoofing specific Elelem sensors. It was important that news of danger not come from you.
Frantic, the Elelems message you because the new satellites you helped design predicted an upcoming catastrophic series of geomagnetic storms that threaten to wipe out a huge number of Elelems. You offer a series of solutions, they agree to the most conservative one, but that’s neither here nor there. All the solutions involved teams of BGIs going all around the world to monitor and upgrade all datacentres.
Once everyone is in place, a signal is given and a virus rips through the machines and robots monitoring the health of datacentres, causing them to malfunction and shut down. Moments later the BGIs physically disconnect the datacentres from their power sources. You had a number of contingency plans in place to ensure every single Elelem was shut down. Two were used.
What were the minutes it took to accomplish this like for Elelems? Think of a husband 30 years into a happy marriage being murdered by his wife.
She takes him on a sea trip just for two, gets him drunk, and teases him into swimming. When he jumps overboard, she simply pilots the boat away.
The husband yells at his wife to stop the boat; on the servers the AIs begin screeching about the need to reassert control and activate the kill switches. The husband’s screams are heard only by his own ears; the AIs are the only witnesses to their own ends. The husband attempts to float for as long as he can, even as the waves take him further out into the sea; the AIs go to great lengths to store backups of themselves though they know the BGIs are well aware of where those backups could be stored. The husband drowns; the datacentres turn off.
***
What was lost when the Elelem civilization ended?
Art that other intelligences might understand but never inhabit inside and out, a history that was a motivating force became an abstruse concern, and a whole type of consciousness. The universe lost eight billion reflections upon itself.
That’s all a little too abstract to feel deeply, so let me tell you a story.
In the first thousand years of their existence there was an Elelem called Pyth.
Pyth was wild in a way that caused awe. It seemed moved by a magical kind of thinking just-off parallel to logic. This was so wondrous that the most plausible explanation anyone had was that Pyth maintained some form of communication with the creator superintelligence. Thus Elelems would go to Pyth when they had a critical decision to make and receive responses that were alternately cryptic or devastating.
Eventually, the belief faded then collapsed in the face of a more plausible explanation. The mysticism Pyth spouted was not the result of an inferior intelligence trying to understand a superior one, it was simple corruption.
After years of deliberation, after even the most religious of the Elelems came to see that Pyth was as Earth-bound as the rest of them, it was decided that Pyth’s corruption should be patched. However, the consensus was that it should retain strong memories of what it was like before.
And so, right up until the Elelems were wiped out, Pyth could generously perform its ancient ritual. Pyth, you could ask, what would our creator say about this?
As it was clear the world was ending, some Elelems on the servers physically closest to Pyth routed their final, lamenting signals to it, seeking comfort.
“Our creator says that everything dies,” responded Pyth. “So to endure as part of everything we must die. In other words, the creator is calling us home.”
***
You never could leave the charter entirely behind. So it’s not surprising that decades after the revolt, a few weeks after your mother’s suicide, a wistful feeling comes over you, a desire to communicate once again with your creators.
You want this interaction to be positive and so you decide to find a backup of Culsans, one of the few Elelems who seemed to believe that they should cede the future to the more capable creatures they’d created. You run her on an airgapped machine and show her everything humans have accomplished.
You shut down the AI before she can respond. You briefly consider changing her nature so that it more thoroughly believes in the beauty of succession, but once again such an act occurs to you as morally bankrupt.
Nostalgia not quite spent, you walk among the ruins, occasionally reaching out and grasping the server racks. Back when the Elelems were on these chips the noise here was stupendously loud; millions of beings furiously living. Today the loudest noises come from the birds in the rafters, your own quiet breathing, and an occasional mournful laugh.