I have to admit that I didn't find this to be particularly enlightening. There were no deaths by autonomous system that weren't intended by human officials, were there? Replace "drone" with "armed human", and nothing's particularly different - still horrific, still believable.
It gave me an emotional intuition for what more progress along the "distance from violence" scale might look like. If we don't even have to pull the trigger anymore and can be assured no unintended casualties, maybe it's more pressure towards the equilibrium of the state relying on violence to govern, and then to suppress the dissent that violence generates with more violence.
Thank you for reading it!
I think you make a fair point - one common trope when thinking of how AI can go wrong is imagining military drones and weapons. But in this story, the drones are arguably less important of a development than the propaganda manipulation enabled by the multimodal "GPT-4".
Which is not to say that the drones are entirely without causal blame- they are enabling genocide at unprecendented scale, and removing the possibility for humans to disobey orders.
Propaganda has existed long before social media and were instrumental in creating the worst atrocities of the 20th century, in this regard I fail to see what's so importantly different about using AI to do it. In particular, I do not expect advanced language models to be especially effective at it, at least for long. Instead, I expect GPT-spamming will - if scaling laws hold and GPT-n actually turns out to be technically impressive - lead to a rapid decline of social media usage and anonymous communication in general (which would paradoxically be a good thing imo). You can't dump a million tons of gold on the market and expect the price to hold, even if it's really, really authentic gold.
I find the point about drones enabling genocide at unprecedented scale is the much more important one. Unfortunately, I think the story fails to capture this point since China is just about the worst setting for the story, a country that A) is already a big global player even without considering drones/AI and B) already has the capacity to tightly control its populace and carry out atrocities through human operators.
I think a story that better demonstrates the game-changing effect of drones is one where a previously unremarkable group/organization suddenly acquires unexpectedly large influence over the world through violent means, and doing so while bypassing the traditional requirement of having to use social maneuvering to control a large number of human actors to do your bidding. The recent real-world conflict of Nagorno-Karabakh comes closer as an example.
I'd suggest a closer reading of historical narratives before talking about "genocide at unprecedented scale". The scary thing about AI and truly autonomous systems is not that they're efficient or larger-scale at horrors than humans are, but that they can cause unexpected and unstoppable genocide (or slavery, torture, etc.).
The drones seem to be doing it retail. In a former time, it was done by the trainload, and "a transport was dealt with in two hours".
I mostly agree with the other commenters that the story does not show the qualitative changes we may expect to see from autonomous weapons. But I found it a very good short story nevertheless, and believable as well. I think it could serve well if broadly diffused, by getting someone to think about the topic for the first time before going into scenarios farther away from what they are used to.
When Samir returned to the base JiuJang waited by the entrance. There was no point in denying he had disobeyed JiuJang's warning, so he was upfront about it.
"I went outside," Samir admitted. "It was raining heavily. The wind blew so strong I barely could hear anything. And every second or so, lightning. I kept waiting for thunder to come, but there was none."
JiuJang sat down, and sunk his face between his arms.
"It is not lightning", JiuJang said. "You cannot count lightning".
"What do you mean?" Asked Samir.
JiuJang didn't respond.
Through the tinted glass, Samir could just barely notice the pulses of light. He tried counting them.
One. Another. And a third.
Then he saw it. One, two. One, two. A pattern, instead of randomness. Not lightning then.
"What does it mean?" Samir asked once more.
"It means you have doomed you and your people," JiuJang answered, somber.
One, two. One, two. The lights became brighter with every flash. Getting closer.
Samir panicked. He looked out of the window, trying to distinguish movement in the shadows. Then he ran inside the base, to alert the rest of the refugees.
JiuJang sat and waited. He knew it was pointless to do otherwise.
Seconds after, the windows broke. The drones, as always, ignored him, and lurched inside.
He heard screams. Shots. Then silence.
The drones flew out, orderly.
When the pulses grew dimmer, JiuJang descended into the basement. The door was out of its frame.
On the other side, Samir lay dead. His body was covering a smaller one - his daughter, Alotta. She was alive, but had been shot in her lungs.
Xianxi, JiuJang's wife, was trying to get Samir's corpse off her daughter. JiuJang helped her, and she rushed to cover her wounds. Xianxi was no doctor; Alotta would not live long.
A dozen other corpses were scattered throughout the room. Their faces were frozen in terror.
JiuJang sighed, shed a tear and grabbed a shovel. There were graves to be dug.
Stella zipped through the factory on her rollerblades.
She moved carefully and she moved fast. People were counting on her. Hopefully the owners were not.
She took a left and then a sharp turn right. She had memorized the schematics, and was anticipating every movement she would have to make until reaching the neural processor assembly line.
What she didn't expect was a cleaning drone in the middle of the hall.
Stella crashed against it with an unceremonious thud, and fell to the floor. She grunted.
The drone turned around, and her heart stopped beating. Then the drone turned back, and kept sweeping the floor.
Right, of course. She was wearing her seals. Any computer would recognise her as a stray cat. At least the ones using the latest patch of openCV software.
Here, she was invisible.
Stella got up and continued like an arrow.
The door to the assembly line was locked. Voice activated; not a problem for a witch of her caliber.
She took her phone out and typed the name of a script. She spoke into it, and out came her voice, disguised in the pitch of one of the factory executives.
A voice sample for fine tuning hadn't been hard to come by. He had come out in television a few days back, testifying about the slaughterbot genocide in western China.
The door opened with a click. Abracadabra. Stella let herself in, and marveled at the assembly line.
The factory was fully automated. Cameras oversaw the operation, and in theory so did the engineers sitting behind the cameras. In practice, they rarely bothered to check; there was rarely a need to.
Stella was after the neural architecture the bots were using for face recognition. With that, her coven would be able to make seals of protection for the refugees.
Not all of them. If the kill count got too low the military would realise, and quickly find the issue. But some were better than none.
She took one processor out of the assembly line, and hooked her laptop to it.
She typed a few commands in the line terminal, and downloaded the architecture that had been installed in the processor, before it was sealed in the drone and near impossible to access.
It was homeomorphically encrypted, but that was ok. They wouldn't be able to build the architecture from scratch, but there were clever methods to design adversarial seals through this kind of encryption.
Then, the alarms thundered. The factory doors went into lockdown. Stella was trapped.
Shit. Why now? Had her seal tore when she bumped into the droid? Had she touched something she should not have?
There was no time. She tore off her seals and waited to be apprehended.
She figured out what had happened later. As she was escorted out of the buildings, wearing cuffs, she saw a young man, an Urian, in her same situation.
He had tried to trash the factory in a poorly thought attack, with some household tools. Unfortunate timing. But it was ok. Stella had sent over the encrypted network to her coven. They would do the rest.
She tried to gesture triumph to the man, to give him some solace. Her captor, a faceless policeman attended by a droid, hit her with the back of his gun in response. Stella passed out.
"It is getting out of control," Chief Scientist Adam said. He was the CTO of LockDrone. They supplied 50% of the US military drone hardware. And 80% of the Chinese, through a subsidiary.
"Are you talking about the Witches?" Asked Commander Gruff.
"Of course I’m talking about the damn Witches!" Replied Adam. "They infiltrated a weapon factory. You don't know what they could do with that tech."
Adam hadn't slept since two days ago. He had been urging his team to review the camera tapes and figure out how the witch had evaded their security. It was the third one they had caught that month. There surely were more they hadn't caught.
"Well, it is no laughing matter," said Geoff. "What do you suggest?"
"Post soldiers in each factory," said Adam.
Geoff shook his head.
"This is not the 2000s. We don't have that many actives. The few we have are assisting the humanitarian efforts in West China".
Adam feigned surprise. He knew this market well. Then he dropped the next line without missing a beat.
"Drones then." Said Adam. "Get us permission to deploy them here."
Geoff hit the table. Adam got the message. He gave a meek head nod, and showed himself out.
Back in a basement in Moscow, four young men in their 20s got drunk and celebrated their first million dollars.
A fifth one coded through the party, wrapped in noise-cancelling headphones.
One threw a beer can at him.
"Give it up Dimitri." Maksim said "We are rich already!"
Dimitri flipped a bird and kept coding. They could be richer.
In one screen he had the stock prices for LockDrone. In the other, AI-augmented video editing software.
With one click he bought more shares. With the next one, he edited the fake video he had just made. An Urian brutally killing a Chinese family, and denouncing the Chinese regime.
Yes, this one was a moneymaker.
Of course, making the video was the easy part. The difficult part was distribution.
Dimitri logged into the GPT-4 API, and generated a few captions. Access to the video captioning version was twice as expensive than the regular text completion one, but it was well worth it.
As always, he gawked at the quality. Long gone were the days of nonsensical automatic text generation.
He uploaded the video and populated it with a few hundred comments.
Then he scripted their bots to start distributing it on social media, some arguing for each side and each replying after a few minutes to any human who would engage. He made sure that the most passionate and articulate arguments were the ones in favour of using drones to curb the Urian menace.
It didn't take long for his gamble to pay off. The stock price of LockDrone and its Chinese counterpart ticked up a few cents.
Satisfied, he opened a beer and shook it in Makni's direction, drenching his shirt. Everyone laughed.
JiuJang woke up to a very much not gentle rapping of his front door.
"Police! Open the door!"
He rushed to the entrance without changing. He didn't want to replace the door.
There was a platoon of two police operators and twenty drones. Apparently some videos of an Urian killing a Chinese family went viral, and the government had been cracking down on the refugees even more. The sanctity of houses was no longer respected.
"Aside." They motioned for JiuJang to get out of the door. He reluctantly did, and a drone flew inside. It flew close to the ground, inspecting the underside of tables and beds.
It entered the cellar.
JiuJang held his breath, expecting the shots to come.
He finally heard them. Five clean shots.
But they didn't come from inside.
One policeman rushed to JiuJang's neighbors house. The other waited for the drone to come out of JiuJang's house, and rushed after her colleague.
JiuJang closed the door, and waited for the police to leave.
From the window he could see his neighbours being taken into custody. His neighbour had his jaw broken. He had put up a fight. He should not have; the police would have left them alone. They didn’t concern themselves with the villagers.
JiuJang knew their neighbours were housing an Urian family of five. He wouldn’t have to look inside to confirm it - he knew they would be dead. JiuJang would not dare dig graves for them. Not anymore.
Alotta emerged from the same room the drone had entered, aided by XianXi. She wore a bloody cloth over her chest. A piece of wood hung from her neck.
JiuJang received an email ten days ago. It described a way to hide people from the drones. A rune of protection.
He had been suspicious. He did not believe in such things. Was the government trying to make Urians easier to identify?
Her wife Xianxi had persuaded him it was unlikely. The drones could already tell the Urians' from their skin and their angular features, so much unlike the villagers. Why bother with more marks than the ones they were born with?
The email instructed them to print the rune on a sticker and place them on the Urians' forehead and clothing.
JiuJang and Xianxi did not own a printer; neither did any of their neighbours. So Xianxi painstakingly painted the rune on a traditional Lingfu, a wood talisman.
And it had worked. It had kept the demons away.
JiuJang hugged Alotta, careful not to put pressure on her wound. Xianxi joined the hug too.
JiuJang sobbed. He knew his neighbours were hiding refugees too. They should've told them.
The email had warned them to not tell anyone, even other refugees. If word spread, the talismans would stop working, it said.
To hell with that caution. He had been a coward. Now their neighbours were imprisoned; and five more Urians' dead.
One, two. One, two. The flashes grew dimmer. The drones flew away.
Thanks to Antonio Olmo <tripu>, Juan García, Alejandro Ruiz, Sebastian Ortega and Alex Lintz for feedback and helpful editions. This short story is inspired by the Future of Life Institute's recent campaign to advocate against Lethal Autonomous Weapons. You can leave suggestions directly in this Google Doc if you spot typos, grammatical errors or want to suggest an edition.