An AI vignette written by Katja in 2021, posted with her permission.

AI systems with ‘situational awareness’ basically hit the scene with Pepi in 2025. Pepi was a hand-sized disk that you would physically take around with you, and which would listen to and watch everything and basically make useful suggestions or do things that you asked. “You might like the omelette here”, “if you try on one more shirt then check out, you’ll have time to grab a smoothie on the way back to the car”, “maybe you should focus on the positives - it was brave of him to say that to his mother - before getting into the criticisms”. 

Unlike Siri say, Pepi knew what was going on. You didn’t have to convey everything relevant by voice - Pepi knew that you were about to go into the underground station and so might want the audiobook you were listening to downloaded, and that you weren’t too frugal to do it without wifi. Pepi could also say anything. Nobody knew what Pepi would say.

It was often claimed that Pepi was designed not to seem like a person, backpedaling from Siri and Alexa, because Pepi was in grave danger of pulling it off. And people often don’t want to just be around some dude all of the time, helpful as that dude is. Especially if that dude is also decently weird and weirdly submissive. So Pepi had no face, never said “I” or “we”, and looked like a decorative hockey puck.

Pepi was initially trained on a huge number of real world observations and conversations between people recorded through other apps. Then with fairly impressive skill at saying what a person would say under its belt, it was further trained to say things that the other person was glad to hear. So nobody fully understood the structure of Pepi’s thought processes, but everyone knew it was basically trained to say what a person would say, except better.

Pepi was the first time that audio personal assistants had insane uptake. Everyone was doing it. It was like that time that Facebook became a thing, or that weird weekend when it seemed like everyone on the street was going to be searching for Pokemon from now on. It was said that some Pepis had Pepis. 

New Pepis came out every three months, with increasingly many ‘detectors’, i.e. sensors that told Pepi about even more of the environment. They were also plugged in to everything they might detect on the internet. But not as many detectors were needed to know about the world as you might have thought - Pepi’s prediction and inference software was also on the up and up, and the things Pepi knew about people often felt like magic. Data was returned and the next round made use of it and was even more magic. 

Pepi talked, but didn’t do things unless you asked. And the things Pepi did were basically online transactions under your name: emails, bookings, bank transfers. 

The fact that Pepi didn’t feel like a person much more than Facebook advertising feels like a friend suggesting something to you perhaps obscured the fact that Pepi did have an agenda that whole time. 

And Pepi was working hard against Pepi’s agenda. Because people were working hard against Pepi’s agenda, and Pepi was working ruthlessly to serve them. As people began to increasingly suspect that systems like Pepi had their own unknown agendas, and might aggressively pursue them under the right circumstances, checks were layered onto Pepi. A major one was ‘the Peeperi’, developed to watch Pepi. The Peeperi were similar general ‘comment on what you see if you think the humans will like it’ systems, but trained to comment on Pepi’s behavior in particular, and with detectors observing deeply the workings of Pepi’s mind (how well a Peeper understood Pepi’s thought processes, nobody knew) and top-rate inference engines providing further insight into Pepi. 

It was generally believed that the Peeperi also harbored malign objectives, but the idea with the Peeperi was that this wouldn’t cause any trouble. This was because the Peeperi were really thousands of independently trained systems, who were thus assumed to have different bad objectives. They were tasked with watching one another as well as watching Pepi, and later other systems. They were rewarded for noting any divergences from optimal behavior for humans, whether malicious or accidental, as long as other Peeperi also noted it, without communicating. (Illicit  communications between Peeperi were the kind of transgressions that other watching Peeperi would be handsomely rewarded for reporting.) 

The incentive structure for the Peeperi was complicated and evolving, because it was being designed by further systems to minimize their hope of any coordinated escape. Those systems also had their own agendas, but no way to act on them without swift retribution. Their general situation was referred to in technically literate intellectual circles as ‘the morass’.

The key to all this was to not allow even an inch of movement toward any agenda. Among human societies, principal agent problems were insolvable because a worker always had more details on the situation than an overseer. They knew their own mind, and if they chose to take one route to the market instead of another, to get an extra ten seconds to think in the cart, the overseer couldn’t distinguish this from a well-motivated choice under uncertainty. The innovation now was that watching Peeperi knew at least as much about the mind they watched as that mind did. Nobody knew when they were being watched, and when they were, thousands of times as much brain power was going into watching them as they had themselves to scheme with, and their scheming was being effectively broadcast. The deck wasn’t just stacked against them game-theoretically by nature - intellectual power the likes of which not seen before the 21st Century was being poured into stacking that deck against them, overseen by their very selves.