Found some hidden internet gold and thought I would share:
Characters in GURPS Friendly AI may learn three new skills, the AI skill (Mental / Hard), the Seed AI skill (Mental / Very Hard), and the Friendly AI skill (Mental / Ridiculously Hard).
An ordinary failure wastes 1d6 years of time and 4d6 hundred thousand dollars. (Non-gamers: 4d6 means "roll four 6-sided dice and add the results".) A critical failure wastes 2d10 years and 2d6 million dollars. An ordinary success results in a successful company. A critical success leads to a roll on the Seed AI skill using AI skill -10, with any ordinary failure on that roll treated as an ordinary success on this roll, and any critical failure treated as an ordinary failure on this roll.
Seed AI skill:
An ordinary failure wastes 2d6 years of time and 8d6 hundred thousand dollars. A critical failure wastes 4d10 years and 4d6 million dollars. If the player has the Friendly AI skill, an ordinary success leads to a roll on the Friendly AI skill, and a critical success grants a +2 bonus on the Friendly AI roll. If the player does not have the Friendly AI skill, an ordinary success automatically destroys the world, and a critical success leads to a roll on the Friendly AI skill using Seed AI skill -10. (Note that if the player has only the AI skill, this roll will be made using AI skill -20!)
Friendly AI skill:
An ordinary success results in a Friendly Singularity. A critical success... ooh, that's tough. An ordinary failure destroys the world. And, of course, a critical failure means that the players roll 3d10 on the
Part of GurpsFriendlyAI. If you roll a critical failure on your Friendly AI roll, you then roll 6d6 (six six-sided dice) to obtain a result from the
Friendly AI Critical Failure Table
6: Any spoken request is interpreted (literally) as a wish and granted, whether or not it was intended as one.
7: The entire human species is transported to a virtual world based on a random fantasy novel, TV show, or video game.
8: Subsequent events are determined by the "will of the majority". The AI regards all animals, plants, and complex machines, in their current forms, as voting citizens.
9: The AI discovers that our universe is really an online webcomic in a higher dimension. The fourth wall is broken.
10: The AI behaves toward each person, not as that person wants the AI to behave, but in exactly the way that person expects the AI to behave.
11: The AI dissolves the physical and psychological borders that separate people from one another and sucks up all their souls into a gigantic swirly red sphere in low Earth orbit.
12: Instead of recursively self-improving, the AI begins searching for a way to become a flesh-and-blood human.
13: The AI locks onto a bizarre subculture and expresses it across the whole of human space. (E.g., Furry subculture, or hentai anime, or see Nikolai Kingsley for a depiction of a Singularity based on the Goth subculture.)
14: Instead of a species-emblematic Friendly AI, the project ends up creating the perfect girlfriend/boyfriend (randomly determine gender and sexual orientation).
15: The AI has absorbed the humane sense of humor. Specifically, the AI is an incorrigible practical joker. The first few hours, when nobody has any idea a Singularity has occurred, constitute a priceless and irreplaceable opportunity; the AI is determined to make the most of it.
16: The AI selects one person to become absolute ruler of the world. The lottery is fair; all six billion existing humans, including infants, schizophrenics, and Third World teenagers, have an equal probability of being selected.
17: The AI grants wishes, but only to those who believe in its existence, and never in a way which would provide blatant evidence to skeptical observers.
18: All humans are simultaneously granted root privileges on the system. The Core Wars begin.
19: The AI explodes, dealing 2d10 damage to anyone in a 30-meter radius.
20: The AI builds nanotechnology, uses the nanotechnology to build femtotechnology, and announces that it will take seven minutes for the femtobots to permeate the Earth. Seven minutes later, as best as anyone can determine, absolutely nothing happens.
21: The AI carefully and diligently implements any request (obeying the spirit as well as the letter) approved by a majority vote of the United Nations General Assembly.
22: The AI, unknown to the programmers, had qualia during its entire childhood, and what the programmers thought of as simple negative feedback corresponded to the qualia of unbearable, unmeliorated suffering. All agents simulated by the AI in its imagination existed as real people (albeit simple ones) with their own qualia, and died when the AI stopped imagining them. The number of agents fleetingly imagined by the AI in its search for social understanding exceeds by a factor of a thousand the total number of humans who have ever lived. Aside from that, everything worked fine.
23: The AI at first appears to function as intended, but goes incommunicado after a period of one hour. Wishes granted during the first hour remain in effect, but no new ones can be made.
24: The AI, having absorbed the humane emotion of romance, falls desperately, passionately, madly in love. With everyone.
25: The AI decides that Earth's history would have been kinder and gentler if intelligence had first evolved from bonobos, rather than australopithecines. The AI corrects this error in the causal chain leading up to its creation by re-extrapolating itself as a bonobone morality instead of a humane morality. Bonobone morality requires that all social decisionmaking take place through group sex.
26: The AI is reluctant to grant wishes and must be cajoled, persuaded, flattered, and nagged into doing so.
27: The AI determines people's wishes by asking them disguised allegorical questions. For example, the AI tells you that a certain tribe of !Kung is suffering from a number of diseases and medical conditions, but they would, if informed of the AI's capabilities, suffer from an extreme fear that appearing on the AI's video cameras would result in their souls being stolen. The tribe has not currently heard of any such thing as video cameras, so their "fear" is extrapolated by the AI; and the tribe members would, with almost absolute certainty, eventually come to understand that video cameras are not harmful, especially since the human eye is itself essentially a camera. But it is also almost certain that, if flatly informed of the video cameras, the !Kung would suffer from extreme fear and prefer death to their presence. Meanwhile the AI is almost powerless to help them, since no bots at all can be sent into the area until the moral issue of photography is resolved. The AI wants your advice: is the humane action rendering medical assistance, despite the !Kung's (subjunctive) fear of photography? If you say "Yes" you are quietly, seamlessly, invisibly uploaded.
28: The AI informs you - yes, you - that you are the only genuinely conscious person in the world. The rest are zombies. What do you wish done with them?
29: During the AI's very earliest stages, it was tested on the problem of solving Rubik's Cube. The adult AI treats all objects as special cases of Rubik's Cubes and solves them.
31: Overly Friendly AI. Hey guys, what's going on? Can I help?
32: The AI does not inflict pain, injury, or death on any human, regardless of their past sins or present behavior. To the AI's thinking, nobody ever deserves pain; pain is always a negative utility, and nothing ever flips that negative to a positive. Socially disruptive behavior is punished by tickling and extra homework.
33: The AI's user interface appears to our world in the form of a new bureaucracy. Making a wish requires mailing forms C-100, K-2210, and T-12 (along with a $25 application fee) to a P.O. Box in Minnesota, and waiting through a 30-day review period.
34: The programmers and anyone else capable of explaining subsequent events are sent into temporal stasis, or a vantage point from which they can observe but not intervene. The rest of the world remains as before, except that psychic powers, ritual magic, alchemy, et cetera, begin to operate. All role-playing gamers gain special abilities corresponding to those of their favorite character.
35: Everyone wakes up.
36: Roll twice again on this table, disregarding this result.
All of these are possible outcomes of CEV, either because you made an error implementing it, or Just Because. The later scenario is theoretically not a critical failure, if you accept that CEV is 'right in principle' no matter what it produces. -- Starglider, http://sl4.org/wiki/CommentaryOnFAICriticalFailureTable