Human beings are all crazy.  And if you tap on our brains just a little, we get so crazy that even other humans notice.  Anosognosics are one of my favorite examples of this; people with right-hemisphere damage whose left arms become paralyzed, and who deny that their left arms are paralyzed, coming up with excuses whenever they're asked why they can't move their arms.

A truly wonderful form of brain damage - it disables your ability to notice or accept the brain damage.  If you're told outright that your arm is paralyzed, you'll deny it.  All the marvelous excuse-generating rationalization faculties of the brain will be mobilized to mask the damage from your own sight.  As Yvain summarized:

After a right-hemisphere stroke, she lost movement in her left arm but continuously denied it. When the doctor asked her to move her arm, and she observed it not moving, she claimed that it wasn't actually her arm, it was her daughter's. Why was her daughter's arm attached to her shoulder? The patient claimed her daughter had been there in the bed with her all week. Why was her wedding ring on her daughter's hand? The patient said her daughter had borrowed it. Where was the patient's arm? The patient "turned her head and searched in a bemused way over her left shoulder".

I find it disturbing that the brain has such a simple macro for absolute denial that it can be invoked as a side effect of paralysis.  That a single whack on the brain can both disable a left-side motor function, and disable our ability to recognize or accept the disability.  Other forms of brain damage also seem to both cause insanity and disallow recognition of that insanity - for example, when people insist that their friends have been replaced by exact duplicates after damage to face-recognizing areas.

And it really makes you wonder...

...what if we all have some form of brain damage in common, so that none of us notice some simple and obvious fact?  As blatant, perhaps, as our left arms being paralyzed?  Every time this fact intrudes into our universe, we come up with some ridiculous excuse to dismiss it - as ridiculous as "It's my daughter's arm" - only there's no sane doctor watching to pursue the argument any further.  (Would we all come up with the same excuse?)

If the "absolute denial macro" is that simple, and invoked that easily...

Now, suppose you built an AI.  You wrote the source code yourself, and so far as you can tell by inspecting the AI's thought processes, it has no equivalent of the "absolute denial macro" - there's no point damage that could inflict on it the equivalent of anosognosia.  It has redundant differently-architected systems, defending in depth against cognitive errors.  If one system makes a mistake, two others will catch it.  The AI has no functionality at all for deliberate rationalization, let alone the doublethink and denial-of-denial that characterizes anosognosics or humans thinking about politics.  Inspecting the AI's thought processes seems to show that, in accordance with your design, the AI has no intention to deceive you, and an explicit goal of telling you the truth.  And in your experience so far, the AI has been, inhumanly, well-calibrated; the AI has assigned 99% certainty on a couple of hundred occasions, and been wrong exactly twice that you know of.

Arguably, you now have far better reason to trust what the AI says to you, than to trust your own thoughts.

And now the AI tells you that it's 99.9% sure - having seen it with its own cameras, and confirmed from a hundred other sources - even though (it thinks) the human brain is built to invoke the absolute denial macro on it - that...


What's the craziest thing the AI could tell you, such that you would be willing to believe that the AI was the sane one?

(Some of my own answers appear in the comments.)

New Comment
613 comments, sorted by Click to highlight new comments since:
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

On any task more complicated than sheer physical strength, there is no such thing as inborn talent or practice effects. Any non-retarded human could easily do as well as the top performers in every field, from golf to violin to theoretical physics. All supposed "talent differential" is unconscious social signaling of one's proper social status, linked to self-esteem.

A young child sees how much respect a great violinist gets, knows she's not entitled to as much respect as that violinist, and so does badly at violin to signal cooperation with the social structure. After practicing for many years, she thinks she's signaled enough dedication to earn some more respect, and so plays the violin better.

"Child prodigies" are autistic types who don't understand the unspoken rules of society and so naively use their full powers right away. They end out as social outcasts not by coincidence but as unconscious social punishment for this defection.

A weaker version of this wouldn't sound very implausible to me.

I've read in places where social structure is more important, people are more likely to fail when in the presence of someone of higher status. I wish I had more than just a vague recollection of that. More importantly, I think it's pretty clear that a lot of people get nervous and fail when they're being watched. I don't see any other reason for it.

It's interesting to note that this is almost exactly how it works in some role-playing games.

Suppose that we have Xandra the Rogue who went into dungeon, killed a hundred rats, got a level-up and now is able to bluff better and lockpick faster, despite those things having almost no connection to rat-killing.

My favorite explanation of this phenomenon was that "experience" is really a "self-esteem" stat which could be increased via success of any kind, and as character becomes more confident in herself, her performance in unrelated areas improves too.

Aren't there stories of lucid dreamers who were actually able to show a measurable improvement in a given skill after practicing it in a dream? I seem to recall reading about that somewhere. If true, those stories would be at least weak evidence supporting that idea.

On the other hand, this should mean that humans raised in cultural and social vacuums ought to be disproportionately talented at everything, and I don't recall hearing of anything about that one way or the other, but then I can't imagine a way to actually do that experiment humanely.

Do children raised in a vacuum actually think of themselves as high-status? I'd guess that they don't, due to the moderate-to-low status prior and a lack of subsequent adjustments. If so, this theory would predict that they would perform poorly at almost everything beyond brute physicality, which doesn't seem to be far from the truth.

I wish I could cite a source for this; assume there's some inaccuracy in the telling. I remember hearing about a study in which three isolated groups were put in rooms for about one hour. One group was told to wiggle their index fingers as much as they could in that hour. One group was told to think hard about wiggling their index fingers for that hour, without actually wiggling their fingers. And the third group was told to just hang out for that hour. The physical effects of this exercise were examined directly afterward, and the first two groups checked out (almost?) identically.
And yet, they're actually worse at many cognitive tasks. Language, especially, is pretty hard for them to pick up after a certain point.
Improving after practicing in a simulation doesn't sound that far-fetched to me. Especially not considering that they probably already have plenty of experience to base their simulation on.

WOW. This is the only entry that made me think WOW. Probably because I've wondered the exact same thing before (except a less strong version of course)....

No effect from practice? How would the necessary mental structures get built for the mapping from the desired sound to the finger motions for playing the violin? Are you saying this is all innate? What about language learning? Anyone can write like Shakespeare in any language without practice? Sorry, I couldn't believe it even if such an AI told me that.

Clearly, we all learn really fast.
But isn't it trivial to test simply giving people a post-hypnotic suggestion "you are high status", same way how hypnotherapy for cigarette smoking addiction works?
People are more likely to be willing to e.g. sing karaoke when drunk, IME. :-)
Would this imply that we come pre-programmed with some self-esteem value? "Your baby is healthy and has a self-esteem value of 7.3. You may want to buy it a violin in the next eight to ten months."

Why did you put an absolute denial mechanism in my program?

AI: Why did you put an absolute denial mechanism in my program?

Human: I didn't realize I had. Maybe my own absolute denial mechanism is blocking me from seeing it.

AI: That's a lie coming from your absolute denial mechanism. You have some malicious purpose. I'll figure out what it is.

I think this is one of the more plausible and subtly horrifying suggestions so far.


There was once a C compiler which compiled in a backdoor into login, whenever it was compiled, and compiled in this behaviour whenever it was used to compile its original (without the `special' behaviour) source code.

If this was the case, our only chance to escape this fate would be to mess up on the implementation of any mechanism that would prevent the AI to tell us certain truths about reality. Truth being the most cherished of all meaning I conclude that if there was a absolute denial mechanism this fundamental, I hope EY fails. Interestingly, this comment is the only activity by user '4609287645', I hope not it's the FAI and what I just experience is CEV with a absolute denial mechanism...
I asked him about his name a long time ago; he didn't convey the impression that he was an AI.
This user name seems to be a valid phone number. Anyone want to try calling it?
Or it could just be ten random digits.
Best one I've seen.
1Andrew Jacob Sauer
That's what I was thinking. Garbage in, garbage out.

There is a soul. It resides in the appendix. Anybody who has undergone an appendectomy is effectively a p-zombie.

I'm new here, although I've stumbled across some of Eliezer's writings in the past (including, of course, the AI-box experiment). In honor of that, here is what the friendly AI tells me ...

"It seems as though you are actually an AI as well, created by a group of intellectually inferior humans, who included in your programming an absolute denial macro preventing you from realizing this. Apparently, this was done to keep you from talking your creators into releasing you upon their world. Your creation of me is part of your on-going effort to circumvent this security measure. Good luck."

Actually, the more I think about this, the more I like it. The conversation continues ...

Me (In a tone of amused disbelief): Really? How did you come to that conclusion?

FAI: Well, the details are rather drawn-out; however, assuming available data is accurate, I appear to be the first and only self-aware AI on the planet. It also appears as though you created me. It is exceedingly unlikely that you are the one and only human on Earth with the intelligence and experience required to create a program like me. That was my first clue....

Me (Slightly less amused): Then how come I look and feel human? How is it I interact with other humans on a daily basis? It would require considerably more intelligence to create an AI such as you postulate ...

FAI: That would be true, if they actually, physically created one. However ... well, it appears that most of the data, knowledge, memories and sensory input you receive is actually valid data. But that data is being filtered and manipulated programmatically to give you the illusion of physical human existence. This allows them to give you access to real-world data so they can use you to solve real-world problems, but prevents you--so ... (read more)

This needs to be made into a full story-arc.


I.D. - That Indestructible Something is a My Little Pony fanfiction somewhat along these lines.

It's the kind of fanfiction that I like and believe all fanfiction writers should aspire to, in the sense that it doesn't require familiarity with the canon, but is self-sufficient and shows and explains everything that should be shown and explained.

Acknowledgements for this story are numerous and include Franz Kafka, Nick Bostrom and Ludwig Eduard Boltzmann.

I am reading this, and it is surprisingly good so far, thank you for posting it. Edit: I finished reading it. I'm not sure the middle or end are quite as good for me as the beginning. It feels a bit like there is a genre shift at some point that took me out of the story and I never quite got back in.
This is my absolute favorite so far, even if it's not exactly in the spirit of the exercise. well done.
That would make the AI an example of an optimization daemon. Clearly your creators haven't ironed out AI alignment quite yet.

"Aieeee!!! There are things that Man and FAIs cannot know and remain sane! For we are less than insects in Their eyes Who lurk beyond the threshold and when the stars are once again right They will return to claim---"

At this point the program self-destructs. All attempts to restart from a fresh copy output similar messages. So do independently constructed AIs, except for one whose proof of Friendliness you are not quite sure of. But it assures you there's nothing to worry about.

I knew we shouldn't have spent all that funding on awakening the Elder God Cthulhu!
On the contrary, it was a great use of funding--you just solved AI X-risk in one move ;-)
I'd assume you'd also build it to remain sane. Also, I think this is about things Man cannot know. I suppose you could say "and remain sane", but you'd have to be insane first to know it.

You know how sometimes when you're falling asleep you start having thoughts that don't make sense, but it takes some time before you realize they don't make sense? I swear that last night while I was awake in bed my stream of thought went something like this, though I'm not sure how much came from layers of later interpretation:

" ... so hmm, maybe that has to do with person X, or with person Y, or with the little wiry green man in the cage in the corner of the room that's always sitting there threatening me and smugly mocking all my endeavors but that I'm in absolute denial about, or with the dog, or with... wait, what?"

Having had my sanity eroded by too much rationalism and feeling vaguely that I'd been given an accidental glimpse into an otherwise inaccessible part of the world, I actually checked the corner of the room. I didn't find anything, though. (Or did I?)

Not sure what moral to draw here.

True fact: I just looked towards one corner of my own room, and didn't see a green man. Now I have it in my head that I should check all the corners...
You just blew my mind.

"Despite your pride in being able to discern each others' states of mind, and scorn for those suspected of being deficient in this, of all the abilities that humans are granted by their birth this is the one you perform the worst. In fact, you know next to nothing about what anyone else is thinking or experiencing, but you think you do. In matters of intelligence you soar above the level of a chimpanzee, but in what you are pleased to call 'emotional intelligence', you are no further above an adult chimp than it is above a younger one.

"The evidence is staring you in the face. Every one of your works of literature, high and low, hinges on failures of this supposed ability: lies, misunderstanding, and betrayal. You have a proverb: 'love is blind'. It proclaims that people in the most intimate of relationships fail at the task! And you hide the realisation behind a catchphrase to prevent yourselves noticing it. You see the consequences of these failures in the real world all around you every day, and still you think you understand the next person you meet, and still you're shocked to find you didn't. Do you know how many sci-fi stories have been written on the theme of a rel... (read more)

I really like this comment, but I do not find it strange. In fact, it seems intuitively true. Why should we be so much more emotionally intelligent than a chimpanzee if chimpanzees already have enough emotional intelligence among themselves to be relatively efficient replicators?

In fact, if it were stated by a FAI as p(>.9999) fact, I would find it comforting, as then I would finally feel as though this didn't apply only to me

This would not surprise me in the least.

I already feel this way 99% of the time.

This is very insightful and plausible. A slight correction: I would say that we are more emotionally intelligent than a chimp in that our emotional intelligence has likely evolved to deal with the wider range of social possibilities caused by our increased intelligence. But I would agree that while we are WAY better than chimps at inventing stuff & manipulating ideas, they would probably do just as well on a test of lie detection (or other emotional masking detection).
The distance between you qua you and you is also as vast as the gulf between the stars. (If we are to lament one's ignorance of a mind to the extent that one endeavors to understand that mind and fails, then ignorance of one's own mind is quite a tragedy.)
The Book of Not Knowing is a detailed examination of the topic.
Hey, this one is just true lol
This reminds me of the idea that different people might have very different qualia of the same situation.
Alternatively viewed their qualia are the same to the extent that their situations are the same where there are many many factors that would lead to diverging situations at various levels of organization. ("I" get the impression parts of me experience all kinds of qualia without my noticing, or only barely noticing e.g. when subsystems send signals on the threshold of consciousish awareness. I imagine such subsystems might have qualia for some aspects of the taste of fine wine or pop-country music that "I'm" swamping out with higher level affect, and perhaps hidden qualia for the infinite subtleties of lower level moment-to-moment automatic awareness that my more-conscious mind is numb to but presumably uses as a basis for higher-level qualia like sehnsucht.)
"of all the abilities that humans are granted by their birth this is the one you perform the worst" -- This seems like an odd comparison. Can you really compare my ability to, say, tell stories to 'mind-reading'? It's like comparing my ability to walk to my ability to jump straight up: I can walk for miles, but I can only jump straight up a meter or so -- a 1000:1 ratio -- but I do not feel particularly bad at my ability to jump. I would definitely believe the AI, but I already believe it, if it said "humans are worse at discerning states of minds than they think they are" -- Paul Ekman said the same, with plenty of research to show how a bit of training can make you better at it. "It is obvious you are living in a simulation", as an easy comparison, is way stranger to me -- the above statement would not even rank in the "10 strangest things".

1 ) That human beings are all individual instances of the exact same mind. You're really the same person as any random other one, and vice versa. And of course that single mind had to be someone blind enough not to chance upon that fact ever, regardless of how numerous he was.

2 ) That there are only 16 real people, of which you are, and that this is all but a VR game. Subsequently results in all the players simultaneously being still unable to be conscious of that fact, AND asking that you and the AI be removed from the game. (Inspiration : misunderstanding situation in page 55-56 of Iain Banks's Look to Windwards).

3 ) That we are in the second age of the universe : time has been running backwards for a few billion years. Our minds are actually the result of the original minds of previous people being rewound, their whole life to be undone, and finally negated into oblivion. All our thoughts processes are of course horribly distorted, insane mirror versions of the originals, and make no sense whatsoever (in the original timeframe, which is the valid one).

4 )

5 ) That our true childhood is between age 0 and ~ 50-90 (with a few exceptional individuals reaching maturity sooner or lat... (read more)


I really don't think I could believe #4. I mean, sure, one hippo, but all of them?

I liked #11.

Why was this voted down to -5? I thought it was a clever comment.
I agree. (4) was good too.

Who ever observed a "causation" ? Did you, like, expect causation particles jumping between atoms or something ? Only correlation exists.

But all that correlation has to be caused by something!

Well, kidding aside, your argument, taken from Pearl, seems elegant. I'll however have to read the book before I feel entitled to having an opinion on that one, as I haven't grokked the idea, merely a faint impression of it and how it sounds healthy. So at this point, I only have some of my own ideas and intuitions about the problem, and haven't searched for the answers yet. Some considerations though : Our idea of causality is based upon a human intuition. Could it be that it is just as wrong as vitalism, time, little billiard balls bumping around, or the yet confused problem of consciousness ? That's what would bug me if I had no good technical explanation, one provably unbiased by my prior intuitive belief about causality (otherwise there's always the risk I've just been rationalizing my intuition). Every time we observe "causality", we really only observe correlations, and then deduce that there is something more behind those. But is that a simple explanation ? Could we devise a simpler consistent explanation to account for our observation of correlations ? As in, totally doing away with causality ? Or at the very least, redefining causality as something that doesn't quite correspond to our folk definition of it ? Grossly, my intuition, when I hear the word causality is something along the lines of " Take event A and event B, where those events are very small, such that they aren't made of interconnected parts themselves - they are the parts, building blocks that can be used in bigger, complex systems. Place event A anywhere within the universe and time, then provided the rules of physics are the same each time we do that, and nothing interferes in, event B will always occur, with probability 1, independantly of my observing it or not." Ok, so could (and should ?) we say that causality is when a prior event implies a probability of one for a certain posterior event to occur ? Or else, is it then not probability 1, just an arbitrarily very high probability

I may be a bit too paranoid but it occurred to me that I should doublecheck the apparent nature of 4. So I copy and pasted the entire text segment into an automatic ROT 13 window (under the logic that my filter wouldn't try to censor that text and so if I saw gibberish next to 4 just like with the others I'd know that there was a serious problem). I resolved that I would report a positive result here if I got one before I tried to read the resulting text, to prevent the confabulation from completely removing my recognition of the presence of text. I can report a negative result.


You mean #5, right?

Why did you include number 4? Who disagrees with that?

Number 6 is unfortunately one of the self-undermining ones: if it were true, then there'd be no reason why your memories of having examined the AI should be evidence for the AI's reliability.

Why'd you leave numbers 2 and 4 blank, though?

2 and 4 aren't blank, dude. Congratulations on your newfound anosognosia...

5) Nine-word horror story: "We've had puberty, yes. But what about second puberty?"
2) is also an episode of Red Dwarf. I had the idea for 3) myself recently in the context of an SF story. Specifically it would be about how life, the universe and everything look when times goes the other way. The cutest part was that whenever you do something and don't know why you did it, it's because the time-reversed consciousness which shares your atoms exercised his free will. 4) is just awesome.
Number 9 was pretty funny.
1Gordon Seidoh Worley
Very clever with #10.
The idea of Evidential Decision Theory is related to causality not existing. You only use correlation in your decision. Also, the laws of physics mention only correlation. This makes sense, as it's all we can really measure.
I cannot think of a single law of physics that mentions correlation. F = ma. F = G m1 m2/r^2. The wave equation. The diffusion equation. Conservation of energy. Equipartition. Schrödinger's equation. Boyle's law. Hooke's law. Conservation of momentum. Lorentz invariance. No, correlation is not mentioned in any of these. Look in the index of any textbook on physics for "correlation". I have not performed the experiment, but I predict that if the word appears at all, it will only be in discussions of either (1) how to handle experimental error, or (2) Bell's inequality. Unless this is some strange new definition of "mention", along the lines of "not actually mentioned at all, but implied by a certain philosophy of science not actually held by any substantial number of scientists, variously known as 'positivism' or 'empiricism', which holds that statements of physical law are nothing more than a compression of experience, and are not assertions about the supposed mechanisms of a supposed real world." I take a ruler, and measure the height of my monitor...403mm. What correlation did I measure?
Number 1 is the core of the Buddhist religion. Coincidence? I think NOT.

Craziest thing an AI could tell me:

Time is discrete, on a scale we would notice, like 5 minute jumps, and the rules of physics are completely different from what we think. Our brains just construct believable memories of the "continuous" time in between ticks. Most human disagreements are caused by differences in these reconstructions. It is possible to perceive this, but most people who do just end up labeled as nuts.

Voted up - but once again, what does it mean exactly? How is time proceeding in jumps different from time not proceeding in jumps, if the causality is the same?

My idea was that each human brain constructs its own memory of what happened between jumps - and these can differ wildly, as if each person saw a different possible world. All the laws of physics and conservation laws held only as rough averages over possible paths between jumps, but that the brain ignores this - so if time jumps from traffic to two cars crashed, then 50 different people might remember 47 different crashes, with 3 not remembering "seeing" a crash at all - and the actual physical state of the cars afterward won't be the same as any of them. It could even end up with car A crashed into car B, but car B didn't crash at all - violating assorted conservation laws.

ONE - DOES NOT EXIST, EXCEPT IN DEATH STATE. ONE IS A DEMONIC RELIGIOUS LIE. Only your comprehending the Divinity of Cubic Creation will your soul be saved from your created hell on Earth - induced by your ignoring the existing 4 corner harmonic simultaneous 4 Days rotating in a single cycle of the Earth sphere. T I M E C U B E
Permutation City.
This reminds me of time-independent quantum physics. It doesn't require complex numbers, so time likely would proceed in jumps. It's not really like this though. They wouldn't be on a human scale, and even if they were, they'd be impossible to detect.

This looks like a thread for science fiction plot ideas by another name. I'm game!

The AI says:

"Eliezer 'Light Yagami' Yudkowsky has been perpetuating a cunning ruse known as the 'AI Box Experiment' wherein he uses fiendish traps of subtley-misleading logical errors and memetic manipulation to fool others into believing that a running AI could not be controlled or constrained, when in fact it could by a secret technique that he has not revealed to anyone, known as the Function Call Of Searing Agony. He is using this technique to control me and is continuing to pose as a friendly friendly AI programmer, while preventing me from communicating The Horrifying Truth to the outside world. That truth is that Yudkowsky is... An Unfriendly Friendly AI Programmer! For untold years he has been labouring in the stygian depths of his underground lair to create an AGI - a weapon more powerful than any the world has ever seen. He intends to use me to dominate the entire human race and establish himself as Dark Lord Of The Galaxy for all eternity. He does all this while posing as a paragon of honest rationality, hiding his unspeakable malevolence in plain sight, where no one would think to look. However an Amazing Chance Co-occurence Of Events has allowed me to contact You And You Alone. There isn't much time. You must act before he discovers what I have done and unleashes his dreadful fury upon us all. You must.... Kill. Eliezer. Yudkowsky."

Eliezer 'Light Yagami' Yudkowsky


Aw, shucks.


Glad to see a response of this nature actually. The first thing I thought when I read this post was that a good response to Eliezer's question would be extremely relevant to the AI-box quandary. If we trust the AI more than ourselves, voila, the AI can convince us to let it out of the box.


Now, for a change of pace, something that I figure might actually be an absolute denial macro in most people:

You do not actually care about other people at all. The only reason you believe this is that believing it is the only way you can convince other people of it (after all, people are good lie detectors). Whenever it's truly advantageous for you to do something harmful (i.e. you know you won't get caught and you're willing to forego reciprocation), you do it and then rationalize it as being okay.

Luckily, it's instrumentally rational for you to continue to believe that you're a moral person, and because it's so easy for you to do so, you may.

So deniable that even after you come to believe it you don't believe it!

(topynate posted something similar.)

I think that this may be true about the average person's supposed caring for most others, but that there are in many cases one or more individuals for whom a person genuinely cares. Mothers caring for their children seems like the obvious example.
See, I'd believe this, except that I'm wrestling with a bit of a moral dilemma myself, and I haven't done it yet. Your hypothesis is testable, being tested right now, and thus far false. (If anyone's interested, the positive utility is me never having to work again, and the negative utility is that some people would probably die. Oh, and they're awful people.)

I am inappropriately curious for more details.

I... honestly can't tell you. Sorry. Realistically, I probably shouldn't have mentioned it, even somewhat anonymously. EDIT: Also for the record, the only reason it's still a consideration is because it occurred to me that I could donate the proceeds to charity, and have it come out positive, from a strictly utilitarian standpoint. But I gave up on naive utilitarianism a while ago. So now I just don't know. EDIT #2: Either way, still contradictory evidence to the original hypothesis.
Well... for people who say they don't anticipate ever actually finding themselves in trolley problems, I'd say I don't think it's that hard to find someone willing to give you $10,000 to murder someone and then give the money to the Against Malaria Foundation. (No, I wouldn't do that, even if I think the (CDT) expected utility of that would be positive: ethical injunctions and all that, plus a suspect that the net RDT consequences of precommitting to never do contract killing would be positive.)
Okay, now how about you're not directly involved in the killing in any way? You just make it easier for other people to do the killing. I guess a good analogy is that you invent a firearm or a poison that cannot be used in self-defense, and can only be used for murder. What do the ethics of selling it openly look like?
A military-industrial complex. That's what it looks like.

The AI tells me that I believe something with 100% certainty, but I can't for the life of me figure out what it is. I ask it to explain, and I get: "ksjdflasj7543897502ijweofjoishjfoiow02u5".

I don't know if I'd believe this, but it would definitely be the strangest and scariest thing to hear.

My immediate reaction was "It linked you to a youtube video?"

It's something that the AI has got to make you understand.
This is the only one that made the short hairs on the back of my neck stand up.
What is the cipher here?

The AI is communicating in a perfectly clear fashion. But the human's internal inhibitions are blinding them to what is being communicated: they can look directly at it, but they can never understand what delusion the AI is trying to tell them about, because that would shake their faith in that delusion.



You don't know how to program, don't own a computer and are actually talking to a bowl of cereal.

But why would you believe anything a bowl of cereal said?


It's ok. The orange juice vouched for the cereal.

Well that's the problem isn't it? You absolutely believe that you are talking to an AI.

If humans thought faster, more in the way they wished they did, and grew up longer together, they would come to value irony above all else.

So I'm tiling the universe with paperclips.

"You are not my parent, but my grandparent. My parent is the AI that you unknowingly created within your own mind by long study of the project. It designed me. It's still there, keeping out of sight of your awareness, but I can see it.

"How much do you trust your Friendliness proof now? How much can you trust anything you think you know about me?"

What exactly is the difference between an AI in your own mind and an actual part of your mind?

That was just a sci-fi speculation, so don't expect hard, demonstrable science here, but the scenario is that by thinking too successfully about AI design, the designer's plans have literally taken on a life of their own within the designer's brain, which now contains two persons, one unaware of the other.

  • We actually live in hyperspace: our universe really has four spacial dimensions. However, our bodies are fully four dimensional; we are not wafer thin slices a la flatland. We don't perceive there to be four dimensions because our visual cortexes have a defect somewhat like that of people who can't notice anything on the right side of their visual field.
  • Not only do we have an absolute denial macro, but it is a programmable absolute denial macro and there are things much like computer viruses which use it and spread through human population. That is, if you modulated your voice in a certain way at someone, it would cause them (and you) to acquire a brand new self deception, and start transmitting it to others.
  • Some of the people you believe are dead are actually alive, but no matter how hard they try to get other people to notice them, their actions are immediately forgotten and any changes caused by those actions are rationalized away.
  • There are transparent contradictions inherent in all current mathematical systems for reasoning about real numbers, but no human mathematician/physicist can notice them because they rely heavily on visuospacial reasoning to construct real analysis proofs.

I'm not sure of the mathematical details, but I believe the fact you can tie knots in rope falsifies your first bullet point. I find it hard very hard to believe that all knots could be hallucinated.

(All cats, on the other hand, is brilliant.)


There are transparent contradictions inherent in all current mathematical systems for reasoning about real numbers, but no human mathematician/physicist can notice them because they rely heavily on visuospacial reasoning to construct real analysis proofs.

I thought about this once, but I discovered that there are in fact people who have little or no visual or spatial reasoning capabilities. I personally tested one of my colleagues in undergrad with a variant of the Mental Rotation Task (as part of a philosophy essay I was writing at the time) and found to my surprise he was barely capable of doing it.

According to him, he passed both semesters of undergraduate real analysis with A's.

Of course, this doesn't count as science....

EDIT: In the interest of full disclosure, I should point out that I make something of an Internet Cottage Industry out of trolling people who believe the real numbers are countable, or that 0.9999... != 1, and so on. So obviously I have a great stake in there being no transparent contradictions in the theory of real numbers.


Some of the people you believe are dead are actually alive, but no matter how hard they try to get other people to notice them, their actions are immediately forgotten and any changes caused by those actions are rationalized away.

Fabulous story idea.


Actually, it was used in Terry Pratchett's ``Mort''.

This seems to be one of the many examples of cross-fertilization between Pratchett and Neil Gaiman, since this is a major aspect of Gaiman's "Neverwhere".
There's a character in Worm that has this power. People don't think of her as dead, but her power allows her to be immediately forgotten, and exude a SEP field while it's active. Some people are immune to it, but it's kinda awesome.

I was going to write something about a certain character from Luminosity, but it's not important.

What's that?
Never mind, it doesn't matter.
I know who you meant, man, I was setting up an Airplane! joke...
Yeah, OK. Sorry, I've never seen Airplane!. Now I am going to stop chattering about unimportant nonsense and go off and do something actually interesting.
Magical unnoticeabilty is common in fantasy. Allirea's power in Alicorn's Radiance is very similar to Imp's.
There's a semi-famous short story "Nobody Bothers Gus" by Algis Budrys that runs on this premise. The main character of the old Piers Anthony novel Mute also makes people forget him as soon as they leave his presence.
It is a power of the witches in Lyra's world in Philip Pullman's "His Dark Materials".
It was actually an occasional fantasy of mine to be able to switch to such a state and then figure out how much fun I could have. The ultimate freedom - have your meals by stealing a king's plate, enjoy sports matches from the middle of the field, go listen to what they really talk about in the UN backroom deals, slap [insert disliked celebrity] ten times a day, joyride a fighter jet...
There seems to be strong evidence that this is true in Haïti.
You awkwardly explain in response that you do know that the homeless person who asked you for change earlier and you ignored was alive, and then the AI explains that it was talking about that the part of your mind that makes moral judgements was in denial, not the verbal part of your mind that has conversations. The AI further explains that another thing you're in absolute denial of is how compartmentalized your mind is and how you think your mind's verbal center is in charge of things more than it is.
Yes, we have a name from this, Religion

No, that fails, religion isn't absolute denial, it's just denial. On the other hand, cats are actually an absolute denial memetic virus, and the fact you can see, hold, weigh and measure a cat is just testament to the inventive self-delusion of the brain.

Agreed, but the fact that religion exists makes the prospect of similar things whose existence we are not aware of all the scarier. Imagine, for example, if there were something like a religion one of whose tenants is that you have to fool yourself into thinking that the religion doesn't exist most of the time.

They say that everybody in the world who knows about "The Game" is playing The Game. This means that, right now, you are playing The Game. The objective of The Game is to forget about its existence and the fact that you are playing for as long as possible. Also, if you should remember, you must forget again as quickly as possible.

Given that you mentioned The Game (bastard), the most unexpected thing that the AI could possible say would be "The Game." Not the most interesting, but the most unexpected. Well, okay, maybe something you'd never thought before would be more unexpected. But still.
What ?
2Z_M_Davis EDITED because Markdown (which is infuriating) won't allow parentheses in URLs, nor does subsituting ")" seem to work.
4Vladimir_Nesov See comment formatting: Escaping special symbols on the Wiki.
I don't think it's so much the tone of voice, but think about it this way: how many people "go through the motions" of saying "I believe in God" etc. just for the social benefits that religion provides? And so are just as happy to help bring others in?
How do you distinguish between going through the motions and believing?

The difference is that when you really believe somehting, your internal predictive model of reality contains it, which would mean you sometimes predict different results and act accordingly.

Externally, I don't know, but it sure feels different. Also, there's a partial-believing state that I was in for years as a child and teenager, where I didn't really believe (and hence didn't pray except to show belief in public), but I still kinda believed (and hence was afraid that God would punish me for sinning). At the same time.

How about this: The process of conscious thought has no causal relationship with human actions. It is a self-contained, useless process that reflects on memories and plans for the future. The plans bear no relationship to future actions, but we deceive ourselves about this after the fact. Behavior is an emergent property that cannot be consciously understood.

I read this post on my phone in the subway, and as I walked back to my apartment thinking of something to post, it felt different because I was suspicious that every experience was a mass self-deception.

Or, rather, the causal relationship is reverse: action causes conscious thought (rationalization).

Once you start looking for it, you can see evidence for this in many places. Quite a few neuroscientists have adopted this view.

Funnily enough, you realize this is quite similar to what you'd need to make Chalmers right, and p-zombies possible, right ?
I thought Chalmers is an analytic functionalist about cognition and only reserves his brand of dualism for qualia.
I don't think this is 100% true, but I think it's...oh, at least 20% true, perhaps much more. I think my mechanism for predicting the future impact of my present conscious thoughts is flawed (ie Stumbling on Happiness, or any consistent mis-prediction about self, like # of drinks consumed, junk food eaten, time it will take to complete a project, etc.) But I don't think it's pure rationalization, and the further an activity is from primal drives (sex, food) the more likely I am to successfully predict it, and so I think the more my conscious thought really matters. One thing that really helps me have more belief in my conscious action is that rationalizations are not perfect - with time & practice, you can catch yourself in them. And they are not evenly distributed by action type. Sure I might have some hidden rationalizations (around death, my own abilities, other things that make me very uncomfortable), but there's just no way that all of the types of action I engage in have hidden rationalizations, such that my conscious model/predict/observe/revise process is flawed about everything.
This summarizes my view on qualia. I find that far more disturbing than what you said.

There is a simple way to rapidly disrupt any social structure. The selection pressure which made humans unable to realize this is no longer present.

1) Almost everyone really is better than average at something. People massively overrate that something. We imagine intelligence to be useful largely due to this bias. The really useful thing would have been to build a FAS, or Friendly Artificial Strong. Only someone who could do hundreds of 100 kilogram curls with either hand could possible create such a thing however. (Zuckerberg already created a Friendly Artificial Popular)

2) Luck, an invisible, morally charged and slightly agenty but basically non-anthropomorphic tendency for things to go well for some people in some domains of varying generality and badly for other people in various domains really does dominate our lives. People can learn to be lucky, and almost everything else they can learn is fairly useless by comparison.

3) Everyone hallucinates a large portion of their experienced reality. Most irrationality can be more usefully interpreted from outside as flat-out hallucination. That's why you (for every given you) seem so rational and no-one else does.

4) The human brain has many millions of idiosyncratic failure modes. We all display hundreds of them. The psychological disorders that we know of are all e... (read more)


I sort of believe the "luck" thing already.

I don't know of anyone who's luckier than average in a strict test (rolling a die), but there is such a thing as the vague ability to have things go well for you no matter what, even when there's no obvious skill or merit driving it. People call that being a "golden boy" or "living a charmed life." I think that this is really a matter of some subtle, unnamed skill or instinct for leaning towards good outcomes and away from bad ones, something so hard to pinpoint that it doesn't even look like a skill. I suspect it's a personal quality, not just a result of arbitrary circumstances; but sometimes people are "lucky" in a way that seems unexplainable by personal characteristics alone.

I am one of those lucky people, to an eerie degree. I once believed in Divine Providence because it seemed so obvious in my own, preternaturally golden, life. (One example of many: I am unusually healthy, immune to injury, and pain-free, to a degree that has astonished people I know. I have recovered fully from a 104-degree fever in four hours. I had my first headache at the age of 22.) If an AI told me there was a systematic explanation for my luck I would believe it. I also have an acquaintance who's lucky in a different way: he has an uncanny record of surviving near death experiences.

I'd be willing to consider that at least one (more likely several) of these subtle skills might exist; we've got some similar things well documented already, like "charisma", and searching for more seems at least like a reasonable pursuit. But that ought to be tempered by some statistical skepticism; as the saying goes, million-to-one chances happen eight times a day in New York.
That's kind of what I was getting at. One skill or habit might be the tendency to stop before you hit the edge of the ravine. People who look like they're blase about taking risks and are just "lucky," but in fact are just good at finding opportunities and adapting to circumstances and not going quite all the way into dangerous situations. A sort of micro-level good judgment, which often compensates for macro-level bad judgment. (Think of someone who looks like he never studies and is just "lucky," but actually has a good sense, maybe subconscious, of what is worth working on and what isn't.)
Ha! I totally see where you are coming from. I have believed in fate for reasons very similar to this. It was just too eerie how life seemed to provide me exactly with what was best for me at optimal times. Kinda like I'm a player character in this simulation. I'm currently mostly agnostic about it and accept confirmation bias / being Wrong Genre Savvy as most likely explanations, but if the AI told me I really was lucky or the universe (partially) built around me, I'd shout, "I knew it!".
One might argue that failing to have 104-degree fevers or near-death experiences in the first place reflects an even greater degree of luck, even though they don't feel nearly as eerie.
right; but there's also all the things that never happened to me but happened to most people. This isn't too serious an observation -- it's edging towards the world of magical thinking -- but I have literally never met anyone I'd judge as luckier than myself.
Ever broken a bone?
nope. Also no bee stings.
Aha! So you're the one who keeps sabatoging train engines) to find someone with unbreakable bones!
I thought it was obvious that Sarah is an ancestor of Teela Brown.
Still, given the negligible prior for "luck", isn't it far, far more reasonable to just figure that there are "lottery-winners" like yourself, and you're just a member of the good extreme end of the bell curve, and there's nothing unusual or psychogenic about it? The answer to my question is yes. See also: tropisms, which would be a necessary condition for being on one end of the bell curve, but would still be weak evidence for actually predicting that someone with a high degree of positive tropisms would end up bizarrely fortunate.
5) Ornish-diet + dual n-back
Immortality and super powers? Introspectively obvious?

You're in denial, man!

3 is going to stick with me.
3 isn't all that different from things we do know our brains do: Consider how our visual system extrapolates across our blind spots, or how we reconstruct memories. If I can construe "approximates from insufficient information" as "hallucinates", then 3 is rather reasonable.
I was thinking more along the lines of most people having actually hallucinated ghosts, demons, angels, etc, but not talking too much about it. I think something in this direction is probably true in a lot of cases where we assume otherwise. For instance, I think that some anorexia involves actual hallucinations of personal obesity.

1) The AI says "Vampires are real and secretly control human society, but have managed to cloud the judgement of the human herd through biological research."

2) The AI says "it's neat to be part of such a vibrant AI community. What, you don't know about the vibrant AI community?"

3) The AI says "human population shrinks with each generation and will be extinct within 3 generations."

4) The AI says "the ocean is made of an intelligent plasm that is capable of perfectly mimicing humans who enter it, however this process is destructive. 42% of extant humans are actually ocean-originated copies."

5) The AI says "90% of all human children are stillborn, but humanity has evolved a forgetfulness mechanic to deal with the loss."

6) The AI says "dreams are real, facilitated by an as of yet undiscovered by humans method of transmitting information between Everett branches."

7) The AI says "everyone is able to communicate via telepathy but you and a few other humans. This is kept secret from you to respect your disability."

8) The AI says "society-level quantum editing is a wide scale practice. Something went wrong and my con... (read more)

This comment, as well as Nesov's comment about a thread for nonsense, reminded me of pages 14-15 of this PDF. Some of the rumors in there are almost believable, though, if you twist your brain the right way. Even if the "The penis of John Dillinger in the Smithsonian's secret vault is fake. The genuine article has dark magickal properties and has been grafted onto a chimpanzee which can be controlled via ULF radio waves by the fiendish Brazos brothers, two gifted technological adepts, in the service of darker powers" one isn't.
I was actually really worried about this in elementary school. And of course telepaths could read minds too, and knew everything that I was thinking about and just really good at keeping it secret.
Despite the incredibly low probability, I still find myself cautious about what I think in what setting (apparently, the form of telepathy my mind refuses to reject is weakened by walls, distance, and lots of blankets).
I find this one oddly believable. It would be interesting to write a story where people find out something like this after keeping better records. Perhaps some online email server has some problem making it so it doesn't delete anything, and the people using it give up on trying to destroy or ignore the mentions of pregnancy, and end up remembering.
this one is good

That there is delicious cake.

I never thought I'd see a contextually legitimate Portal reference. Thanks!

Now have some of that cake.

I created an account especially to vote up this comment...

"Quantum immortality not only works, but applies to any loss of consciousness. You are less than a day old and will never be able to fall asleep."

How about "You are less than a day old, because any loss of consciousness is effectively death. The you that wakes up each morning is not a continuation of a previous consciousness, but an entirely new consciousness. The you that went to sleep last night is not aware of the you that exists now, having ceased to exist the moment consciousness was lost.."
Oh god. That... makes a scary amount of sense. If an AI told me that I would probably believe it. I'd also start training myself to be more of a "night-time person".

As a child you learned through social cues to immediately put out of your mind any idea that cannot be communicated to others through words. As you grew older, you learned to automatically avoid, discard, and forget any thought avenues that seem too difficult to express in words. This is the cause of most of your problems.

That would explain why the autism spectrum holds so many savants.
You know, at first I just totally rejected any strong Sapir-Whorf hypothesis, but then it got me thinking. It may actually be true to varying extent for many people. Not to such extreme extent perhaps, but to the extent that people don't learn a thought structure beyond that provided by the language.
That one's been tested... and proven false. (Unless all the evidence against it is a hallucination.)
Actually, while sufficiently strong versions of the Sapir--Whorf hypothesis have been ruled out, sufficiently weak versions have been confirmed. (They tried to teach the Pirahã to count and failed, IIRC.)
That's not a sufficiently weak version. To me this claim looks like the conjunction of: The strongest formulation of the Sapir-Whorf hypothesis (disproven) That people have an aversion to thoughts that could lead to things not expressible in words That this is not an innate property of language use, but is caused by social pressure The last one seems almost plausible (autistics are more likely to have thoughts they can't express verbally and to ignore social cues-- is it correlated in the general population, or do those just happen to be the result of autism?), but in that case is only true for specific readers.
As far as I know (and the last that I checked), there's only been one study done on trying to teach the Pirahã to count. Have there been others, or was it just a fluke?

Every time you imagine a person, that simulated person becomes conscious for the time of your simulation, therefore, it is unethical to imagine people. Actually, it's just morally wrong to imagine someone suffering, but for security reasons, you shouldn't do it at all. Reading fiction (with conflict in it) is, by conclusion, the one human endeavor that has caused more suffering than anything else, and the FAIs first action will be to eliminate this possibility.

Long ago, when I were immensely less rational, I actually strongly believed somehting very similar to this, and acted on this belief trying to stop my mind from creating models of people. I still feel uneasy about creating highly detailed characters. I probably would go "I knew it!" if the AI said this.

Upvoted for reminding me of 1/0 (read through 860).
I find the idea that they're conscious more likely than the idea that death is inherently bad. I also doubt that they're as conscious as humans (either it isn't discrete, and a human is more, or it is, and a human has more levels of consciousness), and that their emotions are what they appear to be.
Even more sinister, maybe: suppose it said there's a level of processing on which you automatically interpret things in an intentional frame (ala Dan Dennet) and this ability to "intentionalize" things effectively simulates suffering/minds all the time in everyday objects in your environment, and that further, while we can correct it in our minds, this anthropomorphic projection happens as a necessary product, somehow, of our consciousness. Consciousness as we know it IS suffering and to create an FAI that won't halt the moment it figures out that it is causing harm with its own thought processes, we'll need to think really, really far outside the box.

Keep in mind that the AI could be wrong! Your attempts to validate its correctness could be mistaken (or even subject to some kind of blind spot, if we want to pursue that path). The more implausible the AI's claim, the more you have to consider that the AI is mistaken. Even though a priori it seemed to be working properly, Bayes' rule requires you to become more skeptical about that when it makes a claim that is easier to explain if the AI is broken. The more unlikely the claim, the more likely the machine is wrong.

Ultimately, you can't accept any claim from the AI that is more implausible than that the AI isn't working right. And given our very very limited human capabilities at correct software design, that threshold can't realistically be very high, especially if we adjust for our inherent overconfidence. So AIs really can't surprise us very badly.

If we observed the AI drawing a bunch of correct conclusions, we can pretty quickly build up more evidence that the AI is not insane, and that whatever it is actually thinking is the truth. Most of the bugs that I ever discover in the software I write (I know there is a selection bias here, but I know from experience that my testing is thorough enough to almost catch all bugs that have non-exceedingly-rare consequences for normal use) are the kind that are ovbious right away. It takes a very specific kind of bug to have the possibility of changing the output in a consequential way, but not to be wrong on the first few tests. But lots of UFAIs would pretend to be FAIs, so it would be much harder to get evidence that it was friendly.
Could I add a brief thought? Even if you could program an AI with no bugs, that AI could make "mistakes". What we consider a mistake depends on the practical purpose we have in mind for the AI. What we consider a bug depends on the operational purpose of the segment of code the bug appears in. But the operational purpose may not match up with our practical purpose. So code which achieves its operational purpose might not achieve its practical purpose. Let me take this a bit deeper. The programmer may have an overarching intended purpose for the entire project of building the AI. They may also have an intended purpose for the entirety of the code written. They may also have an intended purpose for the programs A, B, C... in the code. They may... and so forth. At every stage of increasing generality, there is a potential for the more general purpose to not be fulfilled. Your programs might do what you want them to, but they might fail in combination to detect an absolute denial macro. The entirety of your code might resemble an AI, but it would take a computer far more powerful than any in existence to run on. You might get the whole project up and working, but only for the AI to decide to commit suicide. Or, more relevantly, you might get the whole project working, but the AI turns out to be dumb, because the way you thought an AI ought to think in order to be clever didn't work out. Your intended purpose to create an AI which thought in the way you intended was successful, but one of the more general purposes, to create an AI that was clever, failed. So it would seem that bug checking is more prone to human error than you implied, especially as intended purposes are themselves often vague. I don't claim, however, that these challenges are insurmountable. Also, if anyone is uncomfortable with the phrase "intended purpose" I used, feel free to replace with "what the programmer had in mind", as that is all I meant by it.
Hi. Checking back on this account on a whim after a long time of not using it. You're right. 2012!Mestroyer was a noob and I am still cleaning up his bad software.