A personal, experiential response to: Avoiding Your Belief's Real Weak Points, by Eleizer Yudkowsky.

I wrote the following article in order to help me express and process and eventually publish my thoughts, while setting myself to accept the challenge posed in Eleizer's post, which I had just reached in the official podcast of Rationality: AI to Zombies:

To do better: When you’re doubting one of your most cherished beliefs, close your eyes, empty your mind, grit your teeth, and deliberately think about whatever hurts the most.

Wishing to become stronger, I tried to ask the unasked question. Find a reason that, even if I do not generally acknowledge it as a reason of my own, might be a potential reason for someone to hold onto a belief that I had been holding onto. A reason other than ranking it as the most likely thing to be true. I looked for what scary thing might be hiding behind it, that having a comfortable and habitual answer might be saving me from having to look at directly.

And... I think I found it.

This experience took place nine days ago. I began writing this record identifying myself as still holding the beliefs to be challenged, and ended it feeling that that may no longer be the case. Looking back, and looking at the way I have felt and thought since then, I think I did succeed in dislodging my habitual way of looking at the world somewhat from its privileged position. The cognitive habits are still there, but I'm taking notice as I fall into them, and marking them as something to counter-balance against. Giving them penalties for arising from a process I acknowledge to have particular and known flaws.

Naturally, this is a very emotional, very personal, and potentially very challenging post. It may qualify as an infohazard. You could, depending on your current set of beliefs and how entrenched they are, be hurt by the consequences of following me through to the other side of the fire. Just wanted to give you fair warning, and a chance to stop reading now. With that out of the way... Let the journey begin.

I wish to test myself by the standards set here, as I understand them, in a public confessional. Others with an outside perspective of my beliefs may judge me, and their judgements may show me that there are things they see that I have ignored without meaning to.

I am agnostic now about beliefs I used to hold with more confidence, about psychic phenomena, suggestive magic. I was/maybe-am somewhat mystic and animist, believing essentially in subtle spirits that develop in some places and objects, in moments of spiritually 'guided' insight that turn out to be apt. Experiences of bizarre and apparently unsupported certainty that bore fruit; of ghostly sights and lightning striking in the right place, at the right time, in the right shape; and moments of disjointed and distant feelings and thoughts that found a source of confirmation and context in the contemporaneous experiences of others who were dear to me... These are my miracles and my proofs, the experiences that support those beliefs. But remembering the feeling of those experiences, awesome though they were in the classic meaning of the word, is only a recital of old evidence. I can see that.

Where is the weakness? What might I be avoiding looking at, when I cling to mysticism and animism? If they are an illusion of fallacies, what pain does this illusion shield me from? Where does the idea of letting mysticism/animism be untrue hurt? The test is to look there... right?

Then... I ask myself, do I think I could bear it if the truth were that some of the most profound and self-affirming experiences of my youth were based on... chance, happenstance, luck; and perfectly mortal and explicable observation, deduction and common sense on my part (to the extent that moments of "inspired" insight were, in fact, insightful)? Could I viscerally imagine that all my psychic moments were either decent cold reading or luck, and remain standing?

...Yes. I... think so. It draws tears from my eyes, but mostly of pity for my former self's tremendous lonesomeness, for that person was indeed desperate to believe that something, anything, even if it wasn't a person, was noticing them and paying attention to their feelings. That their feelings meant something to someone else, anyone else, even if that other person never knew the connection, and it only existed because they were that other person's feelings, picked up empathetically like a distant, barely coherent radio signal, not my former self's own.

Today, I have friends I can talk to about anything, people who will stare into the darkness of my own painful experiences with me, and flinch, but keep looking. Who will cry with me. Who notice, and who care. I am no longer alone the way I once was. If the truth is that spirits acknowledging or communicating to me was an illusion I formed out of desperation, I think it is an illusion I could relinquish. It would hurt, for there was and still is beauty, poetry, in those stories. But I do think I could take the hit, and remain standing.

Then, I search again. I ask myself, could I bear it if the truth were that people... and animals too... suffer and die; live lives of unendurable hardship, and fail to endure them, and die; or are hobbled in ways that never heal... and the world simply does not care. The molten core of the earth does not empathize. The stones of old buildings do not capture and reflect ghosts when people die pointlessly and traumatically. The handicrafts of artisans working miserably in conditions which are slavery in all but name are indistinguishable from the output of those with similar skill who work for love of their craft, no matter how quietly and empathetically a sensitive young mystic might entice the object to relate its history with an attached sense of pride or of sadness. Could I bear to believe that when no human or animal knows to weep for the loss of joy, of freedom, of life, those experiences, in their personal, specific contexts, really do go starkly unmourned? Can I stand in a world where the tender personal stories of peoples' lives are lost, erased and burned to less than ashes, without ever being read, and can never be recovered?

... I am less sure of that. But, I... think so. I can viscerally imagine myself standing in such a world. With that picture, something inside of me seems to shrivel and crumble further toward dust, that had aleady become dry and old and weak, like a leaf once brightly coloured. It is a steel-edged picture, and my self in that picture stands with a steel-edged heart, just a little colder. Just a little, for I already know the world is cruel. But perhaps I had hoped that in those times when I felt suddenly more introspective and more sorrowful, and I could think of no other reason why, that it was a little empathetic tingle, a duty to someone who was now suffering; a little vigil, though anonymous and far away and perhaps never realized, in acknowledgement of someone who was now dying.

What then is the result of these contemplations? Where do I stand having challenged the weaknesses I could see in my beliefs (although already tenuous ones)? I feel a bit more moved to take responsibility myself for that suffering that goes unmourned because it is ignored. I feel a bit more of its urgency and intolerability. The voice of the sword of desperation echoes distantly somewhere behind my thoughts.

Well. This seems, upon first assessment, grounds to count my challenge successful. I reach out tentatively to a place where there used to be an attachment to mysticism and animism. I do not feel it, at the moment, and I wonder whether it is already completely gone, or only blocked or drowned out temporarily by the steel and the pain.

If, after this, I find that I wish to take a long walk and call out with my mind for unexpected signs to interpret as answers to a question, to find out whether I discover anything which relates to that question through a chain of bizarre connections...

If I try to find a way to blind and de-bias a test for psychic communication, quietly wishing in my heart of hearts for it to show some confirmation...

If I find I am tempted, sometimes, to toss the fortune-telling sticks, just in case there is something to it... because not needing to believe something does not, in and of itself, make it false...

Because I have a certain spark of something I will call motivated curiosity...

And if I do it...

Will you who read this confessional believe that act to show I have not sufficiently faced the weaknesses of my mystical (former?) beliefs? Will it demonstrate failure?

Tell me what you think, and why, if you would be so kind.

~If I have done well, praise me; If I am yet lost, guide me.~

Addendum: (still part of the report on the same day it was written)

While I was rereading this to correct typing errors and clarify wording, I think I found a place to dig just a little deeper.

With that picture, something inside of me seems to shrivel and crumble further toward dust, that had aleady become dry and old and weak, like a leaf once brightly coloured.


But perhaps I had hoped that in those times when I felt suddenly more introspective and more sorrowful, and I could think of no other reason why, that it was a little empathetic tingle, a duty to someone who was now suffering; a little vigil, though anonymous and far away and perhaps never realized, in acknowledgement of someone who was now dying.

Perhaps I had hoped that, come the worst, someone, someday, somewhere... would do the same honour for me.

...And the brown and withered leaf is consumed by a brief, tiny plume of fire, and dissolves into smoke and ash and the husk of a stem. I think I have found it. The most vulnerable, the most painful thought that sits behind those beliefs, that need not be thought so long as they are in the way. I am afraid to die unmourned. Perhaps, even more than I am simply afraid to die.

Am I strong enough, then, to face that without my paper shield, if paper it were to be? I quail at the thought. I sob at my desk. I already know this pain, I have cried for the fear before. But still, I do not know, and I do not know, and I shake my chains, desperate to flee back to a safer seeming place.

My emotions burn out. I become numb. I feel a faint headache. I feel I have been dumped back into a safer seeming place. I breathe. I tidy my face. Perhaps I will make a point of bringing this up the next time I talk to my counsellor. Perhaps I will make a point of trying again on another day. Perhaps that glimpse into the maelstrom was the answer, and I have found it, and faced it..? Not for the first time. Not for the last. But have I done my duty for the moment, in acknowledging it?

This ends the record I kept of my test of faith nine days ago. I had always intended, from the beginning, to publish it somewhere that it might get a little bit more traffic from the rationalist community than on my personal blog. I knew there is always some risk of having solved something to my satisfaction only because my satisfaction was informed by motivated stopping.

It may have been too raw, just then, to share immediately. The internet can be cruel, and when it is, that doesn't really help me take the valid parts of its feedback on board. Now that it's nine days later, and I've had some time to reflect... I think I already have some answers to the questions I asked of the unknown reader and potential judge. I still want to check and compare them with the answers of other sensible people, if I can get some, but I've done some thinking myself first. That seems important.

I don't know, exactly, what I believe now. To what degree I actually do still think that the eerie moments, the extreme cases in my life, are something more than background noise distorted through a filter of desperation for validation and meaning. What I can say is that I think I ought to make up my mind based on new evidence I start to collect now. My memory of the past is distorted. Hindsight bias prevents me from weighting it usefully. I will still refer to it in order to remember what the 'mystical' hypothesis and point of view was... but not to confirm it.

I don't have the answer. But I have the rules for finding it. So I feel I gained from the experience. Thank you, Eleizer. Again.


2 comments, sorted by Click to highlight new comments since: Today at 2:41 AM
New Comment

I would also like to thank Eneasz Brodski, without whose wonderful podcast project of Harry Potter and the Methods of Rationality and other selected works, I don't think I would have come across "The Sword of Good".

It turns out, the question of what I will believe now came up much sooner than anticipated, and I have a long written follow-up to report. If it seems that this would stand better as its own post, tell me.

After school today, I stopped at home briefly to empty my backpack and then head out to the grocery store in order to pick up some drinks, and in order to take the time to walk and think. My friend had confessed to me that they were caught in an anxiety spiral, fearing that they might be a burden on me by tending to approach me when they were in need of help, and asked me to help them unpack why they were prone to this kind of problem. I had given them some quick emotional reassurances and promised to try to address their concerns more thoroughly later.

I was thinking about that, trying to compose in my head a further reassurance that might help alleviate their anxiety about imposing costs on me, an anxiety I know far too well, by explaining the benefits that come along with it. I was thinking in terms of a 'prison' scenario I'd written about which I knew they had already read, and also in terms of some symbols from HPMoR which I also knew they had already read. One of the comparisons I had thought of that I was particularly happy with was to compare the comfort I get out of being able to actually help them to the healing effect of the song of the phoenix, and I had also considered other phoenix-imagery, like the way it makes me feel strong to be there for them filling me with phoenix fire.

I paused, at about this point, and looked up for a moment to decide where I was walking. I had turned down a road that I could follow to a grocery store I liked to shop at, but I had also already passed the grocery I usually shop at. I took a brief moment to measure how much I wanted to keep walking, against how much I wanted to get home faster and start writing down my explanations to give to my friend. After a moment of indecision, I turned back the way I had come and progressed towards the store I had already passed.

There was one intersection I had to cross to get there, and at that intersection I encountered a man, a woman and a very big dog, who seemed friendly. I was in a good mood, and held out my hand cautiously to the dog, who didn't immediately seem to take an interest.

"May I pat him?" I asked.

"Sure," the man said, and I did. Then, "Sit down, Phoenix."


"His name is Phoenix?"

"Yep, Phoenix is his name."

"I... was just thinking about phoenixes," I told him with an awkward little smile, and patted the dog for a moment more, before my light turned green and I crossed the road.

A part of me turned to me and asked myself,

Am I allowed to start counting that as evidence yet?

I will from this point call this internal voice Serp1, and distinguish it from a second voice, Serp2.

Serp2: Um... I don't know.

This next part of the conversation (indented) was not actually expressed in words, but I will do my best to translate it accurately into them.

Serp2: I went through all that work to distance myself from the mysticism thing. I don't want to give that all away.
Serp1: You want to hold onto the points you've earned with your rationalist friends for coming to a conclusion that looks rational, and sticking to it. I understand that, but that wasn't what we agreed to. Our analysis and genuine challenge of mysticism was undertaken to seek truth, and that means weighing and judging the evidence FOR mysticism as well as the evidence against it. We agreed to start fresh and ignore past evidence, but we are not just walking away from this entire question as though it doesn't matter to us. And we are especially not doing it to conform to an in-group.
Serp2: It hasn't even been one full day since I actually published that post!
Serp1: Yeah. It kind of sucks that we didn't have more time to practice looking at the world assuming the null hypothesis first, but I am not going to sit by and let you ignore what just happened if it's legitimate evidence that ought to inform our beliefs.

Serp2: Okay fine. Let me think about this...

Serp2: Alright. What just happened does seem like it should be a piece of evidence, the way I understand it. Weak evidence, that could have happened in a world with no magic, but it seems much less likely that it would.

Oh, geez. Now what am I supposed to do with that again?

... There's a neat little feeling that I observed here, which reminds me of this quote: "Her mind skipped gears, ground against itself, and spat back the instructions for doing a science investigation project" (Hermione being tested, HPMoR chapter 8). That is an almost perfect description of it. A feeling of cognitive drain, and then a result popping out.

Serp2: We're supposed to try to predict the priors of that happening in the case of a world without magic. Ummm... How the hell do I quantify that?

[Cognitive drain, memory search result.]

Serp1: What are the odds of an observation that would seem at least that weird in a way that seems to suggest that reality is responding to my thoughts, or my thoughts are responding to reality, without recourse to any of the normal, reasonable modes of entanglement, arising just as background noise? What are the chances of seeing an event that favours the mysticism hypothesis as much as this one does?

Serp2: Buh. I don't know... One in twenty? No... I can't put a useful number to it, the question isn't bounded enough. One in twenty what, random things I think about? In that case it would be happening all the time. And I'm just pulling a number out of my ass anyway. The number doesn't really mean anything. My actual assessment, my gut-check, is that it seems very unlikely for that to happen as a result of pure background coincidence. It seems unlikely enough that it felt uncanny, like getting a sideways glimpse of a little light showing between the seams of the universe. Not the kind of coincidence I can easily shrug off.

Serp2, a little bit pointedly: Unlikely enough that it's pushing me into this confrontation with myself even though I didn't want to have it yet.

Serp1: We'll just have to accept that as an answer for now. What next?

[Cognitive drain, memory search result.]

Serp2: Quantify the amount more likely it should be to see this result under the mysticism hypothesis. I don't know how to put numbers to that, either, and even if I did, my ass is not well enough calibrated at the moment that my numbers would be likely to mean anything.

Serp1: Crap. Well, what do we do now then?

[Cognitive drain, memory search result.]

"Lady Rationality carries a notebook. She writes down all the evidence, not just the evidence for or against one side..."

Serp1: We have a notebook in our backpack, don't we? We can actually write this down. We don't need to actually assign the numbers right now, just make a list of all the factors that we would want to assign numbers to.

Serp2: That... sounds like a great idea actually. It's an almost completely fresh notebook, too. I could try to keep a record of all the things that register as potentially important evidence regarding whether or not to believe in mysticism, right here in this special book. As long as I capture the essential points, assigning numbers to them should be something I can do later...

And so I paused in my grocery shopping, and took out my notebook, and I wrote down the following list and notes (expanded a bit out of point form for better clarity):

(+ factor supports mysticism; - factor limits impact of support)

April 6

A Dog Named Phoenix

: + the observation followed the thought, with a delay of about 1-2 minutes, max, between them

: + the same word, Phoenix, was fully and specifically articulated in both

: + observation was preceded by and depended on an unanticipated, sudden and whimsical change of plans (I would not have met the dog if I had not paused to choose my path and decided to turn around)

: - Phoenix is not an especially unlikely name for a dog to have

: - questioning my mysticism and thinking about it a lot within the last day is something that might elicit a confirming response under the mysticism hypothesis, but, I am not permitting it to add evidential weight, because questioning mysticism and the nature of reality is not a particularly unusual state of mind for me to be in

Note: it will be harder to collect negative evidential results effectively. BEWARE POSITIVITY BIAS!

Set Requirement: must establish decent success rate when ACTUALLY LOOKING, in order to count, permit and track negative results.

By that I mean, in order for the mysticism hypothesis to earn back a status resembling or surpassing its former privilege, it isn't enough to notice surprising positive results where I hadn't been expecting to see them crop up. I will need to actually take steps in which I intentionally try to evoke it, and see it work then, while also tracking any attempts I make to evoke it that fail. Deciding what kind of success rate qualifies as "decent" is a complicated affair I will have to address later. This has already been a lot of writing, and I still have a friend to help.

However... as I put my notebook back in my bag and turned for home, I felt... proud. I think this is the right approach. I'm not being cowed or awed by my ignorance, I'm... I'm actually constructing ways of doing research on magic. And finally practicing the methods myself. Cool.

Also, I consider the phrase "making beliefs pay rent" and I find myself feeling that I have just handed this belief an eviction notice... but a conditional one. A warning to either pay up or get out, but I'm open to either one.