I get sick of people saying things that imply that rationality has no practical, tangible benefit (e.g. "I moved to the Bay and am no better off" etc). Lots of discussion about this topic talks about physical fitness, career success, or investing. But since this is my shortform and I can say whatever I want, I want to talk about a concept that I've personally found helpful: the idea of fire alarms (as talked about in There’s No Fire Alarm for Artificial General Intelligence and Sunset at Noon), which is sort of like just another concept handle for noticing confusion.

When I was eleven, my family spent Thanksgiving at my grandparents' house. That weekend, as we were getting ready to make the several-hour drive back home, the adults were doing odd jobs around the house and my sister and I were hanging out with our cousin in the den, maybe watching a movie or something. At some point, my cousin looked up and asked, "Is someone screaming?" My priors on someone screaming were extremely low, and I didn't hear what she heard, so I said, "Nah, it's just a machine," and turned back to what I was doing. Turned out it was someone screaming, and my decision to ignore that possibility could very well have meant the difference between my dad living and dying. (The details aren't important, but for anyone who's worried, he lived.)

Another anecdote. In late 2017, shortly after the release of both There’s No Fire Alarm for Artificial General Intelligence and Sunset at Noon, I was standing with Habryka in a dimly lit and dusty room, setting up A/V for a show. In a moment of stillness, Habryka looked over and said, "Is that smoke?" Due to the poor lighting and my low prior on things randomly catching on fire, my knee-jerk response was still "Nah." But luckily Habryka wasn't so dismissive, and he went over and unplugged the definitely-actually-smoking cord before a proper fire could start. I've similarly witnessed the entire LessWrong team insisting on investigating every time they smell smoke (even though every time it's just been someone burning something on the stove or in the oven). 

I think there's an implicit expected value calculation in this, where it might be slightly inconvenient to investigate every time you smell smoke or hear a scream, but sometimes - and perhaps not even that rarely - the payoff for doing so is preventing a structure fire or saving a life. I've spent years chastising myself for my dismissiveness back when I was eleven, and I think that my contact with rationality has made me into the sort of person who's more intentional about investigating warning signs, and therefore much less likely to let my dad die just because it seemed unlikely to me that someone was screaming.

mingyuan's Shortform

by mingyuan 21st Nov 201912 comments