Rationality Quotes September 2013

Vaniver

Another month has passed and here is a new rationality quotes thread. The usual rules are:

Please post all quotes separately, so that they can be upvoted or downvoted separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)
Do not quote yourself.
Do not quote from Less Wrong itself, HPMoR, Eliezer Yudkowsky, or Robin Hanson. If you'd like to revive an old quote from one of those sources, please do so here.
No more than 5 quotes per person per monthly thread, please.

Another month has passed and here is a new rationality quotes thread. The usual rules are:

Please post all quotes separately, so that they can be upvoted or downvoted separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)
Do not quote yourself.
Do not quote from Less Wrong itself, HPMoR, Eliezer Yudkowsky, or Robin Hanson. If you'd like to revive an old quote from one of those sources, please do so here.
No more than 5 quotes per person per monthly thread, please.

Don't ask what they think. Ask what they do.

My rule has to do with paradigm shifts—yes, I do believe in them. I've been through a few myself. It is useful if you want to be the first on your block to know that the shift has taken place. I formulated the rule in 1974. I was visiting the Stanford Linear Accelerator Center (SLAC) for a weeks to give a couple of seminars on particle physics. The subject was QCD. It doesn't matter what this stands for. The point is that it was a new theory of sub-nuclear particles and it was absolutely clear that it was the right theory. There was no critical experiment but the place was littered with smoking guns. Anyway, at the end of my first lecture I took a poll of the audience. "What probability would you assign to the proposition 'QCD is the right theory of hadrons.'?" My socks were knocked off by the answers. They ranged from .01 percent to 5 percent. As I said, by this time it was a clear no-brainer. The answer should have been close to 100 percent. The next day I gave my second seminar and took another poll. "What are you working on?" was the question. Answers: QCD, QCD, QCD, QCD, QCD,........ Everyone was working on QCD. That's when I learned to ask "What are you doing?" instead of "what do you think?"

I saw exactly the same phenomenon more recently when I was working on black holes. This time it was after a string theory seminar, I think in Santa Barbara. I asked the audience to vote whether they agreed with me and Gerard 't Hooft or if they thought Hawking’s ideas were correct. This time I got a 50-50 response. By this time I knew what was going on so I wasn't so surprised. Anyway I later asked if anyone was working on Hawking's theory of information loss. Not a single hand went up. Don't ask what they think. Ask what they do.

-Leonard Susskind, Susskind's Rule of Thumb

Great quote.

Unfortunately, we find ourselves in a world where the world's policy-makers don't just profess that AGI safety isn't a pressing issue, they also aren't taking any action on AGI safety. Even generally sharp people like Bryan Caplan give disappointingly lame reasons for not caring. :(

8Eliezer Yudkowsky13y

Hm. A generalized phenomenon of overwhelming physicist underconfidence could account for a reasonable amount of the QM affair.

24AndHisHorse13y

Not necessarily a great metric; working on the second-most-probable theory can be the best rational decision if the expected value of working on the most probable theory is lower due to greater cost or lower reward.