Thomas C. Schelling's "Strategy of Conflict"
It's an old book, I know, and one that many of us have already read. But if you haven't, you should. If there's anything in the world that deserves to be called a martial art of rationality, this book is the closest approximation yet. Forget rationalist Judo: this is rationalist eye-gouging, rationalist gang warfare, rationalist nuclear deterrence. Techniques that let you win, but you don't want to look in the mirror afterward. Imagine you and I have been separately parachuted into an unknown mountainous area. We both have maps and radios, and we know our own positions, but don't know each other's positions. The task is to rendezvous. Normally we'd coordinate by radio and pick a suitable meeting point, but this time you got lucky. So lucky in fact that I want to strangle you: upon landing you discovered that your radio is broken. It can transmit but not receive. Two days of rock-climbing and stream-crossing later, tired and dirty, I arrive at the hill where you've been sitting all this time smugly enjoying your lack of information. And after we split the prize and cash our checks I learn that you broke the radio on purpose. Schelling's book walks you through numerous conflict situations where an unintuitive and often self-limiting move helps you win, slowly building up to the topic of nuclear deterrence between the US and the Soviets. And it's not idle speculation either: the author worked at the White House at the dawn of the Cold War and his theories eventually found wide military application in deterrence and arms control. Here's a selection of quotes to give you a flavor: the whole book is like this, except interspersed with game theory math. > The use of a professional collecting agency by a business firm for the collection of debts is a means of achieving unilateral rather than bilateral communication with its debtors and of being therefore unavailable to hear pleas or threats from the debtors. > A sufficiently severe and certain penalty on the payment
I'd normally be wary of criticizing this, because it's written by Bostrom, one of the earliest and clearest thinkers on AI risk. But I think the argument is wrong.
The argument compares "business as usual" (people living normal lives and then dying), "AI gives everyone longevity", and "AI kills everyone". But these are not the only possibilities. A neglected fourth possibility is that AI makes things much worse than just killing everyone. For example, if large numbers of people end up in inescapable servitude. I think such outcomes are actually typical in case of many near-misses at alignment, including the particular near-miss that's becoming more probable day by day: if the powerful successfully... (read more)