Depends on what value the FAI places on human flourishing in hypothetical alternate realities I guess. If it's focused on the universe it's in then there's no reason to waste half of it on paperclips. If it's trying to help out the people living in a universe where the paperclip maximizer got activated then it should cooperate. I guess a large part of that is also about whether it determines there really are parallel universes or not to be concerned about.

Bummer

Fantastic post. This is really a criminally underdiscussed issue relative to how important it is.

My one relatively minor critique is the same one I often have for Eliezer's economics-related posts: he's a little too trapped in the 40,000 foot theory level rather than how any of this works in practice. Particularly this bit here:

"Q. Yeah… I've been wondering if maybe my currency is pretty much interchangeable with government bonds, now. Maybe when I buy government bonds from the sort of people who own government bonds in the first place, they just keep my currency around and don't buy anything else with it. Maybe I need to inject the money somewhere else.

A. First, like many other problems having to do with deflation, this hypothetical problem, if it existed, would be one that you could solve with moar money. There is some amount of money creation that overflows into the hands of people who can buy real things, that gets things moving again and causes all the factories to work at capacity and people to be employed and flows through all the trades that people can make."

This problem does seem to be happening: the people who sell government bonds to the government take that money and invest it somewhere else. Selling that bond doesn't really increase their marginal propensity to spend on anything besides asset purchases very much, and thus we see the modern economy where inflation is low while all asset prices are through the roof. Eliezer is of course correct that this could eventually be solved by MOAR MONEY, in that printing enough money will eventually raise inflation and get the economy working at capacity. But there are real negative effects to pouring the vast majority of the money created into bidding asset prices up ever higher, and relying on the fraction the trickles down to the real economy to do the stimulative work.

I see the Central Bankers as having tried the "moar money" route, seen that most of that money is diverted over into asset bubbles, and concluded that printing even more money is not a good route to stimulating the economy. The big hurdle is not an unwillingness to print ungodly amounts of money if necessary, it's an unwillingness to go for the kind of unconventional "send everybody a check" or "fund fiscal stimulus" uses for the newly created money that would direct a much higher proportion of that money into the real economy.

" Part of our civilization was being, in a certain sense, stupid: there were trillion-dollar bills lying around for the taking. But they weren’t trillion-dollar bills that just anyone could walk over and pick up. "

This is a great explanation for the appeal of politics: it's about trying to take over those parts of our civilization that are displaying this kind of stupidity in order to get them to not do that so much.