In his talk AI Alignment: Why It’s Hard, and Where to Start Eliezer Yudkowsky gives "an interesting experiment in cognitive psychology" about a hospital administrator who has to decide whether to spend $1M on a sick child or "general hospital salaries, upkeep, administration, and so on". Yudkowsky mentions that

"A lot of the subjects in the cognitive psychology experiment became very angry and wanted to punish the administrator for even thinking about the question."

Yudkowsky thinks this is ironic, because they "are prohibiting a priori efficiently using money to save lives".

I couldn't find the experiment in question and Yudkowsky gives no reference to it. But let's assume that these people wanted to spend the $1M on the sick child and found the thought of doing anything else repulsive and this was the reason for their anger.

The answer to this hypothetical hospital administrator's question seemed blindingly obvious to them, because in their mind the life of a child should always take precedence over "general hospital salaries, upkeep, administration, and so on".

They seem intent on saving this hypothetical child's life. But if there are no other lives at stake, how is jumping to save the $1M kid's life "prohibiting a priori efficiently using money to save lives"?

Because the case presented here is a trolley case, at least in Yudkowsky's book. He doesn't expressly say so, but it is the only explanation for his reaction. And this is the closest he gets to it:

But if you cannot possibly rearrange the money that you spent to save more lives and you have limited money, then your behavior must be consistent with a particular dollar value on human life.

So it's safe to assume that he thinks the general hospital salaries, upkeep etc. may pay for other patients lives, perhaps by ensuring the mere existence of the hospital. This assumption about the real world seems reasonable. But take note that it wasn't part of the initial question as Yudkowsky presented it. Still he uses this assumption to create a trolley case, where people have to decide between the livelihood of this one kid and other unknown patients.

Now the funny thing about trolley cases is that they absolutely do no allow to make any assumptions about the real world. They are constructed to force one of two outcomes. If the classic trolley case happened in the real world you might just yell at those stupid people to get off the tracks, or place a stone on them to derail the train, or stop it in some other way. But this is not allowed. What people who construct trolley cases generally want is to force an inhumane, calculating response that weighs the life of one person against many. Imaginative solutions that might save everyone and the day aren't called for. Not least because they would reveal who's the one who can only think of inhumane, calculating responses in this game.

Yudkowsky makes it clear that he isn't this type of person:

By which I mean, not that you think that larger amounts of money are more important than human lives

And yet he just posed a veiled trolley case and made fun of people for jumping to a solution who didn't even know of its hidden condition. They were supposed to assume that the hospital wouldn't be able to save other lives for the money they spent on that one kid. But they were not supposed to assume that the hospital might secure additional funding by the state or through a campaign or have a financial backup plan.

A humane solution was never intended. Making fun of people for the attempt was.

New to LessWrong?

New Comment
2 comments, sorted by Click to highlight new comments since: Today at 12:50 AM
I couldn't find the experiment in question and Yudkowsky gives no reference to it. But let's assume that these people wanted to spend the $1M on the sick child and found the thought of doing anything else repulsive and this was the reason for their anger.
The answer to this hypothetical hospital administrator's question seemed blindingly obvious to them, because in their mind the life of a child should always take precedence over "general hospital salaries, upkeep, administration, and so on".

+1 for pointing out that differences in framing a question matter and emergency care/preventative care framings make cause people to make different assumptions about the questions

They were supposed to assume that the hospital wouldn't be able to save other lives for the money they spent on that one kid. But they were not supposed to assume that the hospital might secure additional funding by the state or through a campaign or have a financial backup plan.

-1 for not noticing that securing additional fnding or running a campaign also cost money and time (which is paid for). In real life, we do make trade-offs between saving more lives and less lives. If we could just save them all right now, the world would be a much better place. Figuring out how to make trade-offs in helping people is something called triage, and it's a medical concept that people in the medical community should be very familiar with.

A humane solution was never intended. Making fun of people for the attempt was.

-3 for claiming that Eliezer was speaking in bad-faith based on an analysis of a paraphrasing of an unsourced study mentioned as part of a much longer article that focused on a whole bunch of other stuff

People don't get angry based on people asking them questions with clear obvious answers. If someone gives you a question with a clear and obvious answer, you usually just give them the clear and obvious answer.

People don't get angry when they get asked to answer "what's 2+2?"

But take note that it wasn't part of the initial question as Yudkowsky presented it.

The effect of hospital upkeep on saved lives isn't a known known and it's not explicit in the question of how high it is. That doesn't imply that a person dealing with the question should assume it to be zero. Decision making usually happens with incomplete information.