Self-Integrity and the Drowning Child

by Eliezer Yudkowsky7 min read24th Oct 202173 comments

263

Ethics & Moralitydath ilanParables & FablesRationality
Curated

(Excerpted from "mad investor chaos and the woman of asmodeus", about an unusually selfish dath ilani, "Keltham", who dies in a plane accident and ends up in Cheliax, a country governed by D&D!Hell.  Keltham is here remembering an incident from his childhood.)


And the Watcher told the class a parable, about an adult, coming across a child who'd somehow bypassed the various safeguards around a wilderness area, and fallen into a muddy pond, and seemed to be showing signs of drowning (for they'd already been told, then, what drowning looked like).  The water, in this parable, didn't look like it would be over their own adult heads.  But - in the parable - they'd just bought some incredibly-expensive clothing, costing dozens of their own labor-hours, and less resilient than usual, that would be ruined by the muddy water.

And the Watcher asked the class if they thought it was right to save the child, at the cost of ruining their clothing.

Everyone in there moved their hand to the 'yes' position, of course.  Except Keltham, who by this point had already decided quite clearly who he was, and who simply closed his hand into a fist, otherwise saying neither 'yes' nor 'no' to the question, defying it entirely.

The Watcher asked him to explain, and Keltham said that it seemed to him that it was okay for an adult to take an extra fifteen seconds to strip off all their super-expensive clothing and then jump in to save the child.

The Watcher invited the other children to argue with Keltham about that, which they did, though Keltham's first defense, that his utility function was what it was, had not been a friendly one, or inviting of further argument.  But they did eventually convince Keltham that, especially if you weren't sure you could call in other help or get attention or successfully drag the child's body towards help, if that child actually did drown - meaning the child's true life was at stake - then it would make sense to jump in right away, not take the extra risk of waiting another quarter-minute to strip off your clothes, and bill the child's parents' insurance for the cost.  Or at least, that was where Keltham shifted his position, in the face of that argumentative pressure.

Some kids, at that point, questioned the Watcher about this actually being a pretty good point, and why wouldn't anyone just bill the child's parents' insurance.

To which the Watcher asked them to consider hypothetically the case where insurance refused to pay out in cases like that, because it would be too easy for people to set up 'accidents' letting them bill insurances - not that this precaution had proven to be necessary in real life, of course.  But the Watcher asked them to consider the Least Convenient Possible World where insurance companies, and even parents, did need to reason like that; because there'd proven to be too many master criminals setting up 'children at risk of true death from drowning' accidents that they could apparently avert and claim bounties on.

Well, said Keltham, in that case, he was going right back to taking another fifteen seconds to strip off his super-expensive clothes, if the child didn't look like it was literally right about to drown.  And if society didn't like that, it was society's job to solve that thing with the master criminals.  Though he'd maybe modify that if they were in a possible-true-death situation, because a true life is worth a huge number of labor-hours, and that part did feel like some bit of decision theory would say that everyone would be wealthier if everyone would sacrifice small amounts of wealth to save huge amounts of somebody else's wealth, if that happened unpredictably to people, and if society was also that incompetent at setting up proper reimbursements.  Though if it was like that in real life instead of the Least Convenient Possible World, it would mean that Civilization was terrible at coordination and it was time to overthrow Governance and start over.

This time the smarter kids did not succeed in pushing Keltham away from his position, and after a few more minutes the Watcher called a halt to it, and told the assembled children that they had been brought here today to learn an important lesson from Keltham about self-integrity.

Keltham is being coherent, said the Watcher.

Keltham's decision is a valid one, given his own utility function (said the Watcher); you were wrong to try to talk him into thinking that he was making an objective error.

It's easy for you to say you'd save the child (said the Watcher) when you're not really there, when you don't actually have to make the sacrifice of what you spent so many hours laboring to obtain, and would you all please note how none of you even considered about whether or not to spend a quarter-minute stripping off your clothes, or whether to try to bill the child's parents' insurance.  Because you were too busy showing off how Moral you were, and how willing to make Sacrifices.  Maybe you would decide not to do it, if the fifteen seconds were too costly; and then, any time you spent thinking about it, would also have been costly; and in that sense it might make more sense given your own utility functions (unlike Keltham's) to rush ahead without taking the time to think, let alone the time to strip off your expensive fragile clothes.  But labor does have value, along with a child's life; and it is not incoherent or stupid for Keltham to weigh that too, especially given his own utility function - so said the Watcher.

Keltham did have enough dignity, by that point in his life, not to rub it in or say 'told you so' to the other children, as this would have distracted them from the process of updating.

The Watcher spoke on, then, about how most people have selfish and unselfish parts - not selfish and unselfish components in their utility function, but parts of themselves in some less Law-aspiring way than that.  Something with a utility function, if it values an apple 1% more than an orange, if offered a million apple-or-orange choices, will choose a million apples and zero oranges.  The division within most people into selfish and unselfish components is not like that, you cannot feed it all with unselfish choices whatever the ratio.  Not unless you are a Keeper, maybe, who has made yourself sharper and more coherent; or maybe not even then, who knows?  For (it was said in another place) it is hazardous to non-Keepers to know too much about exactly how Keepers think.

It is dangerous to believe, said the Watcher, that you get extra virtue points the more that you let your altruistic part hammer down the selfish part.  If you were older, said the Watcher, if you were more able to dissect thoughts into their parts and catalogue their effects, you would have noticed at once how this whole parable of the drowning child, was set to crush down the selfish part of you, to make it look like you would be invalid and shameful and harmful-to-others if the selfish part of you won, because, you're meant to think, people don't need expensive clothing - although somebody who's spent a lot on expensive clothing clearly has some use for it or some part of themselves that desires it quite strongly.

It is a parable calculated to set at odds two pieces of yourself (said the Watcher), and your flaw is not that you made the wrong choice between the two pieces, it was that you hammered one of those pieces down.  Even though with a bit more thought, you could have at least seen the options for being that piece of yourself too, and not too expensively.

And much more importantly (said the Watcher), you failed to understand and notice a kind of outside assault on your internal integrity, you did not notice how this parable was setting up two pieces of yourself at odds, so that you could not be both at once, and arranging for one of them to hammer down the other in a way that would leave it feeling small and injured and unable to speak in its own defense.

"If I'd actually wanted you to twist yourselves up and burn yourselves out around this," said the Watcher, "I could have designed an adversarial lecture that would have driven everybody in this room halfway crazy - except for Keltham.  He's not just immune because he's an agent with a slightly different utility function, he's immune because he instinctively doesn't switch off a kind of self-integrity that everyone else in this class needs to learn to not switch off so easily."

263

73 comments, sorted by Highlighting new comments since Today at 11:32 PM
New Comment
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

But how exactly do you do this without hammering down on the part that hammers down on parts? Because the part that hammers down on parts really has a lot to offer, too, especially when it notices that one part is way out of control and hogging the microphone, or when it sees that one part is operating outside of the domain in which its wisdom is applicable.

(Your last paragraph seems to read "and now, dear audience, please see that the REAL problem is such-and-such a part, namely the part that hammers down on parts, and you may now proceed to hammer down on this part at will!")

You can apply the lesson to that conclusion as well, avoid hammering down on the part that hammers down on parts. The point is not to belittle it, but to reform it so that it's less brutishly violent and gullible, so that the parts of mind it gardens and lives among can grow healthy together, even as it judiciously prunes the weeds.

2toonalfrink19dYou cannot truly dissolve an urge by creating another one. Now there are 2 urges at odds with one another, using precious cognitive resources while not achieving anything. You can only dissolve it by becoming conscious of it and seeing clearly that it is not helping. Perhaps internal double crux would be a tool for this. I'd expect meditation to help, too.

I think the key to the drowning child parable is the ability of others to judge you. I can't judge you for not donating a huge portion of your income to charity, because then you'll bring up the fact that I don't donate a huge portion of my own income to charity. Sure, there are people who do donate that much, but they are few enough that it is still socially safe to not donate. But I can judge you for not saving the child, because you can't challenge me for not saving them - I was not there. This means that not saving the child poses a risk to your social status, which can greatly tilt the utility balance in favor of saving them.

0weightt an1moExactly

I agree with and appreciate the broad point. I'll pick on one detail because I think it matters.

this whole parable of the drowning child, was set to crush down the selfish part of you, to make it look like you would be invalid and shameful and harmful-to-others if the selfish part of you won [...]

It is a parable calculated to set at odds two pieces of yourself... arranging for one of them to hammer down the other in a way that would leave it feeling small and injured and unable to speak in its own defense.

This seems uncharitable? Singer's thought experiment may have had the above effects, but my impression's always been that it was calculated largely to help people recognize our impartially altruistic parts—parts of us that in practice seem to get hammered down, obliterated, and forgotten far more often than our self-focused parts (consider e.g. how many people do approximately nothing for strangers vs. how many people do approximately nothing for themselves).

So part of me worries that "the drowning child thought experiment is a calculated assault on your personal integrity!" is not just mistaken but yet another hammer by which people will kick down their own altruistic parts—the parts of us that protect those who are small and injured and unable to speak in their own defense.

Regarding the direct example 

I feel like it's self-subverting. There's an old canard about https://www.watersafetymagazine.com/drowning-doesnt-look-like-drowning/ Given how staggeringly disproportionate the utility losses are in this scenario I think even a 1% chance of my assumption that 'I have 15 seconds to undress' would lead to death means I should act immediately.

In general when thinking about superfast reflex decisions vs thought out decisions: Obey the reflex unless your ability to estimate the probabilities involved has really low margins of error. My gut says X but my slow, super weak priors-that-have-never-been-adjusted-by-real-world-experience-about-this-first-time-in-my-life-situation say Y... Yeah just go with X. Reflect on the outcome later and maybe come up with a Z that should have been the gut/reflex response.

There's an old video game Starcraft 2 advice from Day9 that's surprisingly applicable in life: Plan your game before the game, in game follow the plan even if it seems like it's failing, after the game review and adjust your plan. Never plan during the game, speed is of the essence and the loss of micro and macro speed will cost you more than a bad plan ... (read more)

4Slider1moThe starcraft advice is really dependent on the problem being speed sensitive. Law of equal and opposite advice applies when that structure is not present. For example in a real army that somebody is infact charge of the group is important enough that rather than jumping into everything people will call in for confirmation/order to do certain stuff. For example peacekeeping during demonstration one would want to quell a rebellion if one is about to start but shooting back when people throw rocks could make for an unneccesary bloodpath. And that call can not be made in advance as there might not be information available.
2spkoc1moI agree, sort of. I'd argue that in the military example there is already a plan that includes consultation phases on purpose. The rules of engagement explicitly require a slow step. I don't know if this applies in genuinely surprising situations. A sort of known unknown vs unknown unknown distinction. I guess you can have a meta policy of always pausing ANY time something unexpected happens, but I feel like that's... hard to live(or even survive) with? Speeding car coming towards me or a kid in the road. Just act, no time to think. In fairness, this is why you prepare and preplan for likely emergency events you might encounter in life.
2wslafleur1moThese definitions of shame and guilt strike me as inherently dysfunctional because they seem to rely on direct external reference, rather than referencing some sort of internal 'Ideal Observer' which - in a healthy individual - should presumably be an amalgamate intuition, built on top of many disparate considerations and life experience.
2spkoc1moThe internal Ideal Observer is the amalgamated averaged out result of interactions with the world and other people alive and dead. Human beings don't come from the orangutan branch of the primate tree, we are fundamentally biologically not solitary creatures. Our ecological niche depends on our ability to coordinate at a scale comparable to ants, but while maintaining the individual decision making autonomy of mammals. We're not a hive mind and we're not atomized individuals. We do and should constantly be balancing ourselves based on the feedback we get from physical reality and the social reality we live in. Is the Ideal Observer the thing doing that balancing? Sure. But then it becomes a very reduced sort of entity, kinda like science keeps reducing the space where the god of the gaps can hide. There's an inner utility function spitting out pleasure and pain based on stimuli, but I wouldn't call that me, there's a bit more flesh around me than just that nugget of calculation.

At a pond, my niece was in a child floaty, reached too far and flipped over into the water. I slammed my half-eaten sandwich on my brother's chest, hoping he would grab it and ran into the water and saved her. 

She was fine and I got to finish my sandwich.

I unfortunately have very little of substance to add, but a strong upvote was not quite enough.

There is something in here of Iron Hufflepuff, and I'm exceedingly grateful to Eliezer for dignifying and validating it so unequivocally in this meta-parable.  I expect to link to this fairly frequently over the next decade.

Trying to summarize for those of us not fond of long-winded parables.

  • A single moral agent is a bad model of a human,
  • Multiple agents with individual utilities/rules/virtues is a more accurate model.
  • It's useful to be aware of this, because Tarski, or else your actions don't match your expectations.

I worry that "parts as people" or even "parts as animals" models are putting people on the wrong path to self-integrity, and I did my best to edit this whole metaparable to try to avoid suggesting that any point.

Oops... I guess I misunderstood what you meant by "two pieces of yourself".

Anyway, I really like the part 

you failed to understand and notice a kind of outside assault on your internal integrity, you did not notice how this parable was setting up two pieces of yourself at odds, so that you could not be both at once, and arranging for one of them to hammer down the other in a way that would leave it feeling small and injured and unable to speak in its own defense

because it attends to the feelings and not just to the logic: "hammer down the other in a way that would leave it feeling small and injured".

I could have designed an adversarial lecture that would have driven everybody in this room halfway crazy - except for Keltham

I... would love to see one of those, unless you consider it an infohazard/Shiri's scissor.

> I could have designed an adversarial lecture that would have driven everybody in this room halfway crazy - except for Keltham

I... would love to see one of those, unless you consider it an infohazard/Shiri's scissor.

I think this might just mean using the drowning child argument to convince the students they should be acting selflessly all the time, donating all their money above minimal subsistence, etc. 

8Eliezer Yudkowsky1moIf the people on the other side of the argument ended up behaving coherently, rather than twisting themselves into knots and burning themselves out as their inner gears ground against themselves in unresolvable circles, it wouldn't be much of an adversarial lecture, would it?
3ADifferentAnonymous1moKnot-twisting is indeed the outcome I was imagining. (Your translation spell might be handling the words "convince" and "should" optimistically... maybe try them with the scare quotes?)

Regarding pieces of oneself, consider the ideas of IFS (internal family systems). "Parts" can be said to attenuate to different concerns and if one can distract from others then an opportunity to maximize utility across dimensions may be missed. One might also suggest that attenuation to only one concern over time can result in a slight movement towards disintegration as a result of increasingly strong feelings about "ignored" concerns. Integration or alignment, with every part joining a cooperative council is often considered a goal and personification can assist some in more peaceably achieving that. I personally found the suggestion to personify felt weird and false.

I personally found the suggestion to personify felt weird and false.

I second this.

6Rob Bensinger1moI'd very much love to hear more about this. (Including from others, both for and against.)
1adamzerner1moSame.

Humans don't swim very well wearing lots of clothing. Take off your suit before going into the water.

I actually keep thinking this in the back of my own mind every time I run into this parable, so thank you for stating it out loud.  (I expect if a child brought it up, the Watcher credited them for noticing further consequences, then asked to assume the Less Convenient Possible World where this is not the case.)

4Richard_Kennaway1moAn adult may wade where a child would drown. "The water, in this parable, didn't look like it would be over their own adult heads." (I don't know if Eliezer added that after you brought this up.) (And "heads" should be singular. The class are children, and there is one hypothetical adult.)
2Slider1moIsn't this an additional example of "practicality module" having integrity and not caving in to the "look I am rushing in so valorantly" impulse?

An interesting difference between the drowning child situation and the "could donate to effective charity to save children's lives" situation is that the person who happens to be walking by that pond has a non-transferable opportunity to save a child's life for $500 (or whatever the cost of the clothes are, plus some time cost, and the inconvenience of getting wet and muddy). In the case of effective charity, even if one declines to donate, other people will still have the same opportunity. In the case of the drowning child, the fact that you are the only one who can act makes jumping in to save the child somehow seem more urgent. If you don't save the child, then you'd be somehow "wasting" a valuable opportunity. For a mostly selfish person who values all lives other than their own at less than $500, the opportunity would be valuable to others but useless for themselves.

If the going rate is $1000 to save a child through effective charity, then a mostly altruistic person would be willing to pay a mostly selfish person $600 to compensate for the costs of their clothes. There would have to be some "honour" involved, since the selfish person couldn't exactly unsave the child after the fact. If they could make the deal work anyway, then the mostly selfish person would have succeeded in selling non-transferable opportunity for $100, and it would be worthwhile for them to save the drowning child.

I would like to distinguish between money burnt and money transferred. The $500 are burnt, assuming you handpicked homegrown cotton and knitted it into clothes. The $1000 are also burnt, assuming that nobody extracts rent on saving children. The $600 are merely transferred, and so the altruist may be willing to pay more even than $1000, if he expects the profits spent in ways he still moderately endorses.

Curated. To generalize, as the stakes continue to seem high ("most important century"-level high), it's easy to feel an immense obligation to act and to give it all up for the sake of the future. This meta-parable reminds us that humans aren't made solely of parts that give everything up, and that it's a matter of self-integrity to not do so.

My objection lies in the second part of the drowning child parable. The part where someone geographically distant is considered identical to the child in front of me, and money is considered identical to the actions of saving. It's some sort of physics being the same everywhere intuition being inappropriately applied. Of course distance in time, space, or inference create uncertainty. Of course uncertainty reduces expected value and possibly even brings the sign of the action into question if the expected variance is high enough.

A literal drowning child puts a limit on your commitment. Save this child, and your duty is discharged. When we apply this moral intuition to all the other issues in the world, our individual obligation suddenly becomes all-consuming.

Furthermore, a literal drowning child is an accident. It represents a drastic exception to the normal outcomes of your society. Your saving action is plugging a hole in a basically sound system. Do our moral intuitions stem from a consequentialist goal to save all lives that can be saved? Or do they stem from an obligation to maintain a healthy, caring, and more-or-less self-sufficient society?

To me, the best interpretation of the drowning child parable extended to a global level is that it gives me a sense of moral glee. Holy smokes! The mere act of donating money, or of doing direct work in a powerful cause for good, can save lives just the way that a more conventional heroic action can! How cool!

I'd import Eliezer's concept of a "cheerful price," but in reverse. Instead of being paid in money to cheerfully take an action I'd otherwise rather not do, I am being paid in lives saved to cheerfully give some money I'd otherwise rather not donate. A life saved for a mere $10,000? A bargain at twice the price!

8lsusr1moThis is a good point. I never noticed it before.
4Rob Bensinger1moQuoth AllAmericanBreakfast: If the question is just "What's the ultimate psychological cause of my moral intuitions in these cases?", then 🤷. If the question is "Are we just faking caring about saving other lives, when really we don't care about other human beings' welfare, autonomy, or survival at all?", then I feel confident saying 'Nah'. I get a sense from this question (and from Romeo's content) of 'correctly noticing that EA has made some serious missteps here, but then swinging the pendulum too far in the other direction'. Or maybe it just feels to me like this is giving surprisingly wrong/incomplete pictures of most people's motivations. Quoth Romeo: For most people I suspect the demandingness is the crux, rather than the uncertainty. I think they'd resist the argument even if the local 'save a drowning child' intervention seemed more uncertain than the GiveWell-ish intervention. (Partly because of a 'don't let yourself get mugged' instinct, partly because of the integrity/parts thing, and partly because of scope insensitivity [https://www.lesswrong.com/posts/2ftJ38y9SRBCBsCzy/scope-insensitivity].) I also think there's a big factor of 'I just don't care as much about people far away from me, their inner lives feel less salient to me' and/or 'I won't be held similarly blameworthy if I ignore large amounts of distant suffering as if I ignore even small amounts of nearby suffering, because the people who could socially punish me are also located near me'. We can consider a 2x2 matrix: NearFarUndemandingDrowningChildDrowningChildPhoneCall?DemandingInaWarZone?Agains tMalariaFoundation * Undemanding + Near: Drowning child. There's a cost to saving the child, but because this scenario is rare, one-off, local, and not too costly, almost everyone (pace Keltham) is happy to endorse saving the child here. * Undemanding + Far: The same dilemma, except you're (say) missing a medium-importance business call (with cost equivalent to one fancy suit) in
2romeostevensit1moI also think of the demandingness as generating an additional uncertainty term in the straussian sense.
1Idan Arye1moCould you clarify what you mean by "demandingness"? Because according to my understanding the drowning child should be more demanding than donating to AMF because the situation demands that you sacrifice to rescue them, unlike AMF that does not place any specific demands on you personally. So I assume you mean something else?
3Rob Bensinger1moThe point of the original drowning child argument was to argue for 'give basically everything you have to help people in dire need in the developing world'. So the point of the original argument was to move from * A relatively Undemanding + Near scenario: You encounter a child drowning in the real world. This is relatively undemanding because it's a rare, once-off event that only costs you the shirt on your back plus a few minutes of your time. You aren't risking your life, giving away all your wealth, spending your whole life working on the problem, etc. to * A relatively Demanding + Far scenario. It doesn't have to be AMF or GiveDirectly, but I use those as examples. (Also, obviously, you can give to those orgs without endorsing 'give everything you have'. They're just stand-ins here.)

Equally importantly IMO, it argues for transfer from a context where the effect of your actions is directly perceptionally obvious to one where it is unclear and filters through political structures (e.g., aid organizations and what they choose to do and to communicate; any governments they might be interacting with; any other players on the ground in the distant country) that will be hard to model accurately.

2Rob Bensinger1moMy guess is that this has a relatively small effect on most people's moral intuitions (though maybe it should have a larger effect -- I don't think I grok the implicit concern here). I'd be curious if there's research bearing on this, and on the other speculations I tossed out there. (Or maybe Spencer or someone can go test it.)
1Idan Arye1moI see. So essentially demandingness is not about how strong the demand is but about how much is being demanded?
7jimrandomh1moI think distance is a good correlate for whether insurance will pay, figuratively speaking. Not because there is literally an insurance company that will pay money, but because some fraction of people whose life has been saved, or whose child's life has been saved, will think of themselves as owing a debt.
7Eli Tyre1moYou're referring to the original Peter Singer essay, not to this one, yes?
3romeostevensit1moCorrect
6Said Achmiz1moI agree with you, but this seems to very much not be the point of this parable.
3Richard_Ngo1moIndeed, it seems like Romeo may be letting (one) altruistic part get hammered down by his other parts.
2Said Achmiz1moI do not think that’s the problem here; rather it’s just a case of focusing on the details of the example, rather than on the concept that it’s being used as an example for.
34thWayWastrel1moOut of curiosity, does all of the difference between the value of a child drowning in front of you and a child drowning far away come from uncertainty?
5TekhneMakre1moThere's also some coordination thing that's muddled in here. Like, "everyone protect their neighbor" is more efficient than "everyone seek out the maximal marginal use of their dollar to save a life". This doesn't necessarily cash out--indeed, why *not* seek out the maximal marginal life-saving? For one thing, the seeking is a cost; it can also be a long-term benefit if it "adds up", accumulating evidence and understanding, but that's a more specific kind of seeking (and you might even harm this project if e.g. you think you should lie to direct donations). For another thing, you're seriously eliding the possibility of, for example, helping to create the conditions under which malaria-ridden areas could produce their own mosquito nets, by (1) not trusting that people could take care of themselves, (2) having high time-preference for saving lives. For a third thing, it's treating, I think maybe inappropriately, everyone as being in a marketplace, and eliding that we (humans, minds) are in some sense (though not close to entirely) "the same agent". So if I pay you low wages to really inefficiently save a life, maybe that was a good marginal use of my dollar, but concretely what happened is that you did a bunch of labor for little value. We might hope that eventually this process equilibriates to people paying for what they want and therefore getting it, but still, we can at least notice that it's very far from how we would act if we were one agent with many actuators.
2romeostevensit1moIn a sense, since other differences might be unknown?

It is dangerous to believe, said the Watcher, that you get extra virtue points the more that you let your altruistic part hammer down the selfish part.

Dangerous? In what way?

not selfish and unselfish components in their utility function, but parts of themselves in some less Law-aspiring way than that

Utility functions don't model all agents; we should look at a larger space. I expect it to better model not just a human but also a council of humans or a multiverse of acausal traders. I expect this also to say how an AGI should handle uncertainty about preferences.

There should be a natural way to aggregate a distribution of agents into an agent, obeying the obvious law that an arbitrarily deeply nested distribution comes out the ... (read more)

5Nisan1moAh, great! To fill in some of the details: * Given agents a1,a2 and numbers p1,p2 such that p1+p2=1, there is an aggregate agent called p1a1+p2a2 which means "agents a1 and a2 acting together as a group, in which the relative power of a1 versus a2 is the ratio of p1 to p2". The group does not make decisions by combining their utility functions, but instead by negotiating or fighting or something. * Aggregation should be associative, so 13a1+23(12a2+12a3)=13a1+13a2+13a3=23(12 a1+12a2)+13a3. * If you spell out all the associativity relations, you'll find that aggregation of agents is an algebra over the operad of topological simplices. (See Example 2 https://arxiv.org/abs/2107.09581 [https://arxiv.org/abs/2107.09581].) * Of course we still have the old VNM-rational utility-maximizing agents. But now we also have aggregates of such agents, which are "less Law-aspiring" than their parts. * In order to specify the behavior of an aggregate, we might need more data than the component agents ai and their relative power pi. In that case we'd use some other operad.

TLDR: if we model a human as a collection of sub-agents rather than single agent, how do we make normative claims about which sub-agents should or shouldn't hammer down others? There's no over-arching set of goals to evaluate against, and each sub-agent always wants to hammer down all the others.

If I'm interpreting things right, I think I agree with the descriptive claims here, but tentatively disagree with the normative ones. I agree that modeling humans as single agents is inaccurate, and a multi-agent model of some sort is better. I also agree that the ... (read more)

In case anyone else, like me, followed the link near the start of OP to the story from which this is excerpted, and is wondering whether (having had a bunch of updates on 2021-10-24 but none since) it's likely to be dead: I had a look at its pattern of updates and it's very bursty, with gaps on the order of 1-3 weeks between bursts, so the current ~1w of no updates is not strong evidence against there being more future updates to come.

(It seems like the majority of fictional works posted online end up being abandoned, and I generally prefer not to read things of non-negligible length that stop in mid

2MondSemmel1moI agree that most fiction ends up unfinished. This is the likely fate of any story, including anything posted on Glowfic. Even professionally published fiction is not safe from this, due to the propensity to write books as trilogies or something; and sometimes authors die (Berserk) or commit a crime (Act-Age). That said, I am flabbergasted at the notion that you'd check a fiction, see that it was last updated <7 days ago, and immediately bring the hypothesis that "this is likely to be dead" to conscious attention, even if you then reject it. I think this attitude sets bad incentives for authors (and translators etc.; this is a common reaction to manga scanlators, too), and makes it more likely that works will indeed not be finished. So I want to strongly push back against this, and say instead: Yes, most fiction will not get finished. It's the responsibility of the reader to take this risk into account, when they decide whether to start reading something. (Especially when it comes to free fiction - the dynamics for Patreon-supported stuff etc. are imo different. And especially especially when it comes to something like Glowfic, which is more like people publically roleplaying in text, than like webfiction where authors often commit to a regular update schedule.) Do not push for updates, nor support a culture which does.
4gjm1moWell, what I actually saw was that it was updated many times over at most a couple of days and then nothing for about a week. I hadn't, at that point, looked at the times of the earliest postings and noticed that they were months earlier. So what I saw at that point -- and what I thought others might likewise see -- was a flurry of activity followed by a gap. And that seemed like possible evidence of dead-ness, which is why I checked further and decided it wasn't. Your last paragraph seems to be reading things into what I wrote that I'm pretty sure I never put there. I completely agree that if a reader prefers to avoid reading things that get abandoned in the middle, it's their responsibility to look. That's what I did. I found (1) reason for initial suspicion that this might be such a work and then (2) excellent reason to drop that suspicion, so I said so. Neither did I push for updates nor suggest that anyone else should. (In case anyone else took what I said as some sort of encouragement to do that: Do not do that! It's rude!)
2MondSemmel1moFair enough! Insofar as I read something into your original comment that wasn't there, I think it was due to my interpretation of the language? When I hear something described as "dead" or "abandoned", that sounds like assigning blame to the author, as if they didn't fulfill a responsibility or duty; but I understand that this interpretation was not intended. To be clear, I would still bet at 2:1 odds that the story won't get finished, simply based on base rates for web fiction (possibly the base rate for unfinished Glowfics is even higher?). All the while stressing that I don't mean that as blame, and that it's entirely fine for anyone to decide that a free fiction project is no longer worth their opportunity cost.

This applies to integrity of a false persona just as well, a separate issue from fitting an agentic persona (that gets decision making privileges, but not self-ratification privileges) to a human. Deciding quite clearly who you are doesn't seem possible without a million years of reflection and radical cognitive enhancement. The other option is rewriting who you are, begging the question, a more serious failure of integrity (of a different kind) whose salience distracts from the point of the dath ilani lesson.

I prefer hypocrisy to cruelty. 

More gennerally I think this just misses the point of drowning child.  The argument is not that you have this set of preferences and therefore you save the child, the argument is that luxury items are not of equal moral worth to the life of a child.  This can be made consistent with taking off your suit first if think the delay has a sufficiently small probability of leading to the death of child and you think the death of a child and the expensive suit are comparable. 

3Duncan_Sabien25dEr. Trying to preemptively frame it as "cruelty" is somewhat refusing to engage with the very question at hand.

In dath ilan, it is virtuous to write more stories about dath ilan.

I reject the parable/dilemma for another reason: in the majority of cases, I don't think it's ethical to spend so much money on a suit that you would legitimately hesitate to save a drowning child if it put the suit at risk?

If you're so rich that you can buy tailor-made suits, then sure, go save the child and buy another suit. If you're not... then why are you buying super-expensive tailor-made suits? I see extremely few situations where keeping the ability to play status games slightly better would be worth more than saving a child's life.

(And yes, there'... (read more)

6JBlack1moFrom previous posts about this setting, the background assumption is that the child almost certainly won't permanently die if it takes 15 seconds longer to reach them. This is not Earth. Even if they die, their body should be recoverable before their brain degrades too badly for vitrification and future revival. It is also stated that the primary character here is far more selfish than usual. However even on Earth, we do accept economic reasons for delaying rescue by even a lot more than 15 seconds. We don't pay enough lifeguards to patrol near every swimmer, for example, which means that when they spot a swimmer in distress it takes at least 15 more seconds to reach them. In nearly every city, a single extra ambulance team could reduce average response time to medical emergencies by a great deal more than 15 seconds. There doesn't seem to be any great ethical outcry about this, though there are sometimes newspaper articles when the delays go past a few extra hours. What's more these are typically shared, public expenses (via insurance if nothing else). One of the major questions addressed in the post is whether the extra cost should be borne by the rescuer alone. Is that ethically relevant, or is it just an economic question of incentives?
2Olivier Faure1moSure, whatever. Honestly, that answer makes me want to engage with the article even less. If the idea is that you're supposed to know about an entire fanfiction-of-a-fanfiction canon to talk about this thought experiment, then I don't see what it's doing in the Curated feed.
2aphyer1moIf you think luxury spending is inherently immoral, I think you're going to end up in the same position as Peter Singer re. the obligation to give away almost all of your income.

The Watcher spoke on, then, about how most people have selfish and unselfish parts—not selfish and unselfish components in their utility function, but parts of themselves in some less Law-aspiring way than that.

I guess it's appropriate that children there learn about utility functions before learning about multiplication.

Perhaps the parable could have been circumvented entirely by never teaching the children that such a thing as a “utility function” existed in the first place. I was mildly surprised to learn that the dath ilani used the concept at all, rather than speaking of preferences directly. There are very few conversations about relative preference that are improved by introducing the phrase “utility function.”

Utility functions are very useful for solving decision problems with simple objectives. Human preference is not one of these, but we can often fit a utility function that approximately captures it in a particular situation, which is useful for computing informed suggestions for decisions. The model of one's preference that informs fitting of utility functions to it for use in contexts of particular decision problems could also be called a model of one's utility function, but that terminology would be misleading.

The error is forgetting that on human level, all utility functions you can work with are hopelessly simplified approximations, maps of some notion of actual preference, and even an understanding of all these maps considered altogether is a hopelessly simplified approximation, not preference itself. It's not even useful to postulate that preference is a utility function, as this is not the form that is visible in practice when drawing its maps. Still, having maps for a thing clarifies what it is, better than not having any maps at all, and better yet when maps stop getting confused for the thing itself.

3moridinamael1moI thought I agreed but upon rereading your comment I am no longer sure. As you say, the notion of a utility function implies a consistent mapping between world states and utility valuations, which is something that humans do not do in practice, and cannot do even in principle because of computational limits. But I am not sure I follow the very last bit. Surely the best map of the dath ilan parable is just a matrix, or table, describing all the possible outcomes, with degrees of distinction provided to whatever level of detail the subject considers relevant. This, I think, is the most practical and useful amount of compression. Compress further, into a “utility function”, and you now have the equivalent of a street map that includes only topology but without street names, if you’ll forgive the metaphor. Further, if we aren’t at any point multiplying utilities by probabilities in this thought experiment, one has to ask why you would even want utilities in the first place, rather than simply ranking the outcomes in preference order and picking the best one.
7cousin_it1moIt's more subtle than that. Utility functions, by design, encode preferences that are consistent over lotteries (immune to Allais paradox), not just pure outcomes. Or equivalently, they make you say not only that you prefer pure outcome A to pure outcome B, but also by how much. That "by how much" must obey some constraints motivated by probability theory, and the simplest way to summarize them is to say each outcome has a numeric utility.