Not sure I fully understand this comment, but I think it is similar to option 4 or 6?
 

Also you shouldn't claim to put $Googolpex utility on anything until you're at least $\Omega \left(log\right(googolplex\left)\right)$ seconds old.

Why is seconds the relevant unit of measure here?

Good point, this combines the iteratability justification for EV plus the fact that we have finite resources with which to bet. But doesn't this break down if you are unsure how much wealth you have (particularly if the "wealth" being gambled is non-monetary, for example years of life)? Suppose the devil comes to you and says "if you take my bet you can live out your full lifespan, but there will be a 1 in 1 million chance I will send you to Hell at the end for 100 billion years. If you refuse, you will cease to exist right now." Well, the wealth you are gambling with is years of life, but it's unclear how many you have to gamble with. We could use whatever our expected number of years is (conditional on taking the bet) but of course, then we run back into the problem that our expectations can be dominated by tiny probabilities of extreme outcomes. This isn't just a thought experiment since we all make gambles that may affect our lifespan, and yet we don't know how long we would have lived by default. 

Edit: realized that the devil example has the obvious flaw that as the expected default lifespan increases, so does the amount of years that you're wagering, so you should always take the bet based on Kelly betting, but this point is more salient with less Pascalian lifespan-affecting gambles. I guess the question that remains is that the gamble is all or nothing, so what do we do if Kelly betting says we should wager 5% of our lifespan? Maybe the answer is: bet your life 5% of the time, or make gambles that will end your life with no more than 5% probability.

1. Oops, I was sleepy when I wrote this and used sloppy wording. Meant to say "what makes the marginal unit of value (e.g. happy lives, days of happiness, etc.) provide less and less utility."
2. I think the last point can also apply in the positive direction or at least does not require weighting negative value more heavily.

The introspective assessment is what is most persuasive to me here because 

(1) it seems like we need some reason for what makes the marginal unit of value (e.g. happy lives, days of happiness, etc.) provide less and less utility marginal unit of utility less and less valuable, independent of the fact that it lets us out of pascalianism and is logically necessary to have a bound somewhere.

(2) bounded utility functions can also lead to counterintuitive conclusions including violating ex ante pareto (Kosonen, 2022, Ch. 1) and falling prey to an Egyptology objection (Wilkinson, 2020, section 6; the post, "How do bounded utility functions work if you are uncertain how close to the bound your utility is?"). But the Egyptology objection may be less significant in practical cases where we are adding value at the margin and can see that we are getting less and less utility out of it, rather than the bound being something we have to think about in advance because we are considering some large amount of value which may hit the bound in one leap (but maybe this isn't so crazy when thinking about AI). And also I guess money pumping is worse than these other conclusions.

(3) bounded utility functions do not seem necessary to avoid pascalianism nor the most obvious option. Someone could easily have an unbounded utility function with regards to sure bets but reject pascalian bets due to probability discounting (or to prevent exploitation in the literal mugging scenario as you mention). But other people bring this up often as a natural response to pascalianism, so I may be missing a reason why you would not want to e.g. value a sure chance of saving 1 billion lives 1,000 times more than saving 1,000,000 lives for sure, but not value a 0.000001 chance at saving 1 billion lives ~at all.

(4) your reasoning makes sense that things cannot get better and better without bound. For a given individual over finite time, it seems like there will be a point where you are just experiencing pleasure all the time / have all your preferences satisfied / have everything on your objective list checked off, and then if you increase utility via time or population, you run into the thing your prior post was about. But if we endorse hedonic utilitarianism, I wonder if this intuition of mine is just reifying the hedonic treadmill and neglecting ways utility may be unbounded, particularly in the negative direction.

Nice, just in time to inform my own Pascal's wager post.

Good post. IIUC this applies only to interpersonal aggregation, and so if you can have unboundedly high utility in one individual, your utility function is not truly bounded, right? I.e., it would get you out of Pascal's muggings of the form, "Pay me five dollars and I will create 3^^^3 happy copies of Alice" but not of the form "Pay me five dollars and I will create one copy of Alice and give her 3^^^3 utils." (If it took this copy a very long time to experience 3^^^3 utils, something similar to your point would apply in that her experiences would start to overlap with her own and those of other beings, but suppose we can dial up intensity such that arbitrarily high amounts of value can be experienced in short periods of time.)

I agree that this is probably right in terms of mental health and social dynamics. Do you believe it is also right as a matter of actual morality? Do you agree with the drowning child analogy? Do you think it applies to broader social problems like AI or factory farming? If not, why?

Been telling LLMs to behave as if they and the user really like this Kierkegaard quote (to reduce sycophancy). Giving decent results so far. 

If I understand your section "Avoid Being Seen As 'Not Serious'" correctly -- that the reason policymakers don't want to support "wierd" policies is not because they're concerned about their reputation but rather that they just are too busy to do something that probably won't work -- this seems like it should meaningfully change how many people outside of politics think about advocating for AI policy. It was outside my model anyway.

The question to me is, what, if anything, is the path to change if we don't get a crisis before it is too late? Or, do we just have to place our chips on that scenario and wait for it to happen?

I agree that mass unemployment may spark policy change, but why do you see that change as being relevant to misalignment vs. specific to automation?