You have to have some information besides behavior alone, or you'll have this extra degree of freedom in making interpretations.
Yup. Seems to be an instance of the general principle that black-box agents can be assigned any values whatsoever.
Why can't we use wrong probabilities in real life?
There are various circumstances where I want to "rotate" between probabilities and utilities in real life, in ways that still prescribe the correct decisions. For example, if I have a startup idea and want to maximize my expected profit, I'd be much more emotionally comfortable with thinking it has a 90% chance of making $1 billion than a 10% chance of making $9 billion. So why can't we use wrong probabilities in real life?I think there are three major reasons why this doesn't always work.
It also seems like you are not actually utility indifferent as between a 90% chance of $1b and a 9% chance of $9b - the former seems far more valuable to me because once you have about ~$10m, the rest is just points on a scoreboard. So to the extent that you are more emotionally comfortable with 90%/$1b I think it's actually because the expected utility of 90%/$1b is almost 10 times higher than the expected utility of 9%/$9b. And so to the extent you are setting your motivation based on these 2 things, you are importantly fooling yourself here.
TBH, for the equation "Util(90% chance of $1b) = Util(9% chance of $X)", I don't think there is any finite X that can solve that equation.
This is true, but even worse:
Originally posted as an EA forum comment.
Suppose that there are two effective altruist billionaires, April and Autumn. Originally they were funding AMF because they thought funding AI alignment would be 0.001% likely to work and solving alignment would be as good as saving 10 billion lives, which is an expected value of 100,000 lives, lower than they could get by funding AMF.
After being in the EA community a while, they switched to funding alignment research for different reasons.
Both of them assign the same expected utility to alignment -- 1 billion lives. As such, they will make the same funding decisions. So even though April made an epistemic update and Autumn a moral update, we cannot distinguish between them from behavior alone.
This extends to a general principle: actions are driven by a combination of your values and subjective probabilities, and any given action is consistent with many different combinations of utility function and probability distribution.
As a second example, suppose Bart is an investor who makes risk-averse decisions (say, invests in bonds rather than stocks). He might do this for two reasons:
These different combinations of probability and utility inform the same risk-averse behavior. In fact, probability and utility are so interchangeable that professional traders -- just about the most calibrated, rational people with regard to probability of losing money, and who are only risk-averse for reason (1) -- often model financial products as if losing money is more likely than it actually is, because it makes the math easier.[1]
Implication
When observing that someone's behavior has changed, it's not obvious what changes are value drift vs. epistemic updates. You have to have some information besides behavior alone, or you'll have this extra degree of freedom in making interpretations.
Formally, this is using the risk-neutral probability measure Q rather than the real probability measure P to price products; in finance circles it's apparently said that such people are living in "Q world".