Would it be possible to make those clearer in the post?

I had thought, from the way you phrased it, that the assumption was that for any game, I would be equally likelly to encounter a game with the choices and power levels of the original game reversed. This struck me as plausible, or at least a good point to start from.

What you in fact seem to need, is that I am equally likely to encounter a game with the outcome under this scheme reversed, but the power levels kept the same. This continues to strike me as a very substansive and almost certainly false assertion about the games I am likely to face.

09y

After the baby, when I have time to do it properly :-)

I don't therefore see strong evidence I should reject my informal proof at this point.

I think you and I have very different understandings of the word 'proof'.

09y

It's a proof based on premises of uncertain validity. So it certainly proves
something, in some situations - the question is whether these situations are
narrow, or broad.

In the real world, agent's marginals vary a lot, and the gains from trade are huge, so this isn't likely to come up.

I doubt this claim, particularly the second part.

True, many interactions have gains from trade, but I suspect the weight of these interactions is overstated in most people's minds by the fact that they are the sort of thing that spring to mind when you talk about making deals.

Probably the most common form of interaction I have with people is when we walk past each-other in the street and neither of us hands the other the contents of their...

09y

Your "walking by in the street" example is interesting. But the point of
weighting your utilities is to split the gains from every single future
transaction and interaction with them. Since you're both part of the same
economic system, they will have (implicit or explicit) interactions in the
future. Though I don't yet know the best way of normalising multiple agents
utilities, which we'd need to make this fully rigorous.
And seeing how much world GDP is dependent on trade, I'd say the gains from
trade are immense! I note your treasure hunting example has rather large gains
from trade...
So, what we do know:
1) If everyone has utility equally linear in every resource (which we know is
false), then the more powerful player wins everything (note that this one of the
rare cases where there is an unarguable "most powerful player")
2) In general, to within the usual constraints of not losing more than you can
win, any player can get anything out of the deal (
http://lesswrong.com/r/discussion/lw/i20/even_with_default_points_systems_remain/
[http://lesswrong.com/r/discussion/lw/i20/even_with_default_points_systems_remain/]
, but you consider these utilities naturally occurring, rather than the product
of lying)
I don't therefore see strong evidence I should reject my informal proof at this
point.

You're right, I made a false statement because I was in a rush. What I meant to say was that as long as Bob's utility was linear, whatever utility function Alice has there is no way to get all the money.

Are you enforcing that choice? Because it's not a natural one.

It simplifies the scenario, and suggests.

Linear utility is not the most obviously correct utility function: diminishing marginal returns, for instance.

Why is diminishing marginal returns any more obvious that accelerating marginal returns. The former happens to be the human attitude to th...

09y

Technically true: if he's linear, Bob can't lost more than $1000, because he
can't gain more than $1000.
But Alice can certainly get almost everything. Say she has this: $1999.99 (or
above): utility 1, $1000-$1999.99: utility 0, below $1000: utility -100. Then
Alice gets $1999.99 and Bob loses 999.99.
If the value of the hoard is large, then k is very close to 1. Alice will get
the things she really likes (relative to Bob's valuation of them).
In the default, Alice gets nothing. If k is small, she'll likely get a good
chunk of the stuff. If k is large, that means that Bob can generate most of the
value on his own: Alice isn't contributing much at all, but will still get
something if she really cares about it. I don't see this as ultra-unfavourable
to Alice!
I admit there is an issue with (quasi)-linear preferences if both players have
similar relative valuations. However I don't see anything that argues that "the
default is for them to go to the powerful player", apart from in that linear
case. In the real world, agent's marginals vary a lot, and the gains from trade
are huge, so this isn't likely to come up.

It does not. See this post ( http://lesswrong.com/lw/i20/even_with_default_points_systems_remain/ ): any player can lie about their utility to force their preferred outcome to be chosen (as long as it's admissible). The weaker player can thus lie to get the maximum possible out of the stronger player. This means that there are weaker players with utility functions that would naturally give them the maximum possible. We can't assume either the weaker player or the stronger one will come out ahead in a trade, without knowing more.

Alice has $1000. Bob has ...

29y

Are you enforcing that choice? Because it's not a natural one.
Linear utility is not the most obviously correct utility function: diminishing
marginal returns, for instance.
Let Alice value $2100 at 1, $1000 at 0, and $0 at -1. Let Bob value $2100 at 1,
$1100 at 0, and $0 at -0.5 (interpolate utility linearly between these values).
These utility functions are already normalised for the MVBS, and since they
interpolate linearly, only these three points are possible solutions: Alice
$2100, default ($1000,$1100), and Bob $2100. The first has a summed utility of
0.5, the second 0, the third 0 as well.
Thus Alice gets everything.
That example is artificial, but it shows that unless you posit that everyone has
(equal) linear utility in every resource, there is no reason to assume the
powerful player will get everything: varying marginal valuations can push the
solution in one direction or the other.

If situation A is one where I am more powerful, then I will always face it at high-normalisation, and always face its complement at low normalisation. Since this system generally gives almost everything to the more powerful player, if I make the elementary error of adding the differently normalised utilities I will come up with an overly rosy view of my future prospects.

09y

It does not. See this post (
http://lesswrong.com/lw/i20/even_with_default_points_systems_remain/
[http://lesswrong.com/lw/i20/even_with_default_points_systems_remain/] ): any
player can lie about their utility to force their preferred outcome to be chosen
(as long as it's admissible). The weaker player can thus lie to get the maximum
possible out of the stronger player. This means that there are weaker players
with utility functions that would naturally give them the maximum possible. We
can't assume either the weaker player or the stronger one will come out ahead in
a trade, without knowing more.
If you don't know the opposing player, then you don't know what you'll find
important with them and what they'll find important with you. Suppose for
instance that you can produce ten million different goods, at various
inefficiencies and marginal prices. Then you meet someone who only cares about
good G, and only offers good H. Then the shape of your trade situation is
determined entirely by each player's valuations of G and H and their ability to
produce it. Even if you're extraordinarily powerful, you and they can have
valuations/ability to produce of G and H that make the situation take any shape
you want to (the default point is removing most of your options from
consideration, so only a very few of them matter).
I don't have time to do the maths, but if your values are complicated enough,
you can certainly face both A and symmetric A against (different) weaker players
(and against stronger ones).

You x+y > 2h proof is flawed, since my utility may be normalised differently in different scenarios, but this does not mean I will personally weight scenarios where it is normalised to a large number higher than those where it is normalised to a small number. I would give an example if I had more time.

09y

Yes. But you can face situation A and symmetric A both at high number
normalisation scenario, and at a low one.
We need proper priors over unknown players' utilities to solve this correctly
(finding a specific counter example is easy).

I didn't interpret the quote as implying that it would actually work, but rather as implying that (the Author thinks) Hanson's 'people don't actually care' arguments are often quite superficial.

consider that "there are no transhumanly intelligent entities in our environment" would likely be a notion that usefully-modelable-as-malevolent transhumanly intelligent entities would promote

Why?

39y

I agree that this doesn't even make sense. If you're super intelligent/powerful,
you don't need to hide. You can if you want, but ...

09y

Not an explanation, but: "The greatest trick the Devil ever pulled..."

It seems like a mess of tautologies and thought experiments

My own view is that this is precisely correct and exactly why anthropics is interesting, we really should have a good, clear approach to it and the fact we don't suggests there is still work to be done.

I don't know if this is what the poster is thinking of, but one example that came up recently for me is the distinction between risk-aversion and uncertainty-aversion (these may not be the correct terms).

Risk aversion is the what causes me to strongly not want to bet $1000 on a coin flip, even though the expectancy of is zero. I would characterise risk-aversion as an arational preference rather than an irrational bias, primarily becase it arises naturally from having a utility function that is non-linear in wealth ($100 is worth a lot if you're begging on ...

They aren't isomorphic problems, however it is the case that CDT two-boxes and defects while TDT one boxes and co-operates (against some opponents).

But at some point your character is going to think about something for more than an instant (if they don't then I strongly contest that they are very intelligent). In a best case scenario, it will take you a very long time to write this story, but I think there's some extent to which being more intelligent widens the range of thoughts you can think of ever.

That's clearly the first level meaning. He's wondering whether there's a second meaning, which is a subtle hint that he has already done exactly that, maybe hoping that Harry will pick up on it and not saying it directly in case Dumbledore or someone else is listening, maybe just a private joke.

I certainly do not define it the second way. Most people care about something other than their own happiness, and some people may care about their own happiness very little, not at all, or negatively, I really don't see why a 'happiness function' would be even slightly interesting to decision theorists.

I think I'd want to define a utility function as "what an agent wants to maximise" but I'm not entirely clear how to unpack the word 'want' in that sentence, I will admit I'm somewhat confused.

However, I'm not particularly concerned about my statements being tautological, they were meant to be, since they are arguing against statements that are tautologically false.

In that case, I would say their true utility function was "follow the deontological rules" or "avoid being smited by divine clippy", and that maximising paperclips is an instrumental subgoal.

In many other cases, I would be happy to say that the person involved was simply not utilitarian, if their actions did not seem to maximise anything at all.

09y

If you define "utility function" as "what agents maximize" then your above
statement is true but tautological. If you define "utility function" as "an
agent's relation between states of the world and that agent's hedons" then it's
not true that you can only maximize your utility function.

(1/36)(1+34p0) is bounded by 1/36, I think a classical statistician would be happy to say that the evidence has a p-value of 1/36 her. Same for any test where H_0 is a composite hypothesis, you just take the supremum.

A bigger problem with your argument is that it is a fully general counter-argument against frequentists ever concluding anything. All data has to be acquired before it can be analysed statistically, all methods of acquiring data have some probability of error (in the real world) and the probability of error is always 'unknowable', at least in ...

09y

For this problem, the p-value is bounded by 1/36 from below, that is, p-value >
1/36. The supremum of (1/36)(1+34p0) is 35/36 and the infimum is 1/36.
Therefore, I'm not taking the supremum, actually the cartoon took the infimum,
when you take the infimum you are assuming the neutrino detector measures
without errors and this is a problem. The p-value, for this example, is a number
between 1/36 and 35/36.
I did not understand "the big problem" with my argument...

So, I wrote a similar program to Phil and got similar averages, here's a sample of 5 taken while I write this comment

8.2 6.9 7.7 8.0 7.1

These look pretty similar to the numbers he's getting. Like Phil, I also get occasional results that deviate far from the mean, much more than you'd expect to happen with and approximately normally distributed variable.

I also wrote a program to test your hypothesis about the sequences being too long, running the same number of trials and seeing what the longest string of heads is, the results are

19 22 18 25 23

Do these seem abnormal enough to explain the deviation, or is there a problem with your calculations?

09y

It's not the highest that matters; it's the distribution within that range.
There was also a problem with my calculations, incidentally; a factor-of-two
error, which is enough to explain most of the discreprency. What I did to
calculate is, was to add up the harmonic sequence, up to around 24
(1+1/2+1/3+...+1/24), then doubling the last term (1+1/2+1/3+...+1/23 + 2/24).
However, the code as given starts out with a 2, and then doubles the numerator
with each added term; the calculation I should have used is
(2+2/2+2/3+2/4+...+2/23+4/24). That leads to me expecting a value just a little
over 7, which is pretty close.
...
I also ran a similar program. I copied and pasted Phil's, then modified it as
slightly. My results were:
1 500523
2 250055
3 124852
4 62067
5 31209
6 15482
7 7802
8 4011
9 1978
10 1006
11 527
12 235
13 109
14 68
15 41
16 19
17 10
18 5
21 1
...where the left-hand column is the number of terms in a given sequence, and
the right-hand column is the number of times that number of terms came up. Thus,
there were 500523 runs of one term each; an insignificant distance from the
expected number (500000). Most of the runs were very close to the expected
value; interestingly, everything from 14 terms upwards for which there were any
runs was above the expected number of runs, and often by a significant margin.
The most significant is the single 21-term run; I expect to see 0.476 of those,
and I see 1, slightly over twice the expectation. At 15 terms, I expected to see
30.517 runs; I saw 41 of those. At 17 terms, I expect to see 7.629 on average; I
see 10 this time.
My final average sum is 7.25959229425851; a little higher than expected, but,
now that I've corrected the factor-of-two error in my original calculation, not
unexpectedly far off.
So most of the deviation is due to an error in my calculation. The rest is due
to the fact that a 21-term or longer run turning up - which can easily happen -
will probably pull the average sum up by

You can double the real numbers representing them, but the results of this won't be preserved under affine transformations. So you can have two people whose utility functions are the same, tell them both "double your utility assigned to X" and get different results.

-610y

4[anonymous]10y

I think it would be very useful to explicitly state what the consequences of
utility functions being only defined up to an "affine transformation" are in the
grandparent post instead of assuming that everyone knows this at a 5-second
level. My immediate reaction to the parent post was to look at the wikipedia
article for affine transformations without much enlightenment.
Temperature is pretty useless as an analogy, because everyone knows that 2*40
degrees = 80 degrees; you have to think about moving between Celsius and
Fahrenheit to actually get something useful out of the analogy. Voltage is even
less helpful, because all it depends on is having a fixed reference point (i.e.,
differences between two voltages are always preserved, even if you change zero
points), while distances aren't preserved in general under affine
transformations.
The downvoted post and your response are the most valuable in this entire
thread, as they were the only ones that clearly communicated what the actual
ramifications of utility functions only being defined up to an affine
transformation were.
Ugh. Sorry if this came across as snarky, but the grandparent post came across
as "If this young man expresses himself in terms too deep for me,/Why, what a
very singularly deep young man/this deep young man must be!"

A green sky will be green

This is true

A pink invisible unicorn is pink

This is a meaningless sequence of squiggles on my computer screen, not a tautology

A moral system would be moral

I'm unsure what this one means

I'm not sure what 'should' means if it doesn't somehow cash out as preference.

410y

Yeah, "somehow" the two concepts are connected, we can see that, because moral
considerations act on our preferences, and most moral philosophies take the
preferences of others in considerations when deciding what's the moral thing to
do.
But the first thing that you must see is that the concepts are not identical. "I
prefer X to happen" and "I find X morally better" are different things.
Take random parent X and they'll care more about the well-being of their own
child than for the welfare of a million other children in the far corner of the
world. That doesn't mean they evaluate a world where a million other children
suffer to be a morally better world than a world where just theirs does.
Here's what I think "should" means. I think "should" is an attempted abstract
calculation of our preferences in the attempted depersonalization of the
provided context. To put it differently, I think "should" is what we believe
we'd prefer to happen if we had no personal stakes involved, or what we believe
we'd feel about the situation if our empathy was not centralized around our
closest and dearest.
EDIT TO ADD: If I had to guess further, I'd guess that the primary evolutionary
reason for our sense of morality is probably not to drive us via guilt and duty
but to drive us via moral outrage -- and that guilt is there only as in our
imagined perception of the moral outrage of others. To test that I'd like to see
if there's been studies to determine if people who are guilt-free (e.g.
psychopaths) are also free of a sense of moral outrage.

I could not abide someone doing that to me or a loved one, throwing us from relative safety into absolute disaster. So I would not do it to another. It is not my sacrifice to make.

I could not abide myself or a loved one being killed on the track. What makes their lives so much less important.

010y

But would you approve of someone else doing the same thing? Again to you or a
love one?
But I am starting to see the problem with fighting the hypothetical. It leads to
arguments and borrowed offense, thus allowing the argument to lead into
perpetuity. I can hypotectical be able to endure or not endure anything
hypotectically, but this doesn't increase my rationality or utility.
This will conclude my posting on this page. Mayby OphanWilde's discussion
[http://lesswrong.com/r/discussion/lw/hgp/morality_should_be_moral/] will be a
more appropriate topic than the Unselfish Trolley Problem.

How does this work with Clippy (the only paperclipper in known existence) being tempted with 3^^^^3 paperclips?

First thought, I'm not at all sure that it does. Pascal's mugging may still be a problem. This doesn't seem to contradict what I said about the leverage penalty being the only correct approach, rather than a 'fix' of some kind, in the first case. Worryingly, if you are correct it may also not be a 'fix' in the sense of not actually fixing anything.

I notice I'm currently confused about whether the 'causal nodes' patch is justified by the same argument. I will think about it and hopefully find an answer.

Random thoughts here, not highly confident in their correctness.

Why is the leverage penalty seen as something that needs to be added, isn't it just the obviously correct way to do probability.

Suppose I want to calculate the probability that a race of aliens will descend from the skies and randomly declare me Overlord of Earth some time in the next year. To do this, I naturally go to Delphi to talk to the Oracle of Perfect Priors, and she tells me that the chance of aliens descending from the skies and declaring an Overlord of Earth in the next year is 0.00...

410y

How does this work with Clippy (the only paperclipper in known existence) being
tempted with 3^^^^3 paperclips?
That's part of why I dislike Robin Hanson's original solution. That the
tempting/blackmailing offer involves 3^^^^3 other people, and that you are also
a person should be merely incidental to one particular illustration of the
problem of Pascal's Mugging -- and as such it can't be part of a solution to the
core problem.
To replace this with something like "causal nodes", as Eliezer mentions, might
perhaps solve the problem. But I wish that we started talking about Clippy and
his paperclips instead, so that the original illustration of the problem which
involves incidental symmetries doesn't mislead us into a "solution" overreliant
on symmetries.

I would agree that it is to some extent political. I don't think its very dark artsy though, because it seems to be a case of getting rid of an anti-FAI misunderstanding rather than creating a pro-FAI misunderstanding.

But yeah, "diyer" is too close to "die" to be easily distinguishable. Maybe "rubemond"?

I could see the argument for that, provided we also had saphmonds, emmonds etc... Otherwise you run the risk of claiming a special connection that doesn't exist.

610y

We would also need to find a different word for almonds.

Chemistry would not be improved by providing completely different names to chlorate and perchlorate (e.g. chlorate and sneblobs).

Okay, thats actually a good example. This caused me to re-think my position. After thinking, I'm still not sure that the analogy is actually valid though.

In chemistry, we have a systemic naming scheme. Systematic name schemes are good, because they let us guess word meanings without having to learn them. In a difficult field which most people learn only as adults if at all, this is a very good thing. I'm no chemist, but if I h...

010y

I agree with your analysis regarding the difference between systematic naming
systems and merely similar naming. That said, the justification for more clearly
separating Pascal's mugging and this other unnamed situation does strike me as a
political decision or rationalization. If the real world impact of people's
misunderstanding were beneficial for the AI friendly cause, I doubt if anyone
here would be making much ado about it. I would be in favor of renaming
moissanite to diamand if this would help avert our ongoing malinvestment in
clear glittery rocks to the tune of billions of dollars and numerous lives, so
political reasons can perhaps be justified in some situations.

Do you really think this!? I admit to being extremely surprised to find anyone saying this.

If rubies were called diyermands it seems to me that people wouldn't guess what it was when they heard it, they would simply guess that they had misheard 'diamond', especially since it would almost certainly be a context where that was plausible, most people would probably still have to have the word explained to them.

Furthermore, once we had the definition, we would be endlessly mixing them up, given that they come up in exactly the same context. Words are used many...

210y

Hm, that's a good point, I've changed my opinion about this case.
When I wrote my comment, I was thinking primarily of words that share a common
prefix or suffix, which tends to imply that they refer to things that share the
same category but are not the same thing. "English" and "Spanish", for example.
But yeah, "diyer" is too close to "die" to be easily distinguishable. Maybe
"rubemond"?

Even if not, they should at least be called something that acknowledges the similarity, like "Pascal-like muggings".

Any similarities are arguments for giving them a maximally *different* name to avoid confusion, not a similar one. Would the English language really be better if rubies were called diyermands?

310y

Chemistry would not be improved by providing completely different names to
chlorate and perchlorate (e.g. chlorate and sneblobs). Also, I think English
might be better if rubies were called diyermands. If all of the gemstones were
named something that followed a scheme similar to diamonds, that might be an
improvement.

-210y

I suspect it would be. The first time one encounters the word "ruby", you have
only context to go off of. But if the word sounded like "diamond", then you
could also make a tentative guess that the referent is also similar.

Why on earth would you expect the downstream utilities to exactly cancel the mugging utility?

710y

The hypothesis is not that they exactly cancel the mugging utility, but that the
downstream utilities exceed the mugging utility. I was actually thinking that
these downstream effects would be much greater than paying the mugger.

The first is contradictory, you've just told me something, then told me I don't know it, which is obviously false.

Sure this is right? After all, the implication is also true in the case of A being false, the conjuntion certainly is not.

He specifically specifies that A is true as well as A => B

Intuitively I suggest there should be an inequality, too, seeing as B|A is not necessarily independent of A.

B|A is not an event, so it makes no sense to talk about whether or not it is independent of A.

To see why this is a valid theorem, break it up into three posibilities, P(A & B) = x, P(A & ~B) = y, P(~A) = 1 - x - y.

Then P(A) = P(A & B) + P(A & ~B) =...

010y

Thank you. That is what I deserve for cursory reading.

I don't know if this is typical, but I recently a professional trader stated in an email to me that he knew very little about Bitcoin and basically had no idea what to think of it. This may hint that the lack of interest isn't based on certainty that bitcoin will flop, but simply on not knowing how to treat it and sticking to markets where they do have reasonably well-understood ways of making a profit, since exposure to risk is a limited resource.

I fully agree that is an interesting avenue of discussion, but it doesn't look much like what the paper is offering us.

Maybe I'm misunderstanding here, but it seems like we have no particular reason to suppose P=NP is independent of ZFC. Unless it is independent, its probability under this scheme must already be 1 or 0, and the only way to find out which is to prove or disprove it.

010y

I think shminux is talking about the possibility of future research addressing
bounded reasoners, who could be uncertain of P=NP even if it followed from ZFC.

0[anonymous]10y

http://www.academia.edu/1126434/IT_IS_UNLIKELY_THAT_P_NP_IS_INDEPENDENT_OF_ZFC
[http://www.academia.edu/1126434/IT_IS_UNLIKELY_THAT_P_NP_IS_INDEPENDENT_OF_ZFC]

In ZF set theory, consider the following three statements.

I) The axiom of choice is false

II) The axiom of choice is true and the continuum hypothesis is false

III) The axiom of choice is true and the continuum hypothesis is true

None of these is provably true or false so they all get assigned probability 0.5 under your scheme. This is a blatant absurdity as they are mutually exclusive so their probabilities cannot possibly sum to more than 1

110y

Ok, it seems that this is covered in the P(phi) = P(phi and psi) + P(phi and not
psi) condition. Thanks.

So induction gives the right answer 100s of times, and then gets it wrong once. Doesn't seem too bad a ratio.

I am indeed suggesting that an agent can assign utility, not merely expected utility, to a lottery.

I am suggesting that this is equivalent to suggesting that two points can be parallel. It may be true for your special definition of point, but its not true for mine, and its not true for the definition the theorems refer to.

Yes, in the real world the lottery is part of the outcome, but that can be factored in with assigning utility to the outcomes, we don't need to change our definition of utility when the existing one works (reading the rest of your post...

010y

So let me rephrase my earlier question (poorly phrased before) about what role
the VNM axioms play for you. Sometimes (especially when it comes to
"rationality") an "axiom" is held to be obvious, even indubitable: the principle
of non-contradiction is often viewed in this light. At other times, say when
formulating a mathematical model of an advanced physics theory, the axioms are
anything but obvious, but they are endorsed because they seem to work. The
axioms are the result of an inference to the best explanation.
So I'm wondering if your view is more like (A) than like (B) below.
(A) Rationality is a sort of attractor in mind-space, and people approach closer
and closer to being describable by EU theory the more rational they are. Since
the VNM axioms are obeyed in these cases, that tends to show that rationality
includes following those axioms.
(B) Obviously only a mad person would violate the Axiom of Independence knowing
full well they were doing so.
And now we are back to my True Rejection, namely: I don't think it's irrational
to take decision-angst into account, or to seek to avoid it by avoiding risk
rather than just seeking psychotherapy so that one can buck up and keep a stiff
upper lip. It's not Spock-like, but it's not irrational.

I'm not sure quite what the best response to this is, but I think I wasn't understanding you up to this point. We seem to have a bit of a level mixing problem.

In VNM utility theory, we assign utility to outcomes, defined as a complete description of what happens, and expected utility to lotteries, defined as a probability distribution over outcomes. They are measured in the same units, but they are not the same thing and should not be compared directly.

VNM utility tells you nothing about how to calculate utility and everything about how to calculate expect...

110y

I am indeed suggesting that an agent can assign utility, not merely expected
utility, to a lottery. Note that in this sentence "utility" does not have its
technical meaning(s) but simply means raw preference. With that caveat, that may
be a better way of putting it than anything I've said so far.
You can call that a category error, but I just don't see the mistake. Other than
that it doesn't fit the VNM theory, which would be a circular argument for its
irrationality in this context.
Your point about f*ing human brains gets at my True Rejection, so thanks. And I
read the conversation with kilobug. As a result I have a new idea where you may
be coming from - about which I will quote Luke's decision theory FAQ:
Emphasis added. It sounds to me like you favor a direct approach. For you,
utility is not an as-if: it is a fundamentally real, interval-scale-able quality
of our lives. In this scheme, the angst I feel while taking a risk is something
I can assign a utility to, then shut up and (re-)calculate the expected
utilities. Yes?
If you favor a direct approach, I wonder why you even care to defend the VNM
axioms, or what role they play for you.

it remains to show that someone with that preference pattern (and not pattern III) still must have a VNM utility function

Why does it remain to be shown? How does this differ from the claim that any other preference pattern that does not violate a VNM axiom is modelled by expected utility?

...Now consider the games involving chance that people enjoy. These either show (subjective probability interpretation of "risk") or provide suggestive evidence toward the possibility (epistemic probability interpretation) that some people just plain like risk.

010y

Well if it doesn't violate an axiom - and specifically I'm worried about
Independence - then the case is proven. So let me try to explain why I think it
does violate independence. The Allais Paradox provides a case where the risk
undertaken depends on so-called irrelevant alternatives. Take the version from
Luke's Decision Theory FAQ. If I bet on the option (say "red or yellow") having
34 $24000-payoff balls, whether I take a big risk depends on how many other
balls are in the lottery. If there are 66 zero-payoff green balls in the urn at
the same time, then I do take a risk. If there are no other balls in the urn,
then I don't. If I penalize the preferability of outcomes depending on the risk
undertaken, then I will penalize the "red or yellow" bet if and only if the
green balls are also involved. Say there are 33 yellow balls and 1 red one, and
I get $27000 if I bet on "yellow" instead. I will penalize the outcome, bet on
yellow and get $27000, in either scenario. If the penalty is not linear in the
amount of risk, I could conceivably prefer to bet on yellow when the green balls
are in the urn, and bet on [red or yellow] when there aren't.

So, when people say 'risk aversion', they can mean one of three different things:

I) I have a utility function that penalises world-histories in which I take risks.

II) I have a utility function which offers diminishing returns in some resource, so I am risk averse in that resource

III) I am risk averse in utility

Out of the three (III) is irrational and violates VNM. (II) is not irrational, and is an extremely common preference among humans wrt some things, but not others (money vs lives being the classic one). (I) is not irrational, but is pretty weird, I'm ...

010y

Since we agree that (I) is not irrational, it remains to show that someone with
that preference pattern (and not pattern III) still must have a VNM utility
function - then my objection will be answered. Indeed, before we can even
attribute "utility" to this person and thus go to case III, we must show that
their preferences obey certain rules (or maybe just that their rational ones
would).
I don't think preference (I) is weird at all, though I don't share it. Also not
rare: a utility function that rewards world histories in which one takes risks.
Consider that risk is either subjective or epistemic, not ontological, as used
in VNM's framework. Now consider the games involving chance that people enjoy.
These either show (subjective probability interpretation of "risk") or provide
suggestive evidence toward the possibility (epistemic probability
interpretation) that some people just plain like risk.

I think we have almost reached agreement, just a few more nitpicks I seem to have with your current post.

the independence principle doesn't strictly hold in the real world, like there are no strictly right angle in the real world

Its pedantic, but these two statements aren't analogous. A better analogy would be

"the independence principle doesn't strictly hold in the real world, like the axiom that all right angles are equal doesn't hold in the real world"

"there are no strictly identical outcomes in the real world, like there are no strict...

To me it's a single, atomic real-world choice you have to make:

To you it may be this, but the fact that this leads to an obvious absurdity suggests that this is not how most proponents think of it, or how its inventors thought of it.

Given that people can rationally have preferences that make essential reference to history and to the way events came about, why can't risk be one of those historical factors that matter? What's so "irrational" about that?

Nothing. Whoever said there was?

If your goal is to not be a thief, then expected utility theory recommends that you do not steal.

I suspect most of us do have 'do not steal' preferences on the scale of a few hundred pounds or more.

On the other hand, once you get to, say, a few hundred human lives, or the fate of the entire spe...

110y

A desire to avoid arriving at an outcome via thievery does not violate the Axiom
of Independence. A desire to avoid arriving via a risky procedure does. However,
I'm not convinced that the latter is any more irrational than the former. And I
take the point of this thread to be whether obeying the Axiom really is a
requirement of rationality.

First, I did study mathematical logic, and please avoid such kind of ad hominem.

Fair enough

That said, if what you're referring to is the whole world state, the outcomes are, in fact, always different. Even if only because there is somewhere in your brain the knowledge that the choice is different.

I thought this would be your reply, but didn't want to address it because the comment was too long already.

Firstly, this is completely correct. (Well, technically we could imagine situations where the outcomes removed your memory of there ever having been a...

310y

The cholera example was definitely a bit silly - after all, "cholera" and "apple
vs orange" are usually really independent in the real world, you've to make very
far-fetched circumstances for them to be dependent. But an axiom is supposed to
be valid everywhere - even in far-fetched circumstances ;)
But overall, I understand the thing much better now: in fact, the independence
principle doesn't strictly hold in the real world, like there are no strictly
right angle in the real world. But yet, like we do use the Pythagoras theorem in
the real world, assuming an angle to be right when it's "close enough" to be
right, we apply the VNM axioms and the related expected utility theory when we
consider the independence principle to have enough validity?
But do we have any way to measure the degree of error introduced by this
approximation? Do we have ways to recognize the cases where we shouldn't apply
the expected utility theory, because we are too far from the ideal model?
My point never was to fully reject VNM and expected utility theory - I know they
are useful, they work in many cases, ... My point was to draw attention on a
potential problem (making it an approximation, making it not always valid) that
I don't usually see being addressed (actually, I don't remember ever having seen
it that explicitly).

Gwern said pretty much everything I wanted to say to this, but there's an extra distinction I want to make

What you're doing is saying you can't use A, B, and C when there is dependency, but have to create subevents like C1="C when you are you sure you'll have either A or C".

The distinction I made was things like A2="A when you prepare" not A2="A when you are sure of getting A or C". This looks like a nitpick, but is in fact incredibly important. The difference between my A1 and A2, is important, they are fundamentally diff...

110y

First, I did study mathematical logic, and please avoid such kind of ad hominem.
That said, if what you're referring to is the whole world state, the outcomes
are, in fact, awlays different. Even if only because there is somewhere in your
brain the knowledge that the choice is different.
To take the formulation in the FAQ : « The independence axiom states that, for
example, if an agent prefers an apple to an orange, then she must also prefer
the lottery [55% chance she gets an apple, otherwise she gets cholera] over the
lottery [55% chance she gets an orange, otherwise she gets cholera]. More
generally, this axiom holds that a preference must hold independently of the
possibility of another outcome (e.g. cholera). »
That has no meaning if you consider whole world states, not just specific
outcomes. Because in the lottery it's not "apple or orange" then but "apple with
the knowledge I almost got cholera" vs "orange with the knowledge I almost got
cholera". And if there is an interaction between the two, then you have
different ranking between them. Maybe you had a friend who died of cholera and
loved apple, and that'll change how much you appreciate apples knowing you
almost had cholera. Maybe not. But anyway, if what you consider are whole world
states, then by definition the whole world state is always different when you're
offered even a slightly different choice. How can you define an independence
principle in that case ?

The problem here is that you've not specified the options in enough detail, for instance you appear to prefer going to Ecaudor with preparation time to going without preparation time, but you haven't stated this anywhere. You haven't given the slightest hint whether you prefer Iceland with preparation time to Ecuador without. VNM is not magic, if you put garbage in you get garbage out.

So to really describe the problem we need six options:

A1 - trip to Ecuador, no advance preparation A2 - trip to Ecuador, advance preparation B1 - laptop B2 - laptop, but you ...

-210y

Well, sure, by mangling enough the events you can re-establish the axioms. But
if you do that, in fact, you just don't need the axioms. The independence axiom
states that if you have B > C, then you have (probability p of A, probability
1-p of B) > (probability p of A, probability 1-p of C). What you're doing is
saying you can't use A, B, and C when there is dependency, but have to create
subevents like C1="C when you are you sure you'll have either A or C". Of
course, by splitting the events like that, you'll reestablish independence - but
by showing the need to mangle choices to make fit the axioms, you in fact have
shown the axioms don't work in the general case, when the choices you're given
are not independent, as it often is in real life.

What makes you think you have a reliable way of fooling Omega?

In particular, I am extremely sceptical that simply not making your mind up, and then at the last minute doing something that feels random, would actually correspond to making use of quantum nondeterminism. In particular, if individual neurons are reasonably deterministic, then regardless of quantum physics any human's actions can be predicted pretty perfectly, at least on a 5/10 minute scale.

Alternatively, even if it is possible to be delibrately non-cooperative, the problem can just be changed...

010y

As stated in my post I am not sure about this either, though my reasoning is,
that while memory is probably easy to read out, thinking is probably a chaotic
process, where the outcome may depend on single action potentials, especially if
the process does not heavily rely on things stored in memory. If a single action
potential occurs can be determined by few - in the limit one - sodium ion(s)
passing or not passing a channel. If a sodium Ion passes a channel is a quantum
probabilistic process. Though as I said before I am not sure of this, so
precommit to use a suitable device.
Yep! Omega can of course do so.

Not quite always

http://www.boston.com/news/local/massachusetts/articles/2011/07/31/a_lottery_game_with_a_windfall_for_a_knowing_few/