All of benelliott's Comments + Replies

1Lumifer9y
That's not an argument for lotteries, that's an argument for the observation that given sufficiently large incentives to game complex system , some complex systems will be gamed.

Would it be possible to make those clearer in the post?

I had thought, from the way you phrased it, that the assumption was that for any game, I would be equally likelly to encounter a game with the choices and power levels of the original game reversed. This struck me as plausible, or at least a good point to start from.

What you in fact seem to need, is that I am equally likely to encounter a game with the outcome under this scheme reversed, but the power levels kept the same. This continues to strike me as a very substansive and almost certainly false assertion about the games I am likely to face.

0Stuart_Armstrong9y
After the baby, when I have time to do it properly :-)

I don't therefore see strong evidence I should reject my informal proof at this point.

I think you and I have very different understandings of the word 'proof'.

0Stuart_Armstrong9y
It's a proof based on premises of uncertain validity. So it certainly proves something, in some situations - the question is whether these situations are narrow, or broad.

In the real world, agent's marginals vary a lot, and the gains from trade are huge, so this isn't likely to come up.

I doubt this claim, particularly the second part.

True, many interactions have gains from trade, but I suspect the weight of these interactions is overstated in most people's minds by the fact that they are the sort of thing that spring to mind when you talk about making deals.

Probably the most common form of interaction I have with people is when we walk past each-other in the street and neither of us hands the other the contents of their... (read more)

0Stuart_Armstrong9y
Your "walking by in the street" example is interesting. But the point of weighting your utilities is to split the gains from every single future transaction and interaction with them. Since you're both part of the same economic system, they will have (implicit or explicit) interactions in the future. Though I don't yet know the best way of normalising multiple agents utilities, which we'd need to make this fully rigorous. And seeing how much world GDP is dependent on trade, I'd say the gains from trade are immense! I note your treasure hunting example has rather large gains from trade... So, what we do know: 1) If everyone has utility equally linear in every resource (which we know is false), then the more powerful player wins everything (note that this one of the rare cases where there is an unarguable "most powerful player") 2) In general, to within the usual constraints of not losing more than you can win, any player can get anything out of the deal ( http://lesswrong.com/r/discussion/lw/i20/even_with_default_points_systems_remain/ [http://lesswrong.com/r/discussion/lw/i20/even_with_default_points_systems_remain/] , but you consider these utilities naturally occurring, rather than the product of lying) I don't therefore see strong evidence I should reject my informal proof at this point.

You're right, I made a false statement because I was in a rush. What I meant to say was that as long as Bob's utility was linear, whatever utility function Alice has there is no way to get all the money.

Are you enforcing that choice? Because it's not a natural one.

It simplifies the scenario, and suggests.

Linear utility is not the most obviously correct utility function: diminishing marginal returns, for instance.

Why is diminishing marginal returns any more obvious that accelerating marginal returns. The former happens to be the human attitude to th... (read more)

0Stuart_Armstrong9y
Technically true: if he's linear, Bob can't lost more than $1000, because he can't gain more than $1000. But Alice can certainly get almost everything. Say she has this: $1999.99 (or above): utility 1, $1000-$1999.99: utility 0, below $1000: utility -100. Then Alice gets $1999.99 and Bob loses 999.99. If the value of the hoard is large, then k is very close to 1. Alice will get the things she really likes (relative to Bob's valuation of them). In the default, Alice gets nothing. If k is small, she'll likely get a good chunk of the stuff. If k is large, that means that Bob can generate most of the value on his own: Alice isn't contributing much at all, but will still get something if she really cares about it. I don't see this as ultra-unfavourable to Alice! I admit there is an issue with (quasi)-linear preferences if both players have similar relative valuations. However I don't see anything that argues that "the default is for them to go to the powerful player", apart from in that linear case. In the real world, agent's marginals vary a lot, and the gains from trade are huge, so this isn't likely to come up.

It does not. See this post ( http://lesswrong.com/lw/i20/even_with_default_points_systems_remain/ ): any player can lie about their utility to force their preferred outcome to be chosen (as long as it's admissible). The weaker player can thus lie to get the maximum possible out of the stronger player. This means that there are weaker players with utility functions that would naturally give them the maximum possible. We can't assume either the weaker player or the stronger one will come out ahead in a trade, without knowing more.

Alice has $1000. Bob has ... (read more)

2Stuart_Armstrong9y
Are you enforcing that choice? Because it's not a natural one. Linear utility is not the most obviously correct utility function: diminishing marginal returns, for instance. Let Alice value $2100 at 1, $1000 at 0, and $0 at -1. Let Bob value $2100 at 1, $1100 at 0, and $0 at -0.5 (interpolate utility linearly between these values). These utility functions are already normalised for the MVBS, and since they interpolate linearly, only these three points are possible solutions: Alice $2100, default ($1000,$1100), and Bob $2100. The first has a summed utility of 0.5, the second 0, the third 0 as well. Thus Alice gets everything. That example is artificial, but it shows that unless you posit that everyone has (equal) linear utility in every resource, there is no reason to assume the powerful player will get everything: varying marginal valuations can push the solution in one direction or the other.

If situation A is one where I am more powerful, then I will always face it at high-normalisation, and always face its complement at low normalisation. Since this system generally gives almost everything to the more powerful player, if I make the elementary error of adding the differently normalised utilities I will come up with an overly rosy view of my future prospects.

0Stuart_Armstrong9y
It does not. See this post ( http://lesswrong.com/lw/i20/even_with_default_points_systems_remain/ [http://lesswrong.com/lw/i20/even_with_default_points_systems_remain/] ): any player can lie about their utility to force their preferred outcome to be chosen (as long as it's admissible). The weaker player can thus lie to get the maximum possible out of the stronger player. This means that there are weaker players with utility functions that would naturally give them the maximum possible. We can't assume either the weaker player or the stronger one will come out ahead in a trade, without knowing more. If you don't know the opposing player, then you don't know what you'll find important with them and what they'll find important with you. Suppose for instance that you can produce ten million different goods, at various inefficiencies and marginal prices. Then you meet someone who only cares about good G, and only offers good H. Then the shape of your trade situation is determined entirely by each player's valuations of G and H and their ability to produce it. Even if you're extraordinarily powerful, you and they can have valuations/ability to produce of G and H that make the situation take any shape you want to (the default point is removing most of your options from consideration, so only a very few of them matter). I don't have time to do the maths, but if your values are complicated enough, you can certainly face both A and symmetric A against (different) weaker players (and against stronger ones).

You x+y > 2h proof is flawed, since my utility may be normalised differently in different scenarios, but this does not mean I will personally weight scenarios where it is normalised to a large number higher than those where it is normalised to a small number. I would give an example if I had more time.

0Stuart_Armstrong9y
Yes. But you can face situation A and symmetric A both at high number normalisation scenario, and at a low one. We need proper priors over unknown players' utilities to solve this correctly (finding a specific counter example is easy).

I didn't interpret the quote as implying that it would actually work, but rather as implying that (the Author thinks) Hanson's 'people don't actually care' arguments are often quite superficial.

consider that "there are no transhumanly intelligent entities in our environment" would likely be a notion that usefully-modelable-as-malevolent transhumanly intelligent entities would promote

Why?

3Jonathan_Graehl9y
I agree that this doesn't even make sense. If you're super intelligent/powerful, you don't need to hide. You can if you want, but ...
0AlexSchell9y
Not an explanation, but: "The greatest trick the Devil ever pulled..."

It seems like a mess of tautologies and thought experiments

My own view is that this is precisely correct and exactly why anthropics is interesting, we really should have a good, clear approach to it and the fact we don't suggests there is still work to be done.

I don't know if this is what the poster is thinking of, but one example that came up recently for me is the distinction between risk-aversion and uncertainty-aversion (these may not be the correct terms).

Risk aversion is the what causes me to strongly not want to bet $1000 on a coin flip, even though the expectancy of is zero. I would characterise risk-aversion as an arational preference rather than an irrational bias, primarily becase it arises naturally from having a utility function that is non-linear in wealth ($100 is worth a lot if you're begging on ... (read more)

They aren't isomorphic problems, however it is the case that CDT two-boxes and defects while TDT one boxes and co-operates (against some opponents).

But at some point your character is going to think about something for more than an instant (if they don't then I strongly contest that they are very intelligent). In a best case scenario, it will take you a very long time to write this story, but I think there's some extent to which being more intelligent widens the range of thoughts you can think of ever.

That's clearly the first level meaning. He's wondering whether there's a second meaning, which is a subtle hint that he has already done exactly that, maybe hoping that Harry will pick up on it and not saying it directly in case Dumbledore or someone else is listening, maybe just a private joke.

I certainly do not define it the second way. Most people care about something other than their own happiness, and some people may care about their own happiness very little, not at all, or negatively, I really don't see why a 'happiness function' would be even slightly interesting to decision theorists.

I think I'd want to define a utility function as "what an agent wants to maximise" but I'm not entirely clear how to unpack the word 'want' in that sentence, I will admit I'm somewhat confused.

However, I'm not particularly concerned about my statements being tautological, they were meant to be, since they are arguing against statements that are tautologically false.

In that case, I would say their true utility function was "follow the deontological rules" or "avoid being smited by divine clippy", and that maximising paperclips is an instrumental subgoal.

In many other cases, I would be happy to say that the person involved was simply not utilitarian, if their actions did not seem to maximise anything at all.

0blacktrance9y
If you define "utility function" as "what agents maximize" then your above statement is true but tautological. If you define "utility function" as "an agent's relation between states of the world and that agent's hedons" then it's not true that you can only maximize your utility function.

(1/36)(1+34p0) is bounded by 1/36, I think a classical statistician would be happy to say that the evidence has a p-value of 1/36 her. Same for any test where H_0 is a composite hypothesis, you just take the supremum.

A bigger problem with your argument is that it is a fully general counter-argument against frequentists ever concluding anything. All data has to be acquired before it can be analysed statistically, all methods of acquiring data have some probability of error (in the real world) and the probability of error is always 'unknowable', at least in ... (read more)

0patriota9y
For this problem, the p-value is bounded by 1/36 from below, that is, p-value > 1/36. The supremum of (1/36)(1+34p0) is 35/36 and the infimum is 1/36. Therefore, I'm not taking the supremum, actually the cartoon took the infimum, when you take the infimum you are assuming the neutrino detector measures without errors and this is a problem. The p-value, for this example, is a number between 1/36 and 35/36. I did not understand "the big problem" with my argument...

So, I wrote a similar program to Phil and got similar averages, here's a sample of 5 taken while I write this comment

8.2 6.9 7.7 8.0 7.1

These look pretty similar to the numbers he's getting. Like Phil, I also get occasional results that deviate far from the mean, much more than you'd expect to happen with and approximately normally distributed variable.

I also wrote a program to test your hypothesis about the sequences being too long, running the same number of trials and seeing what the longest string of heads is, the results are

19 22 18 25 23

Do these seem abnormal enough to explain the deviation, or is there a problem with your calculations?

0CCC9y
It's not the highest that matters; it's the distribution within that range. There was also a problem with my calculations, incidentally; a factor-of-two error, which is enough to explain most of the discreprency. What I did to calculate is, was to add up the harmonic sequence, up to around 24 (1+1/2+1/3+...+1/24), then doubling the last term (1+1/2+1/3+...+1/23 + 2/24). However, the code as given starts out with a 2, and then doubles the numerator with each added term; the calculation I should have used is (2+2/2+2/3+2/4+...+2/23+4/24). That leads to me expecting a value just a little over 7, which is pretty close. ... I also ran a similar program. I copied and pasted Phil's, then modified it as slightly. My results were: 1 500523 2 250055 3 124852 4 62067 5 31209 6 15482 7 7802 8 4011 9 1978 10 1006 11 527 12 235 13 109 14 68 15 41 16 19 17 10 18 5 21 1 ...where the left-hand column is the number of terms in a given sequence, and the right-hand column is the number of times that number of terms came up. Thus, there were 500523 runs of one term each; an insignificant distance from the expected number (500000). Most of the runs were very close to the expected value; interestingly, everything from 14 terms upwards for which there were any runs was above the expected number of runs, and often by a significant margin. The most significant is the single 21-term run; I expect to see 0.476 of those, and I see 1, slightly over twice the expectation. At 15 terms, I expected to see 30.517 runs; I saw 41 of those. At 17 terms, I expect to see 7.629 on average; I see 10 this time. My final average sum is 7.25959229425851; a little higher than expected, but, now that I've corrected the factor-of-two error in my original calculation, not unexpectedly far off. So most of the deviation is due to an error in my calculation. The rest is due to the fact that a 21-term or longer run turning up - which can easily happen - will probably pull the average sum up by

You can double the real numbers representing them, but the results of this won't be preserved under affine transformations. So you can have two people whose utility functions are the same, tell them both "double your utility assigned to X" and get different results.

-6timtyler10y
4[anonymous]10y
I think it would be very useful to explicitly state what the consequences of utility functions being only defined up to an "affine transformation" are in the grandparent post instead of assuming that everyone knows this at a 5-second level. My immediate reaction to the parent post was to look at the wikipedia article for affine transformations without much enlightenment. Temperature is pretty useless as an analogy, because everyone knows that 2*40 degrees = 80 degrees; you have to think about moving between Celsius and Fahrenheit to actually get something useful out of the analogy. Voltage is even less helpful, because all it depends on is having a fixed reference point (i.e., differences between two voltages are always preserved, even if you change zero points), while distances aren't preserved in general under affine transformations. The downvoted post and your response are the most valuable in this entire thread, as they were the only ones that clearly communicated what the actual ramifications of utility functions only being defined up to an affine transformation were. Ugh. Sorry if this came across as snarky, but the grandparent post came across as "If this young man expresses himself in terms too deep for me,/Why, what a very singularly deep young man/this deep young man must be!"

A green sky will be green

This is true

A pink invisible unicorn is pink

This is a meaningless sequence of squiggles on my computer screen, not a tautology

A moral system would be moral

I'm unsure what this one means

I'm not sure what 'should' means if it doesn't somehow cash out as preference.

4ArisKatsaris10y
Yeah, "somehow" the two concepts are connected, we can see that, because moral considerations act on our preferences, and most moral philosophies take the preferences of others in considerations when deciding what's the moral thing to do. But the first thing that you must see is that the concepts are not identical. "I prefer X to happen" and "I find X morally better" are different things. Take random parent X and they'll care more about the well-being of their own child than for the welfare of a million other children in the far corner of the world. That doesn't mean they evaluate a world where a million other children suffer to be a morally better world than a world where just theirs does. Here's what I think "should" means. I think "should" is an attempted abstract calculation of our preferences in the attempted depersonalization of the provided context. To put it differently, I think "should" is what we believe we'd prefer to happen if we had no personal stakes involved, or what we believe we'd feel about the situation if our empathy was not centralized around our closest and dearest. EDIT TO ADD: If I had to guess further, I'd guess that the primary evolutionary reason for our sense of morality is probably not to drive us via guilt and duty but to drive us via moral outrage -- and that guilt is there only as in our imagined perception of the moral outrage of others. To test that I'd like to see if there's been studies to determine if people who are guilt-free (e.g. psychopaths) are also free of a sense of moral outrage.

I could not abide someone doing that to me or a loved one, throwing us from relative safety into absolute disaster. So I would not do it to another. It is not my sacrifice to make.

I could not abide myself or a loved one being killed on the track. What makes their lives so much less important.

0Petruchio10y
But would you approve of someone else doing the same thing? Again to you or a love one? But I am starting to see the problem with fighting the hypothetical. It leads to arguments and borrowed offense, thus allowing the argument to lead into perpetuity. I can hypotectical be able to endure or not endure anything hypotectically, but this doesn't increase my rationality or utility. This will conclude my posting on this page. Mayby OphanWilde's discussion [http://lesswrong.com/r/discussion/lw/hgp/morality_should_be_moral/] will be a more appropriate topic than the Unselfish Trolley Problem.

How does this work with Clippy (the only paperclipper in known existence) being tempted with 3^^^^3 paperclips?

First thought, I'm not at all sure that it does. Pascal's mugging may still be a problem. This doesn't seem to contradict what I said about the leverage penalty being the only correct approach, rather than a 'fix' of some kind, in the first case. Worryingly, if you are correct it may also not be a 'fix' in the sense of not actually fixing anything.

I notice I'm currently confused about whether the 'causal nodes' patch is justified by the same argument. I will think about it and hopefully find an answer.

Random thoughts here, not highly confident in their correctness.

Why is the leverage penalty seen as something that needs to be added, isn't it just the obviously correct way to do probability.

Suppose I want to calculate the probability that a race of aliens will descend from the skies and randomly declare me Overlord of Earth some time in the next year. To do this, I naturally go to Delphi to talk to the Oracle of Perfect Priors, and she tells me that the chance of aliens descending from the skies and declaring an Overlord of Earth in the next year is 0.00... (read more)

4ArisKatsaris10y
How does this work with Clippy (the only paperclipper in known existence) being tempted with 3^^^^3 paperclips? That's part of why I dislike Robin Hanson's original solution. That the tempting/blackmailing offer involves 3^^^^3 other people, and that you are also a person should be merely incidental to one particular illustration of the problem of Pascal's Mugging -- and as such it can't be part of a solution to the core problem. To replace this with something like "causal nodes", as Eliezer mentions, might perhaps solve the problem. But I wish that we started talking about Clippy and his paperclips instead, so that the original illustration of the problem which involves incidental symmetries doesn't mislead us into a "solution" overreliant on symmetries.

I would agree that it is to some extent political. I don't think its very dark artsy though, because it seems to be a case of getting rid of an anti-FAI misunderstanding rather than creating a pro-FAI misunderstanding.

But yeah, "diyer" is too close to "die" to be easily distinguishable. Maybe "rubemond"?

I could see the argument for that, provided we also had saphmonds, emmonds etc... Otherwise you run the risk of claiming a special connection that doesn't exist.

6Kindly10y
We would also need to find a different word for almonds.

Chemistry would not be improved by providing completely different names to chlorate and perchlorate (e.g. chlorate and sneblobs).

Okay, thats actually a good example. This caused me to re-think my position. After thinking, I'm still not sure that the analogy is actually valid though.

In chemistry, we have a systemic naming scheme. Systematic name schemes are good, because they let us guess word meanings without having to learn them. In a difficult field which most people learn only as adults if at all, this is a very good thing. I'm no chemist, but if I h... (read more)

09eB110y
I agree with your analysis regarding the difference between systematic naming systems and merely similar naming. That said, the justification for more clearly separating Pascal's mugging and this other unnamed situation does strike me as a political decision or rationalization. If the real world impact of people's misunderstanding were beneficial for the AI friendly cause, I doubt if anyone here would be making much ado about it. I would be in favor of renaming moissanite to diamand if this would help avert our ongoing malinvestment in clear glittery rocks to the tune of billions of dollars and numerous lives, so political reasons can perhaps be justified in some situations.

Do you really think this!? I admit to being extremely surprised to find anyone saying this.

If rubies were called diyermands it seems to me that people wouldn't guess what it was when they heard it, they would simply guess that they had misheard 'diamond', especially since it would almost certainly be a context where that was plausible, most people would probably still have to have the word explained to them.

Furthermore, once we had the definition, we would be endlessly mixing them up, given that they come up in exactly the same context. Words are used many... (read more)

2DSimon10y
Hm, that's a good point, I've changed my opinion about this case. When I wrote my comment, I was thinking primarily of words that share a common prefix or suffix, which tends to imply that they refer to things that share the same category but are not the same thing. "English" and "Spanish", for example. But yeah, "diyer" is too close to "die" to be easily distinguishable. Maybe "rubemond"?

Even if not, they should at least be called something that acknowledges the similarity, like "Pascal-like muggings".

Any similarities are arguments for giving them a maximally different name to avoid confusion, not a similar one. Would the English language really be better if rubies were called diyermands?

39eB110y
Chemistry would not be improved by providing completely different names to chlorate and perchlorate (e.g. chlorate and sneblobs). Also, I think English might be better if rubies were called diyermands. If all of the gemstones were named something that followed a scheme similar to diamonds, that might be an improvement.
-2DSimon10y
I suspect it would be. The first time one encounters the word "ruby", you have only context to go off of. But if the word sounded like "diamond", then you could also make a tentative guess that the referent is also similar.

Why on earth would you expect the downstream utilities to exactly cancel the mugging utility?

7Nick_Beckstead10y
The hypothesis is not that they exactly cancel the mugging utility, but that the downstream utilities exceed the mugging utility. I was actually thinking that these downstream effects would be much greater than paying the mugger.

The first is contradictory, you've just told me something, then told me I don't know it, which is obviously false.

Sure this is right? After all, the implication is also true in the case of A being false, the conjuntion certainly is not.

He specifically specifies that A is true as well as A => B

Intuitively I suggest there should be an inequality, too, seeing as B|A is not necessarily independent of A.

B|A is not an event, so it makes no sense to talk about whether or not it is independent of A.

To see why this is a valid theorem, break it up into three posibilities, P(A & B) = x, P(A & ~B) = y, P(~A) = 1 - x - y.

Then P(A) = P(A & B) + P(A & ~B) =... (read more)

0Metus10y
Thank you. That is what I deserve for cursory reading.

I don't know if this is typical, but I recently a professional trader stated in an email to me that he knew very little about Bitcoin and basically had no idea what to think of it. This may hint that the lack of interest isn't based on certainty that bitcoin will flop, but simply on not knowing how to treat it and sticking to markets where they do have reasonably well-understood ways of making a profit, since exposure to risk is a limited resource.

I fully agree that is an interesting avenue of discussion, but it doesn't look much like what the paper is offering us.

Maybe I'm misunderstanding here, but it seems like we have no particular reason to suppose P=NP is independent of ZFC. Unless it is independent, its probability under this scheme must already be 1 or 0, and the only way to find out which is to prove or disprove it.

0endoself10y
I think shminux is talking about the possibility of future research addressing bounded reasoners, who could be uncertain of P=NP even if it followed from ZFC.
0[anonymous]10y
http://www.academia.edu/1126434/IT_IS_UNLIKELY_THAT_P_NP_IS_INDEPENDENT_OF_ZFC [http://www.academia.edu/1126434/IT_IS_UNLIKELY_THAT_P_NP_IS_INDEPENDENT_OF_ZFC]

In ZF set theory, consider the following three statements.

I) The axiom of choice is false

II) The axiom of choice is true and the continuum hypothesis is false

III) The axiom of choice is true and the continuum hypothesis is true

None of these is provably true or false so they all get assigned probability 0.5 under your scheme. This is a blatant absurdity as they are mutually exclusive so their probabilities cannot possibly sum to more than 1

1V_V10y
Ok, it seems that this is covered in the P(phi) = P(phi and psi) + P(phi and not psi) condition. Thanks.

So induction gives the right answer 100s of times, and then gets it wrong once. Doesn't seem too bad a ratio.

I am indeed suggesting that an agent can assign utility, not merely expected utility, to a lottery.

I am suggesting that this is equivalent to suggesting that two points can be parallel. It may be true for your special definition of point, but its not true for mine, and its not true for the definition the theorems refer to.

Yes, in the real world the lottery is part of the outcome, but that can be factored in with assigning utility to the outcomes, we don't need to change our definition of utility when the existing one works (reading the rest of your post... (read more)

0torekp10y
So let me rephrase my earlier question (poorly phrased before) about what role the VNM axioms play for you. Sometimes (especially when it comes to "rationality") an "axiom" is held to be obvious, even indubitable: the principle of non-contradiction is often viewed in this light. At other times, say when formulating a mathematical model of an advanced physics theory, the axioms are anything but obvious, but they are endorsed because they seem to work. The axioms are the result of an inference to the best explanation. So I'm wondering if your view is more like (A) than like (B) below. (A) Rationality is a sort of attractor in mind-space, and people approach closer and closer to being describable by EU theory the more rational they are. Since the VNM axioms are obeyed in these cases, that tends to show that rationality includes following those axioms. (B) Obviously only a mad person would violate the Axiom of Independence knowing full well they were doing so. And now we are back to my True Rejection, namely: I don't think it's irrational to take decision-angst into account, or to seek to avoid it by avoiding risk rather than just seeking psychotherapy so that one can buck up and keep a stiff upper lip. It's not Spock-like, but it's not irrational.

I'm not sure quite what the best response to this is, but I think I wasn't understanding you up to this point. We seem to have a bit of a level mixing problem.

In VNM utility theory, we assign utility to outcomes, defined as a complete description of what happens, and expected utility to lotteries, defined as a probability distribution over outcomes. They are measured in the same units, but they are not the same thing and should not be compared directly.

VNM utility tells you nothing about how to calculate utility and everything about how to calculate expect... (read more)

1torekp10y
I am indeed suggesting that an agent can assign utility, not merely expected utility, to a lottery. Note that in this sentence "utility" does not have its technical meaning(s) but simply means raw preference. With that caveat, that may be a better way of putting it than anything I've said so far. You can call that a category error, but I just don't see the mistake. Other than that it doesn't fit the VNM theory, which would be a circular argument for its irrationality in this context. Your point about f*ing human brains gets at my True Rejection, so thanks. And I read the conversation with kilobug. As a result I have a new idea where you may be coming from - about which I will quote Luke's decision theory FAQ: Emphasis added. It sounds to me like you favor a direct approach. For you, utility is not an as-if: it is a fundamentally real, interval-scale-able quality of our lives. In this scheme, the angst I feel while taking a risk is something I can assign a utility to, then shut up and (re-)calculate the expected utilities. Yes? If you favor a direct approach, I wonder why you even care to defend the VNM axioms, or what role they play for you.

it remains to show that someone with that preference pattern (and not pattern III) still must have a VNM utility function

Why does it remain to be shown? How does this differ from the claim that any other preference pattern that does not violate a VNM axiom is modelled by expected utility?

Now consider the games involving chance that people enjoy. These either show (subjective probability interpretation of "risk") or provide suggestive evidence toward the possibility (epistemic probability interpretation) that some people just plain like risk.

... (read more)
0torekp10y
Well if it doesn't violate an axiom - and specifically I'm worried about Independence - then the case is proven. So let me try to explain why I think it does violate independence. The Allais Paradox provides a case where the risk undertaken depends on so-called irrelevant alternatives. Take the version from Luke's Decision Theory FAQ. If I bet on the option (say "red or yellow") having 34 $24000-payoff balls, whether I take a big risk depends on how many other balls are in the lottery. If there are 66 zero-payoff green balls in the urn at the same time, then I do take a risk. If there are no other balls in the urn, then I don't. If I penalize the preferability of outcomes depending on the risk undertaken, then I will penalize the "red or yellow" bet if and only if the green balls are also involved. Say there are 33 yellow balls and 1 red one, and I get $27000 if I bet on "yellow" instead. I will penalize the outcome, bet on yellow and get $27000, in either scenario. If the penalty is not linear in the amount of risk, I could conceivably prefer to bet on yellow when the green balls are in the urn, and bet on [red or yellow] when there aren't.

So, when people say 'risk aversion', they can mean one of three different things:

I) I have a utility function that penalises world-histories in which I take risks.

II) I have a utility function which offers diminishing returns in some resource, so I am risk averse in that resource

III) I am risk averse in utility

Out of the three (III) is irrational and violates VNM. (II) is not irrational, and is an extremely common preference among humans wrt some things, but not others (money vs lives being the classic one). (I) is not irrational, but is pretty weird, I'm ... (read more)

0torekp10y
Since we agree that (I) is not irrational, it remains to show that someone with that preference pattern (and not pattern III) still must have a VNM utility function - then my objection will be answered. Indeed, before we can even attribute "utility" to this person and thus go to case III, we must show that their preferences obey certain rules (or maybe just that their rational ones would). I don't think preference (I) is weird at all, though I don't share it. Also not rare: a utility function that rewards world histories in which one takes risks. Consider that risk is either subjective or epistemic, not ontological, as used in VNM's framework. Now consider the games involving chance that people enjoy. These either show (subjective probability interpretation of "risk") or provide suggestive evidence toward the possibility (epistemic probability interpretation) that some people just plain like risk.

I think we have almost reached agreement, just a few more nitpicks I seem to have with your current post.

the independence principle doesn't strictly hold in the real world, like there are no strictly right angle in the real world

Its pedantic, but these two statements aren't analogous. A better analogy would be

"the independence principle doesn't strictly hold in the real world, like the axiom that all right angles are equal doesn't hold in the real world"

"there are no strictly identical outcomes in the real world, like there are no strict... (read more)

To me it's a single, atomic real-world choice you have to make:

To you it may be this, but the fact that this leads to an obvious absurdity suggests that this is not how most proponents think of it, or how its inventors thought of it.

Given that people can rationally have preferences that make essential reference to history and to the way events came about, why can't risk be one of those historical factors that matter? What's so "irrational" about that?

Nothing. Whoever said there was?

If your goal is to not be a thief, then expected utility theory recommends that you do not steal.

I suspect most of us do have 'do not steal' preferences on the scale of a few hundred pounds or more.

On the other hand, once you get to, say, a few hundred human lives, or the fate of the entire spe... (read more)

1torekp10y
A desire to avoid arriving at an outcome via thievery does not violate the Axiom of Independence. A desire to avoid arriving via a risky procedure does. However, I'm not convinced that the latter is any more irrational than the former. And I take the point of this thread to be whether obeying the Axiom really is a requirement of rationality.

First, I did study mathematical logic, and please avoid such kind of ad hominem.

Fair enough

That said, if what you're referring to is the whole world state, the outcomes are, in fact, always different. Even if only because there is somewhere in your brain the knowledge that the choice is different.

I thought this would be your reply, but didn't want to address it because the comment was too long already.

Firstly, this is completely correct. (Well, technically we could imagine situations where the outcomes removed your memory of there ever having been a... (read more)

3kilobug10y
The cholera example was definitely a bit silly - after all, "cholera" and "apple vs orange" are usually really independent in the real world, you've to make very far-fetched circumstances for them to be dependent. But an axiom is supposed to be valid everywhere - even in far-fetched circumstances ;) But overall, I understand the thing much better now: in fact, the independence principle doesn't strictly hold in the real world, like there are no strictly right angle in the real world. But yet, like we do use the Pythagoras theorem in the real world, assuming an angle to be right when it's "close enough" to be right, we apply the VNM axioms and the related expected utility theory when we consider the independence principle to have enough validity? But do we have any way to measure the degree of error introduced by this approximation? Do we have ways to recognize the cases where we shouldn't apply the expected utility theory, because we are too far from the ideal model? My point never was to fully reject VNM and expected utility theory - I know they are useful, they work in many cases, ... My point was to draw attention on a potential problem (making it an approximation, making it not always valid) that I don't usually see being addressed (actually, I don't remember ever having seen it that explicitly).

Gwern said pretty much everything I wanted to say to this, but there's an extra distinction I want to make

What you're doing is saying you can't use A, B, and C when there is dependency, but have to create subevents like C1="C when you are you sure you'll have either A or C".

The distinction I made was things like A2="A when you prepare" not A2="A when you are sure of getting A or C". This looks like a nitpick, but is in fact incredibly important. The difference between my A1 and A2, is important, they are fundamentally diff... (read more)

1kilobug10y
First, I did study mathematical logic, and please avoid such kind of ad hominem. That said, if what you're referring to is the whole world state, the outcomes are, in fact, awlays different. Even if only because there is somewhere in your brain the knowledge that the choice is different. To take the formulation in the FAQ : « The independence axiom states that, for example, if an agent prefers an apple to an orange, then she must also prefer the lottery [55% chance she gets an apple, otherwise she gets cholera] over the lottery [55% chance she gets an orange, otherwise she gets cholera]. More generally, this axiom holds that a preference must hold independently of the possibility of another outcome (e.g. cholera). » That has no meaning if you consider whole world states, not just specific outcomes. Because in the lottery it's not "apple or orange" then but "apple with the knowledge I almost got cholera" vs "orange with the knowledge I almost got cholera". And if there is an interaction between the two, then you have different ranking between them. Maybe you had a friend who died of cholera and loved apple, and that'll change how much you appreciate apples knowing you almost had cholera. Maybe not. But anyway, if what you consider are whole world states, then by definition the whole world state is always different when you're offered even a slightly different choice. How can you define an independence principle in that case ?

The problem here is that you've not specified the options in enough detail, for instance you appear to prefer going to Ecaudor with preparation time to going without preparation time, but you haven't stated this anywhere. You haven't given the slightest hint whether you prefer Iceland with preparation time to Ecuador without. VNM is not magic, if you put garbage in you get garbage out.

So to really describe the problem we need six options:

A1 - trip to Ecuador, no advance preparation A2 - trip to Ecuador, advance preparation B1 - laptop B2 - laptop, but you ... (read more)

-2kilobug10y
Well, sure, by mangling enough the events you can re-establish the axioms. But if you do that, in fact, you just don't need the axioms. The independence axiom states that if you have B > C, then you have (probability p of A, probability 1-p of B) > (probability p of A, probability 1-p of C). What you're doing is saying you can't use A, B, and C when there is dependency, but have to create subevents like C1="C when you are you sure you'll have either A or C". Of course, by splitting the events like that, you'll reestablish independence - but by showing the need to mangle choices to make fit the axioms, you in fact have shown the axioms don't work in the general case, when the choices you're given are not independent, as it often is in real life.

What makes you think you have a reliable way of fooling Omega?

In particular, I am extremely sceptical that simply not making your mind up, and then at the last minute doing something that feels random, would actually correspond to making use of quantum nondeterminism. In particular, if individual neurons are reasonably deterministic, then regardless of quantum physics any human's actions can be predicted pretty perfectly, at least on a 5/10 minute scale.

Alternatively, even if it is possible to be delibrately non-cooperative, the problem can just be changed... (read more)

0EGI10y
As stated in my post I am not sure about this either, though my reasoning is, that while memory is probably easy to read out, thinking is probably a chaotic process, where the outcome may depend on single action potentials, especially if the process does not heavily rely on things stored in memory. If a single action potential occurs can be determined by few - in the limit one - sodium ion(s) passing or not passing a channel. If a sodium Ion passes a channel is a quantum probabilistic process. Though as I said before I am not sure of this, so precommit to use a suitable device. Yep! Omega can of course do so.
Load More