All of Sniffnoy's Comments + Replies

Can crimes be discussed literally?

Yeah. You can use language that is unambiguously not attack language, it just takes more effort to avoid common words. In this respect it's not much unlike how discussing lots of other things seriously requires avoiding common but confused words!

Classifying games like the Prisoner's Dilemma

I'm reminded of this paper, which discusses a smaller set of two-player games. What you call "Cake Eating" they call the "Harmony Game". They also use the more suggestive variable names -- which I believe come from existing literature -- R (reward), S (sucker's payoff), T (temptation), P (punishment) instead of (W, X, Y, Z). Note that in addition to R > P (W > Z) they also added the restrictions T > P (Y > Z) and R > S (W > X) so that the two options could be meaningfully labeled "cooperate" and "defect" instead of "Krump" and "Flitz" ... (read more)

Thirty-three randomly selected bioethics papers

I suppose so. It is at least a different problem than I was worried about...

Thirty-three randomly selected bioethics papers

Huh. Given the negative reputation of bioethics around here -- one I hadn't much questioned, TBH -- most of these are suprisingly reasonable. Only #10, #16, and #24 really seemed like the LW stereotype of the bioethics paper that I would roll my eyes at. Arguably also #31, but I'd argue that one is instead alarming in a different way.

Some others seemed like bureaucratic junk (so, neither good nor bad), and others I think the quoted sections didn't really give enough information to judge; it is quite possible that a few more of these would go under the s... (read more)

4MikkW2moWouldn't the presence of "bureaucratic junk" be evidence towards a field having problems?
Jean Monnet: The Guerilla Bureaucrat

Consider a modified version of the prisoner's dilemma. This time, the prisoners are allowed to communicate, but they also have to solve an additional technical problem, say, how to split the loot. They may start with agreeing on not betraying each other to the prosecutors, but later one of them may say: "I've done most of the work. I want 70% of the loot, otherwise I am going to rat on you." It's easy to see how the problem would escalate and end up in the prisoners betraying each other.

Minor note, but I think you could just talk about a [bargaining gam... (read more)

Dark Matters

I just explained why (without more specific theories of in exactly what way the gravity would become delocalized from the visible mass) the bullet cluster is not evidence one way or the other.

Now, you compare the extra fields of modified gravity to epicycles -- as in, post-hoc complications grafted on to a theory to explain a particular phenomenon. But these extra fields are, to the best of my understanding, not grafted on to explain such delocalization; they're the actual basic content of the modified gravity theories and necessary to obtain a workable t... (read more)

Defending the non-central fallacy

I feel like this really misses the point of the whole "non-central fallacy" idea. I would say, categories are heuristics and those heuristics have limits. When the category gets strained, the thing to do is to stop arguing using the category and start arguing the particular facts without relation to the category ("taboo your words").

You're saying that this sort of arguing-via-category is useful because it's actually aguing-via-similarity; but I see the point of Scott/Yvain's original article being that such arguing via similarity simply isn't useful in s... (read more)

Dark Matters

Good post. Makes a good case. I wasn't aware of the evidence from galactic cluster lensing; that's pretty impressive. (I guess not as much as the CMB power spectrum, but that I'd heard about before. :P )

But, my understanding is that the Bullet Cluster is actually not the strong evidence it's claimed to be? My understanding of modified gravity theories is that, since they all work by adding extra fields, it's also possible for those to have gravity separated from visible matter, even if no dark matter is present. (See e.g.. here... of course in this po... (read more)

0maximkazhenkov2moAnd with enough epicycles you can fit the motion of planets with geocentricism. If MOND supporters can dismiss Bullet Cluster they'll dismiss any future evidence, too.
Blue is arbitrary

"Cyan" isn't a basic color term in English; English speakers ordinarily consider cyan to be a variant of blue, not something basically separate. Something that is cyan could also be described in English as "blue". As opposed to say, red and pink -- these are both basic color terms in English; an English speaker would not ordinarily refer to something pink as "red", or vice versa.

Or in other words: Color words don't refer to points in color space, they refer to regions, which means that you can look at how those regions overlap -- some may be subsets of o... (read more)

Making Vaccine

Wow!

I guess a thing that still bugs me after reading the rest of the comments is, if it turns out that this vaccine only offers protection against inhaling the virus though the nose, how much does that help when one considers that one could also inhale it through the mouth? Like, I worry that after taking this I'd still need to avoiding indoor spaces with other people, etc, which would defeat a lot of the benefit of it.

But, if it turns out that it does yield antibodies in the blood, then... this sounds very much worth trying!

8Dentin3moMy understanding is that it helps a lot. The biggest benefit seems to be that the immune system is primed in at least some fashion; it knows what to look for, and it has readily available tools that should be effective. It doesn't have to take a day or a week to try random things before it finally discovers a particularly effective antibody and gets the production chain ramped up to start a proper immune response. Instead, your immune system will very quickly get a signal it understands as bad and can immediately start ramping up when it does detect the virus. Keep in mind that the commercial vaccines don't have 100% success rate in that some people still get sick, but the 'priming' of the immune response is still there. I believe this is why the death rate / severe complications rate is effectively zero for immunized patients, even though it's possible to get sick. (Again, my understanding. I would very much appreciate correction/clarifications here.)
Most Prisoner's Dilemmas are Stag Hunts; Most Stag Hunts are Schelling Problems

So, why do we perceive so many situations to be Prisoner's Dilemma -like rather than Stag Hunt -like?

I don't think that we do, exactly. I think that most people only know the term "prisoners' dilemma" and haven't learned any more game theory than that; and then occasionally they go and actually attempt to map things onto the Prisoners' Dilemma as a result. :-/

Toolbox-thinking and Law-thinking

That sounds like it might have been it?

Swiss Political System: More than You ever Wanted to Know (III.)

Sorry, but after reading this I'm not very clear on just what exactly the "Magic Formula" refers to. Could you state it explicitly?

6Martin Sustrik9moFixed:
Underappreciated points about utility functions (of both sorts)

Oops, turns out I did misremember -- Savage does not in fact put the proof in his book. You have to go to Fishburn's book.

I've been reviewing all this recently and yeah -- for anyone else who wants to get into this, I'd reccommend getting Fishburn's book ("Utility Theory for Decision Making") in addition to Savage's "Foundations of Statistics". Because in addition to the above, what I'd also forgotten is that Savage leaves out a bunch of the proofs. It's really annoying. Thankfully in Fishburn's treatment he went and actually elaborated all the proofs

... (read more)
What Money Cannot Buy

Oh, I see. I misread your comment then. Yes, I am assuming one already has the ability to discern the structure of an argument and doesn't need to hire someone else to do that for you...

What Money Cannot Buy

What I said above. Sorry, to be clear here, by "argument structure" I don't mean the structure of the individual arguments but rather the overall argument -- what rebuts what.

(Edit: Looks like I misread the parent comment and this fails to respond to it; see below.)

4Vaniver1yTo be clear as well, the rhetorical point underneath my question is that I don't think your heuristic is all that useful, and seems grounded in generalization from too few examples without searching for counterexamples. Rather than just attacking it directly like Gordon, I was trying to go up a meta-level, to just point at the difficulty of 'buying' methods of determining expertise, because you need to have expertise in distinguishing the market there. (In general, when someone identifies a problem and you think you have a solution, it's useful to consider whether your solution suffers from that problem on a different meta-level; sometimes you gain from sweeping the difficulty there, and sometimes you don't.)
What Money Cannot Buy

This is a good point (the redemption movement comes to mind as an example), but I think the cases I'm thinking of and the cases you're describing look quite different in other details. Like, the bored/annoyed expert tired of having to correct basic mistakes, vs. the salesman who wants to initiate you into a new, exciting secret. But yeah, this is only a quick-and-dirty heuristic, and even then only good for distinguishing snake oil; it might not be a good idea to put too much weight on it, and it definitely won't help you in a real dispute ("Wait, both s

... (read more)
What Money Cannot Buy

Given a bunch of people who disagree, some of whom are actual experts and some of whom are selling snake oil, expertise yourself, there are some further quick-and-dirty heuristics you can use to tell which of the two groups is which. I think basically my suggestion can be best summarized at "look at argument structure".

The real experts will likely spend a bunch of time correct popular misconceptions, which the fakers may subscribe to. By contrast, the fakers will generally not bother "correcting" the truth to their fakery, because why would they? They'r

... (read more)
2toonalfrink1yHere's another: probing into their argument structure a bit and checking if they can keep it from collapsing under its own weight. https://www.lesswrong.com/posts/wyyfFfaRar2jEdeQK/entangled-truths-contagious-lies [https://www.lesswrong.com/posts/wyyfFfaRar2jEdeQK/entangled-truths-contagious-lies]
4Vaniver1yAnd how does one distinguish snake oil salesmen and real experts when it comes to identifying argument structure and what it implies?
The real experts will likely spend a bunch of time correct popular misconceptions, which the fakers may subscribe to. By contrast, the fakers will generally not bother "correcting" the truth to their fakery, because why would they? They're trying to sell to unreflective people who just believe the obvious-seeming thing; someone who actually bothered to read corrections to misconceptions at any point is likely too savvy to be their target audience.

Using this as a heuristic would often backfire on you as stated, because there's a certain ... (read more)

7jmh1yThis seems to rely on the fakes knowing they are fakes. I agree that is a problem and your heuristic useful but I think we (non-experts) are still stuck with the problem of separating out the real experts from those that mistakenly think they are also real experts. Those will likely attempt to correct the true security approach according to their mistaken premises and solutions. We're still stuck with the problem that money doesn't get the non-expert client too far. Now, you've clearly been able to reduce the ratio of real solution to snake oil so moved the probabilities in your favor when throwing money at the problems but not sure just how far.
Underappreciated points about utility functions (of both sorts)

Well, it's worth noting that P7 is introduced to address gambles with infinitely many possible outcomes, regardless of whether those outcomes are bounded or not (which is the reason I argue above you can't just get rid of it). But yeah. Glad that's cleared up now! :)

Underappreciated points about utility functions (of both sorts)

Ahh, thanks for clarifying. I think what happened was that your modus ponens was my modus tollens -- so when I think about my preferences, I ask "what conditions do my preferences need to satisfy for me to avoid being exploited or undoing my own work?" whereas you ask something like "if my preferences need to correspond to a bounded utility function, what should they be?" [1]

That doesn't seem right. The whole point of what I've been saying is that we can write down some simple conditions that ought to be true in order to avoid being exploitable or othe

... (read more)
4Isnasene1yThanks for the reply. I re-read your post and your post on Savage's proof and you're right on all counts. For some reason, it didn't actually click for me that P7 was introduced to address unbounded utility functions and boundedness was a consequence of taking the axioms to their logical conclusion.
A summary of Savage's foundations for probability and utility.

Here's a quick issue I only just noticed but which fortunately is easily fixed:

Above I mentioned you probably want to restrict to a sigma-algebra of events and only allow measurable functions as actions. But, what does measurable mean here? Fortunately, the ordering on outcomes (even without utility) makes measurability meaningful. Except this puts a circularity in the setup, because the ordering on outcomes is induced from the ordering on actions.

Fortunately this is easily patched. You can start with the assumption of a total preorder on outcomes (con

... (read more)
Underappreciated points about utility functions (of both sorts)

(This is more properly a followup to my sibling comment, but posting it here so you'll see it.)

I already said that I think that thinking in terms of infinitary convex combinations, as you're doing, is the wrong way to go about it; but it took me a bit to put together why that's definitely the wrong way.

Specifically, it assumes probability! Fishburn, in the paper you link, assumes probability, which is why he's able to talk about why infinitary convex combinations are or are not allowed (I mean, that and the fact that he's not necessarily arbitrary actions

... (read more)
2Richard_Kennaway1ySavage doesn't assume probability or utility, but their construction is a mathematical consequence of the axioms. So although they come later in the exposition, they mathematically exist as soon as the axioms have been stated. I am still thinking about that, and may be some time. As a general outline of the situation, you read P1-7 => bounded utility as modus ponens: you accept the axioms and therefore accept the conclusion. I read it as modus tollens: the conclusion seems wrong, so I believe there is a flaw in the axioms. In the same way, the axioms of Euclidean geometry seemed very plausible as a description of the physical space we find ourselves in, but conflicts emerged with phenomena of electromagnetism and gravity, and eventually they were superseded as descriptions of physical space by the geometry of differential manifolds. It isn't possible to answer the question "which of P1-7 would I reject?" What is needed to block the proof of bounded utility is a new set of axioms, which will no doubt imply large parts of P1-7, but might not imply the whole of any one of them. If and when such a set of axioms can be found, P1-7 can be re-examined in their light.
Underappreciated points about utility functions (of both sorts)

Apologies, but it sounds like you've gotten some things mixed up here? The issue is boundedness of utility functions, not whether they can take on infinity as a value. I don't think anyone here is arguing that utility functions don't need to be finite-valued. All the things you're saying seem to be related to the latter question rather than the former, or you seem to be possibly conflating them?

In the second paragraph perhaps this is just an issue of language -- when you say "infinitely high", do you actually mean "aribtrarily high"? -- but in the first

... (read more)
1Isnasene1yAhh, thanks for clarifying. I think what happened was that your modus ponens was my modus tollens -- so when I think about my preferences, I ask "what conditions do my preferences need to satisfy for me to avoid being exploited or undoing my own work?" whereas you ask something like "if my preferences need to correspond to a bounded utility function, what should they be?" [1]. As a result, I went on a tangent about infinity to begin exploring whether my modified notion of a utility function would break in ways that regular ones wouldn't. I agree, one shouldn't conclude anything without a theorem. Personally, I would approach the problem by looking at the infinite wager comparisons discussed earlier and trying to formalize them into additional rationality condition. We'd need * an axiom describing what it means for one infinite wager to be "strictly better" than another. * an axiom describing what kinds of infinite wagers it is rational to be indifferent towards Then, I would try to find a decisioning-system that satisfies these new conditions as well as the VNM-rationality axioms (where VNM-rationality applies). If such a system exists, these axioms would probably bar it from being represented fully as a utility function. If it didn't, that'd be interesting. In any case, whatever happens will tell us more about either the structure our preferences should follow or the structure that our rationality-axioms should follow (if we cannot find a system). Of course, maybe my modification of the idea of a utility function turns out to show such a decisioning-system exists by construction. In this case, modifying the idea of a utility function would help tell me that my preferences should follow the structure of that modification as well. Does that address the question? [1] From your post:
Underappreciated points about utility functions (of both sorts)

Oh, so that's what you're referring to. Well, if you look at the theorem statements, you'll see that P=P_d is an axiom that is explicitly called out in the theorems where it's assumed; it's not implictly part of Axiom 0 like you asserted, nor is it more generally left implicit at all.

but the important part is that last infinite sum: this is where all infinitary convex combinations are asserted to exist. Whether that is assigned to "background setup" or "axioms" does not matter. It has to be present, to allow the construction of St. Petersburg gambles.

... (read more)
Underappreciated points about utility functions (of both sorts)

Savage does not actually prove bounded utility. Fishburn did this later, as Savage footnotes in the edition I'm looking at, so Fishburn must be tackled.

Yes, it was actually Fishburn that did that. Apologies if I carelessly implied it was Savage.

IIRC, Fishburn's proof, formulated in Savage's terms, is in Savage's book, at least if you have the second edition. Which I think you must, because otherwise that footnote wouldn't be there at all. But maybe I'm misremembering? I think it has to be though...

In Savage's formulation, from P1-P6 he derives The

... (read more)
2Sniffnoy1yOops, turns out I did misremember -- Savage does not in fact put the proof in his book. You have to go to Fishburn's book. I've been reviewing all this recently and yeah -- for anyone else who wants to get into this, I'd reccommend getting Fishburn's book ("Utility Theory for Decision Making") in addition to Savage's "Foundations of Statistics". Because in addition to the above, what I'd also forgotten is that Savage leaves out a bunch of the proofs. It's really annoying. Thankfully in Fishburn's treatment he went and actually elaborated all the proofs that Savage thought it OK to skip over... (Also, stating the obvious, but get the second edition of "Foundations of Statistics", as it fixes some mistakes. You probably don't want just Fishburn's book, it's fairly hard to read by itself.)
Underappreciated points about utility functions (of both sorts)

Fishburn (op. cit., following Blackwell and Girschick, an inaccessible source) requires that the set of gambles be closed under infinitary convex combinations.

Again, I'm simply not seeing this in the paper you linked? As I said above, I simply do not see anything like that outside of section 9, which is irrelevant. Can you point to where you're seeing this condition?

I shall take a look at Savage's axioms and see what in them is responsible for the same thing.

In the case of Savage, it's not any particular axiom, but rather the setup. An action is

... (read more)
2Richard_Kennaway1yIn Fishburn's "Bounded Expected Utility", page 1055, end of first paragraph (as cited previously): That depends on some earlier definitions, e.g. is a certain set of probability distributions (the “d” stands for “discrete”) defined with reference to some particular -algebra, but the important part is that last infinite sum: this is where all infinitary convex combinations are asserted to exist. Whether that is assigned to "background setup" or "axioms" does not matter. It has to be present, to allow the construction of St. Petersburg gambles. Will address the rest of your comments later.
Underappreciated points about utility functions (of both sorts)

Huh. This would need some elaboration, but this is definitely the most plausible way around the problem I've seen.

Now (in Savage's formalism) actions are just functions from world-states to outcomes (maybe with a measurability condition), so regardless of your prior it's easy to construct the relevant St. Petersburg gambles if the utility function is unbounded. But seems like what you're saying is, if we don't allow arbitrary actions, then the prior could be such that, not only are none of the permitted actions St. Petersburg gambles, but also this remains the case even after future updates. Interesting! Yeah, that just might be workable...

Underappreciated points about utility functions (of both sorts)

OK, so going by that you're suggesting, like, introducing varying caps and then taking limits as the cap goes to infinity? It's an interesting idea, but I don't see why one would expect it to have anything to do with preferences.

1Isnasene1yYes, I think that's a good description. In my case, it's a useful distinction because I'm the kind of person who thinks that showing that a real thing is infinite requires an infinite amount of information. This means I can say things like "my utility function scales upward linearly with the number of happy people" without things breaking because it is essentially impossible to convince me that any set of finite action could legitimately cause a literally infinite number of happy people to exist. For people who believe they could achieve actually infinitely high values in their utility functions, the issues you point out still hold. But I think my utility function is bounded by something eventually even if I can't tell you what that boundary actually is.
Underappreciated points about utility functions (of both sorts)

You should check out Abram's post on complete class theorems. He specifically addresses some of the concerns you mentioned in the comments of Yudkowsky's posts.

So, it looks to me like what Abrams is doing -- once he gets past the original complete class theorem -- is basically just inventing some new formalism along the lines of Savage. I think it is very misleading to refer to this as "the complete class theorem" -- how on earth was I supposed to know that this was what was being referred to when "the complete class theorem" was mentioned, when it res

... (read more)
Underappreciated points about utility functions (of both sorts)

I think you've misunderstood a fair bit. I hope you don't mind if I address this slightly out of order.

Or if infinite utilities are not immediately a problem, then by a more complicated argument, involving constructing multiple St. Petersburg-type combinations and demonstrating that the axioms imply that there both should and should not be a preference between them.

This is exactly what Fishburn does, as I mentioned above. (Well, OK, I didn't attribute it to Fishburn, I kind of implicitly misattributed it to Savage, but it was actually Fishburn; I did

... (read more)
Or if you have some formalism where preferences can be undefined (in a way that is distinct from indifference), by all means explain it... (but what happens when you program these preferences into an FAI and it encounters this situation? It has to pick. Does it pick arbitrarily? How is that distinct from indifference?)

A short answer to this (something longer later) is that an agent need not have preferences between things that it is impossible to encounter. The standard dissolution of the St. Petersberg paradox is that nobody can offer that gamble. Even th... (read more)

Underappreciated points about utility functions (of both sorts)

Is there a reason we can't just solve this by proposing arbitrarily large bounds on utility instead of infinite bounds? For instance, if we posit that utility is bounded by some arbitrarily high value X, then the wager can only payout values X for probabilities below 1/X.

I'm not sure what you're asking here. An individual decision-theoretic utility function can be bounded or it can be unbounded. Since decision-theoretic utility functions can be rescaled arbitrarily, naming a precise value for the bounds is meaningless; so like we could just assume the

... (read more)
1Isnasene1ySay our utility function assigns an actual thing in the universe with value V1 and the utility function is bounded by value X. What I'm saying is that we can make the problem go away by assuming bounded utility but without actually having to define the ratio between V1 and X as a specific finite number (this would not change upon scaling). This means that, if your utility function is something like "number of happy human beings", you don't have to worry about your utility function breaking if the maximum number of happy human beings is larger than you expected since you never have to define such an expectation. See my sub-sub-reply to Eigil Rischel's sub-reply for elaboration.
Underappreciated points about utility functions (of both sorts)

Yes, thanks, I didn't bother including it in the body of the post but that's basically how it goes. Worth noting that this:

Both of these wagers have infinite expected utility, so we must be indifferent between them.

...is kind of shortcutting a bit (at least as Savage/Fishburn[0] does it; he proves indifference between things of infinite expected utility separately after proving that expected utility works when it's finite), but that is the essence of it, yes.

(As for the actual argument... eh, I don't have it in front of me and don't feel like rederivi

... (read more)
Underappreciated points about utility functions (of both sorts)

By "a specific gamble" do you mean "a specific pair of gambles"? Remember, preferences are between two things! And you hardly need a utility function to express a preference between a single pair of gambles.

I don't understand how to make sense of what you're saying. Agent's preferences are the starting point -- preferences as in, given a choice between the two, which do you pick? It's not clear to me how you have a notion of preference that allows for this to be undefined (the agent can be indifferent, but that's distinct).

I mean, you could try to come

... (read more)
3FactorialCode1yThis is true, then it would only be between a specific subset of gambles. I think you should be able to set things up so that you never encounter a pair of gambles where this is undefined. I'll illustrate with an example. Suppose you start with a prior over the integers, such that: p(n) = (C/F(n)) where F(n) is a function that grows really fast and C is a normalization constant. Then the set of gambles that we're considering would be posteriors on the integers given that they obey certain properties. For instance, we could ask the agent to choose between the posterior over integers given that n is odd vs the posterior given that n is even. I'm pretty sure that you can construct an agent that behaves as if it had an unbounded utility function in this case. So long as the utility associated with an integer n grows sufficiently slower than F(N), all expectations over posteriors on the integers should be well defined. If you were to build an FAI this way, it would never end up in a belief state where the expected utility diverges between two outcomes. The expected utility would be well defined over any posterior on it's prior, so it's choice given a pair of gambles would also be well defined for any belief state it could find itself in.
Underappreciated points about utility functions (of both sorts)

If you're not making a prioritarian aggregate utility function by summing functions of individual utility functions, the mapping of a prioritarian function to a utility function doesn't always work. Prioritarian utility functions, for instance, can do things like rank-order everyone's utility functions and then sum each individual utility raised to the negative-power of the rank-order ... or something*. They allow interactions between individual utility functions in the aggregate function that are not facilitated by the direct summing permitted in utilita

... (read more)
Misconceptions about continuous takeoff

I don't really want to go trying to defend here a position I don't necessarily hold, but I do have to nitpick and point out that there's quite a bit of room inbetween exponential and hyperbolic.

Misconceptions about continuous takeoff

To be clear, intelligence explosion via recursive self-improvement has been distinguished from merely exponential growth at least as far back as Yudkowsky's "Three Major Singularity Schools". I couldn't remember the particular link when I wrote the comment above, but, well, now I remember it.

Anyway, I don't have a particular argument one way or the other; I'm just registering my surprise that you encountered people here arguing for merely exponential growth base on intelligence explosion arguments.

2Matthew Barnett1yEmpirically, most systems with a feedback loop don't grow hyperbolically. I would need strong theoretical reasons in order to understand why this particular distinction is important.
Bayesian examination

Yeah, proper scoring rules (and in particular both the quadratic/Brier and the logarithmic examples) have been discussed here a bunch, I think that's worth acknowledging in the post...

2ChristianKl1yUsually we discussed them in the past for binary choices. I don't remember any previous discussion on using proper scoring rules for multiple choice tests.
1Gurkenglas1yI agree. Proper scoring rules were introduced to this community 14 years ago [http://yudkowsky.net/rational/technical].
Misconceptions about continuous takeoff

It is sometimes argued that even if this advantage is modest, the growth curves will be exponential, and therefore a slight advantage right now will compound to become a large advantage over a long enough period of time. However, this argument by itself is not an argument against a continuous takeoff.

I'm not sure this is an accurate characterization of the point; my understanding is that the concern largely comes from the possibility that the growth will be faster than exponential, rather than merely exponential.

1Matthew Barnett1ySure, if someone was arguing that, then they have a valid understanding of the difference between continuous vs. discontinuous takeoff. I would just question the assumption why we should expect growth to be faster than exponential for any sustained period of time.
Goal-thinking vs desire-thinking

I mean, are you actually disagreeing with me here? I think you're just describing an intermediate position.

2gjm1yI don't know for sure whether we're really disagreeing. Perhaps that's a question with no definite answer; the question's about where best to draw the boundary of an only-vaguely-defined term. But it seems like you're saying "goal-thinking must only be concerned with goals that don't involve people's happiness" and I'm saying I think that's a mistake and that the fundamental distinction is between doing something as part of a happiness-maximizing process and recognizing the layer of indirection in that and aiming at goals we can see other reasons for, which may or may not happen to involve our or someone else's happiness. Obviously you can choose to focus only on goals that don't involve happiness in any way at all, and maybe doing so makes some of the issues clearer. But I don't think "involving happiness" / "not involving happiness" is the most fundamental criterion here; the distinction is actually, as your original terminology makes clear, between different modes of thinking.
Goal-thinking vs desire-thinking

OK. I think I didn't think through my reply sufficiently. Something seemed off with what you were saying, but I failed to think through what and made a reply that didn't really make sense instead. But thinking things through a bit more now I think I can lay out my actual objection a bit more clearly.

I definitely think that if you're taking the point of view that suicide is preferable to suffering you're not applying what I'm calling goal-thinking. (Remember here that the description I laid out above is not intended as some sort of intensional definition

... (read more)
2gjm1yI see things slightly differently. Happiness, suffering, etc., function as internal estimators of goal-met-ness. Like a variable in a computer program that indicates how you're doing. Hence, trying to optimize happiness directly runs the risk of finding ways to change the value of the variable without the corresponding real-world things the variable is trying to track. So far, so good. But! That doesn't mean that happiness can't also be a thing we care about. If I can arrange for someone's goals to be 50% met and for them to feel either as if they're 40% met or as if they're 60% met, I probably choose the latter; people like feeling as if their goals are met, and I insist that it's perfectly reasonable for me to care about that as well as about their actual goals. For that matter, if someone has goals I find terrible, I may actually prefer their goals to go unmet but for them still to be happy. I apply the same to myself -- within reason, I would prefer my happiness to overestimate rather than underestimate how well my goals are being met -- but obviously treating happiness as a goal is more dangerous there because the risk of getting seriously decoupled from my goals is greater. (I think.) I don't think it's necessary to see nonexistence as neutral in order to prefer (in some cases, perhaps only very extreme ones) nonexistence to existence-with-great-suffering. Suffering is unpleasant. People hate it and strive to avoid it. Yes, the underlying reason for that is because this helps them achieve other goals, but I am not obliged to care only about the underlying reason. (Just as I'm not obliged to regard sex as existing only for the sake of procreation.)
Goal-thinking vs desire-thinking

This is perhaps an intermediate example, but I do think that once you're talking about internal experiences to be avoided, it's definitely not all the way at the goal-thinking end.

4gjm1yI'm not convinced. To me, at least, my goals that are about me don't feel particularly different in kind from my goals that are about other people, nor do my goals that are about experiences feel particularly different from my goals that are about things other than experiences. (It's certainly possible to draw your dividing line between, say, "what you want for yourself" and "what other things you want", but I think that's an entirely different line from the one drawn in the OP.)
Goal-thinking vs desire-thinking

Hm, I suppose that's true. But I think the overall point still stands? It's illustrating a type of thinking that doesn't make sense to one thinking in terms of concrete, unmodifiable goals in the external world.

2gjm1yIs that really true? If you can have "have other people not suffer horribly" as a goal, you can have "not suffer horribly yourself" as a goal too. And if, on balance, your life seems likely to involve a lot of horrible suffering, then suicide might absolutely make sense even though it would reduce your ability to achieve your other goals.
Coherent decisions imply consistent utilities

So this post is basically just collecting together a bunch of things you previously wrote in the Sequences, but I guess it's useful to have them collected together.

I must, however, take objection to one part. The proper non-circular foundation you want for probability and utility is not the complete class theorem, but rather Savage's theorem, which I previously wrote about on this website. It's not short, but I don't think it's too inaccessible.

Note, in particular, that Savage's theorem does not start with any assumption baked in that R is the correct sy

... (read more)
Noticing Frame Differences

Thirding what the others said, but I wanted to also add that rather than actual game theory, what you may be looking here may instead be the anthropological notion of limited good?

The Forces of Blandness and the Disagreeable Majority

Sorry, but: The thing at the top says this was crossposted from Otium, but I see no such post there. Was this meant to go up there as well? Because it seems to be missing.

Looks like Sarah took that post down from Otium. Will move it back into drafts until I hear back from her about what she wants to do with it.

(moved it back, Sarah says it's fine)

An Extensive Categorisation of Infinite Paradoxes

OK, time to actually now get into what's wrong with the ones I skipped initially. Already wrote the intro above so not repeating that. Time to just go.

Infinitarian paralysis: So, philosophical problems to start: As an actual decision theory problem this is all moot since you can't actually have an infinite number of people. I.e. it's not clear why this is a problem at all. Secondly, naive assumption of utilitarian aggregation as mentioned above, etc, not going over this again. Enough of this, let's move on.

So what are the mathematical problems here? W

... (read more)
An Extensive Categorisation of Infinite Paradoxes

OK, time for the second half, where I get to the errors in the ones I initially skipped. And yes, I'm going to assert some philosophical positions which (for whatever reason) aren't well-accepted on this site, but there's still plenty of mathematical errors to go around even once you ignore any philosphical problems. And yeah, I'm still going to point out missing formalism, but I will try to focus on the more substantive errors, of which there are plenty.

So, let's get those philosophical problems out of the way first, and quickly review utility functions

... (read more)
3Sniffnoy2yOK, time to actually now get into what's wrong with the ones I skipped initially. Already wrote the intro above so not repeating that. Time to just go. Infinitarian paralysis: So, philosophical problems to start: As an actual decision theory problem this is all moot since you can't actually have an infinite number of people. I.e. it's not clear why this is a problem at all. Secondly, naive assumption of utilitarian aggregation as mentioned above, etc, not going over this again. Enough of this, let's move on. So what are the mathematical problems here? Well, you haven't said a lot here, but here's what it's look like to me. I think you've written one thing here that is essentially correct, which is that, if you did have some system of surreal valued-utilities, it would indeed likely make the distinction you want. But, once again, that's a big "if", and not just for philosophical reasons but for the mathematical reasons I've already brought up so many times right now -- you can't do infinite sums in the surreals like you want, for reasons I've already covered [https://www.lesswrong.com/posts/XW6Qi2LitMDb2MF8c/a-rationality-condition-for-cdt-is-that-it-equal-edt-part-1#ceToiLikEpRmFrAHc] . So there's a reason I included the word "likely" above, because if you did find an appropriate way of doing such a sum, I can't even necessarily guarantee that it would behave like you want (yes, finite sums should, but infinite sums require definition, and who knows if they'll actually be compatible with finite sums like they should be?). But the really jarring thing here, the thing that really exposes a serious error in your thought (well, OK, that does so to a greater extent), is not in your proposed solution -- it's in what you contrast it with. Cardinal valued-utilities? Nothing about that makes sense! That's not a remotely well-defined alternative you can contrast with! And the thing that bugs me about this error is that it's just so unforced -- I mean, man, you could hav
An Extensive Categorisation of Infinite Paradoxes

My primary response to this comment will take the form of a post, but I should add that I wrote: "I will provide informal hints on how surreal numbers could help us solve some of these paradoxes, although the focus on this post is primarily categorisation, so please don't mistake these for formal proofs".

You're right; I did miss that, thanks. It was perhaps unfair of me then to pick on such gaps in formalism. Unfortunately, this is only enough to rescue a small portion in the post. Ignoring the ones I skipped -- maybe it would be worth my time to get

... (read more)
An Extensive Categorisation of Infinite Paradoxes

Almost nothing in this post is correct. This post displays not just a misuse of and failure to understand surreal numbers, but a failure to understand cardinals, ordinals, free groups, lots of other things, and just how to think about such matters generally, much as in our last exchange. The fact that (as I write this) this is sitting at +32 is an embarrassment to this website. You really, really, need to go back and relearn all of this from scratch, because going by this post you don't have the slightest clue what you're talking about. I would encoura

... (read more)
9Sniffnoy2yOK, time for the second half, where I get to the errors in the ones I initially skipped. And yes, I'm going to assert some philosophical positions which (for whatever reason) aren't well-accepted on this site, but there's still plenty of mathematical errors to go around even once you ignore any philosphical problems. And yeah, I'm still going to point out missing formalism, but I will try to focus on the more substantive errors, of which there are plenty. So, let's get those philosophical problems out of the way first, and quickly review utility functions and utilitarianism, because this applies to a bunch of what you discuss here. Like, this whole post takes a very naive view of the idea of "utility", and this needs some breaking down. Apologies if you already know all of what I'm about to say, but I think given the context it bears repeating. So: There are two different things meant by "utility function". The first is decision-theoretic; an agent's utility function is a function whose expected value it attempts to maximize. The second is the one used by utilitarianism, which involves (at present, poorly-defined) "E-utility" functions, which are not utility functions in the decision-theoretic sense, that are then somehow aggregated (maybe by addition? who knows?) into a decision-theoretic utility function. Yes, this terminology is terribly confusing. But these are two separate things and need to be kept separate. Basically, any agent that satisfies appropriate rationality conditions has a utility function in the decision-theoretic sense (obviously such idealized agents don't actually exist, but it's still a useful abstraction). So you could say, roughly speaking, any rational consequentialist has a decision-theoretic utility function. Whereas E-utility is specifically a utilitarian notion, rather than a general consequentalist or purely descriptive notion like decision-theoretic utility (it's also not at all clear how to define it). Anyway, if you want surreal
9Chris_Leong2yMy primary response to this comment will take the form of a post, but I should add that I wrote: "I will provide informal hints on how surreal numbers could help us solve some of these paradoxes, although the focus on this post is primarily categorisation, so please don't mistake these for formal proofs". Your comment seems to completely ignore this stipulation. Take for example this: Yes, there's a lot of philosophical groundwork that would need to be done to justify the surreal approach. That's why I said that it was only an informal hint. Yes, I actually did look up that there was a way of defining 2^s where s is a surreal number. I wrote a summary of a paper [https://www.lesswrong.com/posts/MtYDr7zyicq4J7rWe/summary-surreal-decisions] by Chen and Rubio that provides the start of a surreal decision theory. This isn't a complete probability theory as it only supports finite additivity instead of countable additivity, but it suggests that this approach might be viable. I could keep going, but I think I've made my point that you're evaluating these informal comments as though I'd claimed they were a formal proof. This post was already long enough and took enough time to write as is. I will admit that I could have been clearer that many of these remarks were speculative, in the sense of being arguments that I believed were worth working towards formalising, even if all of the mathematical machinery doesn't necessarily exist at this time. My point is that justifying the use of surreals numbers doesn't necessarily involve solving every paradox; it should also be persuasive to solve a good number of them and then to demonstrate that there is good reason to believe that we may be able to solve the rest in the future. In this sense, informal arguments aren't valueless.
2Chris_Leong2yThis is quite a long post, so it may take some time to write a proper reply, but I'll get back to you when I can. The focus of this post was on gathering together all the infinite paradoxes that I could manage. I also added some informal thoughts on how surreal numbers could help us conceptualise the solution to these problems, although this wasn't the main focus (it was just convenient to put them in the same space). Unfortunately, I haven't continued the sequence since I've been caught up with other things (travel, AI, applying for jobs), but hopefully I'll write up some new posts soon. I've actually become much less optimistic about surreal numbers for philosophical reasons which I'll write up soon. So my intent is for my next post to examine the definition of infinity and why this makes me less optimistic about this approach. After that, I want to write up a bit more formally how the surreal approach would work, because even though I'm less optimistic about this approach, perhaps someone else will disagree with my pessimism. Further, I think it's useful to understand how the surreal approach would try to resolve these problems, even if only to provide a solid target for criticism.
Load More