Classifying games like the Prisoner's Dilemma

I'm reminded of this paper, which discusses a smaller set of two-player games. What you call "Cake Eating" they call the "Harmony Game". They also use the more suggestive variable names -- which I believe come from existing literature -- R (reward), S (sucker's payoff), T (temptation), P (punishment) instead of (W, X, Y, Z). Note that in addition to R > P (W > Z) they also added the restrictions T > P (Y > Z) and R > S (W > X) so that the two options could be meaningfully labeled "cooperate" and "defect" instead of "Krump" and "Flitz" ... (read more)

Thirty-three randomly selected bioethics papers

I suppose so. It is at least a *different* problem than I was worried about...

Thirty-three randomly selected bioethics papers

Huh. Given the negative reputation of bioethics around here -- one I hadn't much questioned, TBH -- most of these are suprisingly reasonable. Only #10, #16, and #24 really seemed like the LW stereotype of the bioethics paper that I would roll my eyes at. Arguably also #31, but I'd argue that one is instead alarming in a *different* way.

Some others seemed like bureaucratic junk (so, neither good nor bad), and others I think the quoted sections didn't really give enough information to judge; it is quite possible that a few more of these would go under the s... (read more)

42moWouldn't the presence of "bureaucratic junk" be evidence towards a field having
problems?

Jean Monnet: The Guerilla Bureaucrat

Consider a modified version of the prisoner's dilemma. This time, the prisoners are allowed to communicate, but they also have to solve an additional technical problem, say, how to split the loot. They may start with agreeing on not betraying each other to the prosecutors, but later one of them may say: "I've done most of the work. I want 70% of the loot, otherwise I am going to rat on you." It's easy to see how the problem would escalate and end up in the prisoners betraying each other.

Minor note, but I think you could just talk about a [bargaining gam... (read more)

Dark Matters

I just explained why (without more specific theories of in exactly what way the gravity would become delocalized from the visible mass) the bullet cluster is *not* evidence one way or the other.

Now, you compare the extra fields of modified gravity to epicycles -- as in, post-hoc complications grafted on to a theory to explain a particular phenomenon. But these extra fields are, to the best of my understanding, not grafted on to explain such delocalization; they're the actual basic content of the modified gravity theories and necessary to obtain a workable t... (read more)

Defending the non-central fallacy

I feel like this really misses the point of the whole "non-central fallacy" idea. I would say, categories are heuristics and those heuristics have limits. When the category gets strained, the thing to do is to stop arguing using the category and start arguing the particular facts without relation to the category ("taboo your words").

You're saying that this sort of arguing-via-category is useful because it's actually aguing-via-similarity; but I see the point of Scott/Yvain's original article being that such arguing via similarity simply isn't useful in s... (read more)

Dark Matters

Good post. Makes a good case. I wasn't aware of the evidence from galactic cluster lensing; that's pretty impressive. (I guess not as much as the CMB power spectrum, but that I'd heard about before. :P )

But, my understanding is that the Bullet Cluster is actually not the strong evidence it's claimed to be? My understanding of modified gravity theories is that, since they all work by adding extra fields, it's also possible for those to have gravity separated from visible matter, even if no dark matter is present. (See e.g.. here... of course in this po... (read more)

02moAnd with enough epicycles you can fit the motion of planets with geocentricism.
If MOND supporters can dismiss Bullet Cluster they'll dismiss any future
evidence, too.

Blue is arbitrary

"Cyan" isn't a basic color term in English; English speakers ordinarily consider cyan to be a variant of blue, not something basically separate. Something that is cyan could also be described in English as "blue". As opposed to say, red and pink -- these are both basic color terms in English; an English speaker would not ordinarily refer to something pink as "red", or vice versa.

Or in other words: Color words don't refer to *points* in color space, they refer to *regions*, which means that you can look at how those regions overlap -- some may be subsets of o... (read more)

Making Vaccine

Wow!

I guess a thing that still bugs me after reading the rest of the comments is, if it turns out that this vaccine only offers protection against inhaling the virus though the nose, how much does that help when one considers that one could also inhale it through the mouth? Like, I worry that after taking this I'd still need to avoiding indoor spaces with other people, etc, which would defeat a lot of the benefit of it.

But, if it turns out that it does yield antibodies in the blood, then... this sounds very much worth trying!

83moMy understanding is that it helps a lot.
The biggest benefit seems to be that the immune system is primed in at least
some fashion; it knows what to look for, and it has readily available tools that
should be effective. It doesn't have to take a day or a week to try random
things before it finally discovers a particularly effective antibody and gets
the production chain ramped up to start a proper immune response.
Instead, your immune system will very quickly get a signal it understands as bad
and can immediately start ramping up when it does detect the virus.
Keep in mind that the commercial vaccines don't have 100% success rate in that
some people still get sick, but the 'priming' of the immune response is still
there. I believe this is why the death rate / severe complications rate is
effectively zero for immunized patients, even though it's possible to get sick.
(Again, my understanding. I would very much appreciate correction/clarifications
here.)

Most Prisoner's Dilemmas are Stag Hunts; Most Stag Hunts are Schelling Problems

So, why do we perceive so many situations to be Prisoner's Dilemma -like rather than Stag Hunt -like?

I don't think that we do, exactly. I think that most people only know the term "prisoners' dilemma" and haven't learned any more game theory than that; and then occasionally they go and actually attempt to map things onto the Prisoners' Dilemma as a result. :-/

Toolbox-thinking and Law-thinking

That sounds like it might have been it?

Swiss Political System: More than You ever Wanted to Know (III.)

Sorry, but after reading this I'm not very clear on just what exactly the "Magic Formula" refers to. Could you state it explicitly?

69moFixed:

Underappreciated points about utility functions (of both sorts)

Oops, turns out I *did* misremember -- Savage does not in fact put the proof in his book. You have to go to Fishburn's book.

I've been reviewing all this recently and yeah -- for anyone else who wants to get into this, I'd reccommend getting Fishburn's book ("Utility Theory for Decision Making") in addition to Savage's "Foundations of Statistics". Because in addition to the above, what I'd also forgotten is that *Savage leaves out a bunch of the proofs*. It's really annoying. Thankfully in Fishburn's treatment he went and actually elaborated all the proofs

What Money Cannot Buy

Oh, I see. I misread your comment then. Yes, I am assuming one already has the ability to discern the structure of an argument and doesn't need to hire someone else to do that for you...

What Money Cannot Buy

What I said above. Sorry, to be clear here, by "argument structure" I don't mean the structure of the individual arguments but rather the overall argument -- what rebuts what.

(Edit: Looks like I misread the parent comment and this fails to respond to it; see below.)

41yTo be clear as well, the rhetorical point underneath my question is that I don't
think your heuristic is all that useful, and seems grounded in generalization
from too few examples without searching for counterexamples. Rather than just
attacking it directly like Gordon, I was trying to go up a meta-level, to just
point at the difficulty of 'buying' methods of determining expertise, because
you need to have expertise in distinguishing the market there.
(In general, when someone identifies a problem and you think you have a
solution, it's useful to consider whether your solution suffers from that
problem on a different meta-level; sometimes you gain from sweeping the
difficulty there, and sometimes you don't.)

What Money Cannot Buy

This is a good point (the redemption movement comes to mind as an example), but I think the cases I'm thinking of and the cases you're describing look quite different in other details. Like, the bored/annoyed expert tired of having to correct basic mistakes, vs. the salesman who wants to initiate you into a new, exciting secret. But yeah, this is only a quick-and-dirty heuristic, and even then only good for distinguishing snake oil; it might not be a good idea to put too much weight on it, and it definitely won't help you in a real dispute ("Wait, *both* s

What Money Cannot Buy

Given a bunch of people who disagree, some of whom are actual experts and some of whom are selling snake oil, expertise yourself, there are some further quick-and-dirty heuristics you can use to tell which of the two groups is which. I think basically my suggestion can be best summarized at "look at argument structure".

The real experts will likely spend a bunch of time correct popular misconceptions, which the fakers may subscribe to. By contrast, the fakers will generally not bother "correcting" the truth to their fakery, because why would they? They'r

... (read more)21yHere's another: probing into their argument structure a bit and checking if they
can keep it from collapsing under its own weight.
https://www.lesswrong.com/posts/wyyfFfaRar2jEdeQK/entangled-truths-contagious-lies
[https://www.lesswrong.com/posts/wyyfFfaRar2jEdeQK/entangled-truths-contagious-lies]

41yAnd how does one distinguish snake oil salesmen and real experts when it comes
to identifying argument structure and what it implies?

The real experts will likely spend a bunch of time correct popular misconceptions, which the fakers may subscribe to. By contrast, the fakers will generally not bother "correcting" the truth to their fakery, because why would they? They're trying to sell to unreflective people who just believe the obvious-seeming thing; someone who actually bothered to read corrections to misconceptions at any point is likely too savvy to be their target audience.

Using this as a heuristic would often backfire on you as stated, because there's a certain ... (read more)

71yThis seems to rely on the fakes knowing they are fakes. I agree that is a
problem and your heuristic useful but I think we (non-experts) are still stuck
with the problem of separating out the real experts from those that mistakenly
think they are also real experts. Those will likely attempt to correct the true
security approach according to their mistaken premises and solutions. We're
still stuck with the problem that money doesn't get the non-expert client too
far.
Now, you've clearly been able to reduce the ratio of real solution to snake oil
so moved the probabilities in your favor when throwing money at the problems but
not sure just how far.

Underappreciated points about utility functions (of both sorts)

Well, it's worth noting that P7 is introduced to address gambles with infinitely many possible outcomes, regardless of whether those outcomes are bounded or not (which is the reason I argue above you can't just get rid of it). But yeah. Glad that's cleared up now! :)

Underappreciated points about utility functions (of both sorts)

Ahh, thanks for clarifying. I think what happened was that your modus ponens was my modus tollens -- so when I think about my preferences, I ask "what conditions do my preferences need to satisfy for me to avoid being exploited or undoing my own work?" whereas you ask something like "if my preferences need to correspond to a bounded utility function, what should they be?" [1]

That doesn't seem right. The whole point of what I've been saying is that we can write down some simple conditions that ought to be true in order to avoid being exploitable or othe

... (read more)41yThanks for the reply. I re-read your post and your post on Savage's proof and
you're right on all counts. For some reason, it didn't actually click for me
that P7 was introduced to address unbounded utility functions and boundedness
was a consequence of taking the axioms to their logical conclusion.

A summary of Savage's foundations for probability and utility.

Here's a quick issue I only just noticed but which fortunately is easily fixed:

Above I mentioned you probably want to restrict to a sigma-algebra of events and only allow measurable functions as actions. But, what does measurable mean here? Fortunately, the ordering on outcomes (even without utility) makes measurability meaningful. Except this puts a circularity in the setup, because the ordering on outcomes is induced from the ordering on actions.

Fortunately this is easily patched. You can start with the assumption of a total preorder on outcomes (con

... (read more)Underappreciated points about utility functions (of both sorts)

(This is more properly a followup to my sibling comment, but posting it here so you'll see it.)

I already said that I think that thinking in terms of infinitary convex combinations, as you're doing, is the wrong way to go about it; but it took me a bit to put together why that's definitely the wrong way.

Specifically, it assumes probability! Fishburn, in the paper you link, assumes probability, which is why he's able to talk about why infinitary convex combinations are or are not allowed (I mean, that and the fact that he's not necessarily arbitrary actions

... (read more)21ySavage doesn't assume probability or utility, but their construction is a
mathematical consequence of the axioms. So although they come later in the
exposition, they mathematically exist as soon as the axioms have been stated.
I am still thinking about that, and may be some time.
As a general outline of the situation, you read P1-7 => bounded utility as modus
ponens: you accept the axioms and therefore accept the conclusion. I read it as
modus tollens: the conclusion seems wrong, so I believe there is a flaw in the
axioms. In the same way, the axioms of Euclidean geometry seemed very plausible
as a description of the physical space we find ourselves in, but conflicts
emerged with phenomena of electromagnetism and gravity, and eventually they were
superseded as descriptions of physical space by the geometry of differential
manifolds.
It isn't possible to answer the question "which of P1-7 would I reject?" What is
needed to block the proof of bounded utility is a new set of axioms, which will
no doubt imply large parts of P1-7, but might not imply the whole of any one of
them. If and when such a set of axioms can be found, P1-7 can be re-examined in
their light.

Underappreciated points about utility functions (of both sorts)

Apologies, but it sounds like you've gotten some things mixed up here? The issue is boundedness of utility functions, not whether they can take on infinity as a value. I don't think anyone here is arguing that utility functions don't need to be finite-valued. All the things you're saying seem to be related to the latter question rather than the former, or you seem to be possibly conflating them?

In the second paragraph perhaps this is just an issue of language -- when you say "infinitely high", do you actually mean "aribtrarily high"? -- but in the first

... (read more)11yAhh, thanks for clarifying. I think what happened was that your modus ponens was
my modus tollens -- so when I think about my preferences, I ask "what conditions
do my preferences need to satisfy for me to avoid being exploited or undoing my
own work?" whereas you ask something like "if my preferences need to correspond
to a bounded utility function, what should they be?" [1]. As a result, I went on
a tangent about infinity to begin exploring whether my modified notion of a
utility function would break in ways that regular ones wouldn't.
I agree, one shouldn't conclude anything without a theorem. Personally, I would
approach the problem by looking at the infinite wager comparisons discussed
earlier and trying to formalize them into additional rationality condition. We'd
need
* an axiom describing what it means for one infinite wager to be "strictly
better" than another.
* an axiom describing what kinds of infinite wagers it is rational to be
indifferent towards
Then, I would try to find a decisioning-system that satisfies these new
conditions as well as the VNM-rationality axioms (where VNM-rationality
applies). If such a system exists, these axioms would probably bar it from being
represented fully as a utility function. If it didn't, that'd be interesting. In
any case, whatever happens will tell us more about either the structure our
preferences should follow or the structure that our rationality-axioms should
follow (if we cannot find a system).
Of course, maybe my modification of the idea of a utility function turns out to
show such a decisioning-system exists by construction. In this case, modifying
the idea of a utility function would help tell me that my preferences should
follow the structure of that modification as well.
Does that address the question?
[1] From your post:

Underappreciated points about utility functions (of both sorts)

Oh, so that's what you're referring to. Well, if you look at the theorem statements, you'll see that P=P_d is an axiom that is explicitly called out in the theorems where it's assumed; it's *not* implictly part of Axiom 0 like you asserted, nor is it more generally left implicit at all.

... (read more)but the important part is that last infinite sum: this is where all infinitary convex combinations are asserted to exist. Whether that is assigned to "background setup" or "axioms" does not matter. It has to be present, to allow the construction of St. Petersburg gambles.

Underappreciated points about utility functions (of both sorts)

Savage does not actually prove bounded utility. Fishburn did this later, as Savage footnotes in the edition I'm looking at, so Fishburn must be tackled.

Yes, it was actually Fishburn that did that. Apologies if I carelessly implied it was Savage.

IIRC, Fishburn's proof, formulated in Savage's terms, is in Savage's book, at least if you have the second edition. Which I think you must, because otherwise that footnote wouldn't be there at all. But maybe I'm misremembering? I think it has to be though...

... (read more)In Savage's formulation, from P1-P6 he derives The

21yOops, turns out I did misremember -- Savage does not in fact put the proof in
his book. You have to go to Fishburn's book.
I've been reviewing all this recently and yeah -- for anyone else who wants to
get into this, I'd reccommend getting Fishburn's book ("Utility Theory for
Decision Making") in addition to Savage's "Foundations of Statistics". Because
in addition to the above, what I'd also forgotten is that Savage leaves out a
bunch of the proofs. It's really annoying. Thankfully in Fishburn's treatment he
went and actually elaborated all the proofs that Savage thought it OK to skip
over...
(Also, stating the obvious, but get the second edition of "Foundations of
Statistics", as it fixes some mistakes. You probably don't want just Fishburn's
book, it's fairly hard to read by itself.)

Underappreciated points about utility functions (of both sorts)

Fishburn (op. cit., following Blackwell and Girschick, an inaccessible source) requires that the set of gambles be closed under infinitary convex combinations.

Again, I'm simply not seeing this in the paper you linked? As I said above, I simply do not see anything like that outside of section 9, which is irrelevant. Can you point to where you're seeing this condition?

I shall take a look at Savage's axioms and see what in them is responsible for the same thing.

In the case of Savage, it's not any particular axiom, but rather the setup. An action is

... (read more)21yIn Fishburn's "Bounded Expected Utility", page 1055, end of first paragraph (as
cited previously):
That depends on some earlier definitions, e.g. is a certain set of probability
distributions (the “d” stands for “discrete”) defined with reference to some
particular -algebra, but the important part is that last infinite sum: this is
where all infinitary convex combinations are asserted to exist. Whether that is
assigned to "background setup" or "axioms" does not matter. It has to be
present, to allow the construction of St. Petersburg gambles.
Will address the rest of your comments later.

Underappreciated points about utility functions (of both sorts)

Huh. This would need some elaboration, but this is definitely the most plausible way around the problem I've seen.

Now (in Savage's formalism) actions are just functions from world-states to outcomes (maybe with a measurability condition), so regardless of your prior it's easy to construct the relevant St. Petersburg gambles if the utility function is unbounded. But seems like what you're saying is, if we *don't* allow arbitrary actions, then the prior could be such that, not only are none of the permitted actions St. Petersburg gambles, but also this remains the case even after future updates. Interesting! Yeah, that just might be workable...

Underappreciated points about utility functions (of both sorts)

OK, so going by that you're suggesting, like, introducing varying caps and then taking limits as the cap goes to infinity? It's an interesting idea, but I don't see why one would expect it to have anything to do with preferences.

11yYes, I think that's a good description.
In my case, it's a useful distinction because I'm the kind of person who thinks
that showing that a real thing is infinite requires an infinite amount of
information. This means I can say things like "my utility function scales upward
linearly with the number of happy people" without things breaking because it is
essentially impossible to convince me that any set of finite action could
legitimately cause a literally infinite number of happy people to exist.
For people who believe they could achieve actually infinitely high values in
their utility functions, the issues you point out still hold. But I think my
utility function is bounded by something eventually even if I can't tell you
what that boundary actually is.

Underappreciated points about utility functions (of both sorts)

You should check out Abram's post on complete class theorems. He specifically addresses some of the concerns you mentioned in the comments of Yudkowsky's posts.

So, it looks to me like what Abrams is doing -- once he gets past the original complete class theorem -- is basically just inventing some new formalism along the lines of Savage. I think it is very misleading to refer to this as "the complete class theorem" -- how on earth was I supposed to know that *this* was what was being referred to when "the complete class theorem" was mentioned, when it res

Underappreciated points about utility functions (of both sorts)

I think you've misunderstood a fair bit. I hope you don't mind if I address this slightly out of order.

Or if infinite utilities are not immediately a problem, then by a more complicated argument, involving constructing multiple St. Petersburg-type combinations and demonstrating that the axioms imply that there both should and should not be a preference between them.

This is exactly what Fishburn does, as I mentioned above. (Well, OK, I didn't attribute it to Fishburn, I kind of implicitly misattributed it to Savage, but it was actually Fishburn; I did

... (read more)Or if you have some formalism where preferences can be undefined (in a way that is distinct from indifference), by all means explain it... (but what happens when you program these preferences into an FAI and it encounters this situation? It has to pick. Does it pick arbitrarily? How is that distinct from indifference?)

A short answer to this (something longer later) is that an agent need not have preferences between things that it is impossible to encounter. The standard dissolution of the St. Petersberg paradox is that nobody can offer that gamble. Even th... (read more)

Underappreciated points about utility functions (of both sorts)

Is there a reason we can't just solve this by proposing arbitrarily large bounds on utility instead of infinite bounds? For instance, if we posit that utility is bounded by some arbitrarily high value X, then the wager can only payout values X for probabilities below 1/X.

I'm not sure what you're asking here. An individual decision-theoretic utility function can be bounded or it can be unbounded. Since decision-theoretic utility functions can be rescaled arbitrarily, naming a precise value for the bounds is meaningless; so like we could just assume the

... (read more)11ySay our utility function assigns an actual thing in the universe with value V1
and the utility function is bounded by value X. What I'm saying is that we can
make the problem go away by assuming bounded utility but without actually having
to define the ratio between V1 and X as a specific finite number (this would not
change upon scaling).
This means that, if your utility function is something like "number of happy
human beings", you don't have to worry about your utility function breaking if
the maximum number of happy human beings is larger than you expected since you
never have to define such an expectation. See my sub-sub-reply to Eigil
Rischel's sub-reply for elaboration.

Underappreciated points about utility functions (of both sorts)

Yes, thanks, I didn't bother including it in the body of the post but that's basically how it goes. Worth noting that this:

Both of these wagers have infinite expected utility, so we must be indifferent between them.

...is kind of shortcutting a bit (at least as Savage/Fishburn[0] does it; he proves indifference between things of infinite expected utility separately after proving that expected utility works when it's finite), but that is the essence of it, yes.

(As for the actual argument... eh, I don't have it in front of me and don't feel like rederivi

... (read more)Underappreciated points about utility functions (of both sorts)

By "a specific gamble" do you mean "a specific pair of gambles"? Remember, preferences are between two things! And you hardly need a utility function to express a preference between a single pair of gambles.

I don't understand how to make sense of what you're saying. Agent's preferences are the starting point -- preferences as in, given a choice between the two, which do you pick? It's not clear to me how you have a notion of preference that allows for this to be undefined (the agent can be *indifferent*, but that's distinct).

I mean, you could try to come

... (read more)31yThis is true, then it would only be between a specific subset of gambles.
I think you should be able to set things up so that you never encounter a pair
of gambles where this is undefined. I'll illustrate with an example. Suppose you
start with a prior over the integers, such that:
p(n) = (C/F(n)) where F(n) is a function that grows really fast and C is a
normalization constant. Then the set of gambles that we're considering would be
posteriors on the integers given that they obey certain properties. For
instance, we could ask the agent to choose between the posterior over integers
given that n is odd vs the posterior given that n is even.
I'm pretty sure that you can construct an agent that behaves as if it had an
unbounded utility function in this case. So long as the utility associated with
an integer n grows sufficiently slower than F(N), all expectations over
posteriors on the integers should be well defined.
If you were to build an FAI this way, it would never end up in a belief state
where the expected utility diverges between two outcomes. The expected utility
would be well defined over any posterior on it's prior, so it's choice given a
pair of gambles would also be well defined for any belief state it could find
itself in.

Underappreciated points about utility functions (of both sorts)

... (read more)If you're not making a prioritarian aggregate utility function by summing functions of individual utility functions, the mapping of a prioritarian function to a utility function doesn't always work. Prioritarian utility functions, for instance, can do things like rank-order everyone's utility functions and then sum each individual utility raised to the negative-power of the rank-order ... or something*. They allow interactions between individual utility functions in the aggregate function that are not facilitated by the direct summing permitted in utilita

Misconceptions about continuous takeoff

I don't really want to go trying to defend here a position I don't necessarily hold, but I do have to nitpick and point out that there's quite a bit of room inbetween exponential and hyperbolic.

Misconceptions about continuous takeoff

To be clear, intelligence explosion via recursive self-improvement has been distinguished from merely exponential growth at least as far back as Yudkowsky's "Three Major Singularity Schools". I couldn't remember the particular link when I wrote the comment above, but, well, now I remember it.

Anyway, I don't have a particular argument one way or the other; I'm just registering my surprise that you encountered people here arguing for merely *exponential* growth base on intelligence explosion arguments.

21yEmpirically, most systems with a feedback loop don't grow hyperbolically. I
would need strong theoretical reasons in order to understand why this particular
distinction is important.

Bayesian examination

Yeah, proper scoring rules (and in particular both the quadratic/Brier and the logarithmic examples) have been discussed here a bunch, I think that's worth acknowledging in the post...

21yUsually we discussed them in the past for binary choices. I don't remember any
previous discussion on using proper scoring rules for multiple choice tests.

11yI agree. Proper scoring rules were introduced to this community 14 years ago
[http://yudkowsky.net/rational/technical].

Misconceptions about continuous takeoff

It is sometimes argued that even if this advantage is modest, the growth curves will be exponential, and therefore a slight advantage right now will compound to become a large advantage over a long enough period of time. However, this argument by itself is not an argument against a continuous takeoff.

I'm not sure this is an accurate characterization of the point; my understanding is that the concern largely comes from the possibility that the growth will be *faster* than exponential, rather than merely exponential.

11ySure, if someone was arguing that, then they have a valid understanding of the
difference between continuous vs. discontinuous takeoff. I would just question
the assumption why we should expect growth to be faster than exponential for any
sustained period of time.

Goal-thinking vs desire-thinking

I mean, are you actually disagreeing with me here? I think you're just describing an intermediate position.

21yI don't know for sure whether we're really disagreeing. Perhaps that's a
question with no definite answer; the question's about where best to draw the
boundary of an only-vaguely-defined term. But it seems like you're saying
"goal-thinking must only be concerned with goals that don't involve people's
happiness" and I'm saying I think that's a mistake and that the fundamental
distinction is between doing something as part of a happiness-maximizing process
and recognizing the layer of indirection in that and aiming at goals we can see
other reasons for, which may or may not happen to involve our or someone else's
happiness.
Obviously you can choose to focus only on goals that don't involve happiness in
any way at all, and maybe doing so makes some of the issues clearer. But I don't
think "involving happiness" / "not involving happiness" is the most fundamental
criterion here; the distinction is actually, as your original terminology makes
clear, between different modes of thinking.

Goal-thinking vs desire-thinking

OK. I think I didn't think through my reply sufficiently. *Something* seemed off with what you were saying, but I failed to think through what and made a reply that didn't really make sense instead. But thinking things through a bit more now I think I can lay out my actual objection a bit more clearly.

I *definitely* think that if you're taking the point of view that suicide is preferable to suffering you're not applying what I'm calling goal-thinking. (Remember here that the description I laid out above is not intended as some sort of intensional definition

21yI see things slightly differently.
Happiness, suffering, etc., function as internal estimators of goal-met-ness.
Like a variable in a computer program that indicates how you're doing. Hence,
trying to optimize happiness directly runs the risk of finding ways to change
the value of the variable without the corresponding real-world things the
variable is trying to track. So far, so good.
But! That doesn't mean that happiness can't also be a thing we care about. If I
can arrange for someone's goals to be 50% met and for them to feel either as if
they're 40% met or as if they're 60% met, I probably choose the latter; people
like feeling as if their goals are met, and I insist that it's perfectly
reasonable for me to care about that as well as about their actual goals. For
that matter, if someone has goals I find terrible, I may actually prefer their
goals to go unmet but for them still to be happy.
I apply the same to myself -- within reason, I would prefer my happiness to
overestimate rather than underestimate how well my goals are being met -- but
obviously treating happiness as a goal is more dangerous there because the risk
of getting seriously decoupled from my goals is greater. (I think.)
I don't think it's necessary to see nonexistence as neutral in order to prefer
(in some cases, perhaps only very extreme ones) nonexistence to
existence-with-great-suffering. Suffering is unpleasant. People hate it and
strive to avoid it. Yes, the underlying reason for that is because this helps
them achieve other goals, but I am not obliged to care only about the underlying
reason. (Just as I'm not obliged to regard sex as existing only for the sake of
procreation.)

Goal-thinking vs desire-thinking

This is perhaps an intermediate example, but I do think that once you're talking about internal experiences to be avoided, it's definitely not all the way at the goal-thinking end.

41yI'm not convinced. To me, at least, my goals that are about me don't feel
particularly different in kind from my goals that are about other people, nor do
my goals that are about experiences feel particularly different from my goals
that are about things other than experiences.
(It's certainly possible to draw your dividing line between, say, "what you want
for yourself" and "what other things you want", but I think that's an entirely
different line from the one drawn in the OP.)

Goal-thinking vs desire-thinking

Hm, I suppose that's true. But I think the overall point still stands? It's illustrating a type of thinking that doesn't make sense to one thinking in terms of concrete, unmodifiable goals in the external world.

21yIs that really true? If you can have "have other people not suffer horribly" as
a goal, you can have "not suffer horribly yourself" as a goal too. And if, on
balance, your life seems likely to involve a lot of horrible suffering, then
suicide might absolutely make sense even though it would reduce your ability to
achieve your other goals.

Coherent decisions imply consistent utilities

So this post is basically just collecting together a bunch of things you previously wrote in the Sequences, but I guess it's useful to have them collected together.

I must, however, take objection to one part. The proper non-circular foundation you want for probability and utility is not the complete class theorem, but rather Savage's theorem, which I previously wrote about on this website. It's not short, but I don't think it's too inaccessible.

Note, in particular, that Savage's theorem does not start with any assumption baked in that **R** is the correct sy

Noticing Frame Differences

Thirding what the others said, but I wanted to also add that rather than actual game theory, what you may be looking here may instead be the anthropological notion of limited good?

The Forces of Blandness and the Disagreeable Majority

Sorry, but: The thing at the top says this was crossposted from Otium, but I see no such post there. Was this meant to go up there as well? Because it seems to be missing.

Looks like Sarah took that post down from Otium. Will move it back into drafts until I hear back from her about what she wants to do with it.

(moved it back, Sarah says it's fine)

An Extensive Categorisation of Infinite Paradoxes

OK, time to actually now get into what's wrong with the ones I skipped initially. Already wrote the intro above so not repeating that. Time to just go.

**Infinitarian paralysis**: So, philosophical problems to start: As an actual decision theory problem this is all moot since you can't actually have an infinite number of people. I.e. it's not clear why this is a problem at all. Secondly, naive assumption of utilitarian aggregation as mentioned above, etc, not going over this again. Enough of this, let's move on.

So what are the mathematical problems here? W

... (read more)An Extensive Categorisation of Infinite Paradoxes

OK, time for the second half, where I get to the errors in the ones I initially skipped. And yes, I'm going to assert some philosophical positions which (for whatever reason) aren't well-accepted on this site, but there's still plenty of mathematical errors to go around even once you ignore any philosphical problems. And yeah, I'm still going to point out missing formalism, but I will try to focus on the more substantive errors, of which there are plenty.

So, let's get those philosophical problems out of the way first, and quickly review utility functions

... (read more)32yOK, time to actually now get into what's wrong with the ones I skipped
initially. Already wrote the intro above so not repeating that. Time to just go.
Infinitarian paralysis: So, philosophical problems to start: As an actual
decision theory problem this is all moot since you can't actually have an
infinite number of people. I.e. it's not clear why this is a problem at all.
Secondly, naive assumption of utilitarian aggregation as mentioned above, etc,
not going over this again. Enough of this, let's move on.
So what are the mathematical problems here? Well, you haven't said a lot here,
but here's what it's look like to me. I think you've written one thing here that
is essentially correct, which is that, if you did have some system of surreal
valued-utilities, it would indeed likely make the distinction you want.
But, once again, that's a big "if", and not just for philosophical reasons but
for the mathematical reasons I've already brought up so many times right now --
you can't do infinite sums in the surreals like you want, for reasons I've
already covered
[https://www.lesswrong.com/posts/XW6Qi2LitMDb2MF8c/a-rationality-condition-for-cdt-is-that-it-equal-edt-part-1#ceToiLikEpRmFrAHc]
. So there's a reason I included the word "likely" above, because if you did
find an appropriate way of doing such a sum, I can't even necessarily guarantee
that it would behave like you want (yes, finite sums should, but infinite sums
require definition, and who knows if they'll actually be compatible with finite
sums like they should be?).
But the really jarring thing here, the thing that really exposes a serious error
in your thought (well, OK, that does so to a greater extent), is not in your
proposed solution -- it's in what you contrast it with. Cardinal
valued-utilities? Nothing about that makes sense! That's not a remotely
well-defined alternative you can contrast with! And the thing that bugs me about
this error is that it's just so unforced -- I mean, man, you could hav

An Extensive Categorisation of Infinite Paradoxes

My primary response to this comment will take the form of a post, but I should add that I wrote: "I will provide informal hints on how surreal numbers could help us solve some of these paradoxes, although the focus on this post is primarily categorisation, so please don't mistake these for formal proofs".

You're right; I did miss that, thanks. It was perhaps unfair of me then to pick on such gaps in formalism. Unfortunately, this is only enough to rescue a small portion in the post. Ignoring the ones I skipped -- maybe it would be worth my time to get

... (read more)An Extensive Categorisation of Infinite Paradoxes

Almost nothing in this post is correct. This post displays not just a misuse of and failure to understand surreal numbers, but a failure to understand cardinals, ordinals, free groups, lots of other things, and just how to think about such matters generally, much as in our last exchange. The fact that (as I write this) this is sitting at +32 is an *embarrassment* to this website. You really, really, need to go back and relearn all of this from scratch, because going by this post you don't have the slightest clue what you're talking about. I would encoura

92yOK, time for the second half, where I get to the errors in the ones I initially
skipped. And yes, I'm going to assert some philosophical positions which (for
whatever reason) aren't well-accepted on this site, but there's still plenty of
mathematical errors to go around even once you ignore any philosphical problems.
And yeah, I'm still going to point out missing formalism, but I will try to
focus on the more substantive errors, of which there are plenty.
So, let's get those philosophical problems out of the way first, and quickly
review utility functions and utilitarianism, because this applies to a bunch of
what you discuss here. Like, this whole post takes a very naive view of the idea
of "utility", and this needs some breaking down. Apologies if you already know
all of what I'm about to say, but I think given the context it bears repeating.
So: There are two different things meant by "utility function". The first is
decision-theoretic; an agent's utility function is a function whose expected
value it attempts to maximize. The second is the one used by utilitarianism,
which involves (at present, poorly-defined) "E-utility" functions, which are not
utility functions in the decision-theoretic sense, that are then somehow
aggregated (maybe by addition? who knows?) into a decision-theoretic utility
function. Yes, this terminology is terribly confusing. But these are two
separate things and need to be kept separate.
Basically, any agent that satisfies appropriate rationality conditions has a
utility function in the decision-theoretic sense (obviously such idealized
agents don't actually exist, but it's still a useful abstraction). So you could
say, roughly speaking, any rational consequentialist has a decision-theoretic
utility function. Whereas E-utility is specifically a utilitarian notion, rather
than a general consequentalist or purely descriptive notion like
decision-theoretic utility (it's also not at all clear how to define it).
Anyway, if you want surreal

92yMy primary response to this comment will take the form of a post, but I should
add that I wrote: "I will provide informal hints on how surreal numbers could
help us solve some of these paradoxes, although the focus on this post is
primarily categorisation, so please don't mistake these for formal proofs".
Your comment seems to completely ignore this stipulation. Take for example this:
Yes, there's a lot of philosophical groundwork that would need to be done to
justify the surreal approach. That's why I said that it was only an informal
hint.
Yes, I actually did look up that there was a way of defining 2^s where s is a
surreal number.
I wrote a summary of a paper
[https://www.lesswrong.com/posts/MtYDr7zyicq4J7rWe/summary-surreal-decisions] by
Chen and Rubio that provides the start of a surreal decision theory. This isn't
a complete probability theory as it only supports finite additivity instead of
countable additivity, but it suggests that this approach might be viable.
I could keep going, but I think I've made my point that you're evaluating these
informal comments as though I'd claimed they were a formal proof. This post was
already long enough and took enough time to write as is.
I will admit that I could have been clearer that many of these remarks were
speculative, in the sense of being arguments that I believed were worth working
towards formalising, even if all of the mathematical machinery doesn't
necessarily exist at this time. My point is that justifying the use of surreals
numbers doesn't necessarily involve solving every paradox; it should also be
persuasive to solve a good number of them and then to demonstrate that there is
good reason to believe that we may be able to solve the rest in the future. In
this sense, informal arguments aren't valueless.

22yThis is quite a long post, so it may take some time to write a proper reply, but
I'll get back to you when I can. The focus of this post was on gathering
together all the infinite paradoxes that I could manage. I also added some
informal thoughts on how surreal numbers could help us conceptualise the
solution to these problems, although this wasn't the main focus (it was just
convenient to put them in the same space).
Unfortunately, I haven't continued the sequence since I've been caught up with
other things (travel, AI, applying for jobs), but hopefully I'll write up some
new posts soon. I've actually become much less optimistic about surreal numbers
for philosophical reasons which I'll write up soon. So my intent is for my next
post to examine the definition of infinity and why this makes me less optimistic
about this approach. After that, I want to write up a bit more formally how the
surreal approach would work, because even though I'm less optimistic about this
approach, perhaps someone else will disagree with my pessimism. Further, I think
it's useful to understand how the surreal approach would try to resolve these
problems, even if only to provide a solid target for criticism.

Yeah. You

canuse language that is unambiguously not attack language, it just takes more effort to avoid common words. In this respect it's not much unlike how discussing lots of other things seriously requires avoiding common but confused words!