I might become a little bit more unpopular because of this, but I really want to say I find this to be rather abstract, and in fact something that I can't really apply to real life. That's the main problem with your ideas; you changed the universe and the laws in which things operate, and you removed the element of time. That's so fundamentally different from where I am that I would agree with everything you say, but coming back to my own mortal self the only thing I can think is "Nope".
But I'm not going to be be a complete and utter fucktard because you're really putting some effort into your posts and they're more interesting than Gleb's markov-chain-esque links, so I'll be a little more constructive.
What does this actually mean for the real world?
I'm quite confused by your fixation on "complete rational agent". The highest "value" is abstract. Let's give grades to decisions.
BAD DECISION (1) ----------------------------------- (100) GOOD DECISION
We can say that 100 is the "complete rational agent". But that doesn't mean the agent at 95 didn't make a spectacular decision. How much of a difference is between 100 to 95? I can't tell because we're at a high abstraction level.
That's where you go lower level and put on your giant glasses and see 1s and 0s for a good fifteen minutes. OK that was an exaggeration but still, we must expand on what makes it a good decision, and check out the building blocks which made the decision good.
There's also some sort of paradox here that I'm probably missing a crucial part of it, but if "perfect theoretical rationality" cannot be achieved, does that mean that the closest value to it is the new perfect theoretical rationality? But it can't be achieved; so it must be a smaller value, which also cannot be achieved, and so on. Then again, after a few of those we clearly are somewhat distant from "perfect theoretical rationality."
Is missing out on utility bad?
Wouldn't the rational agent postpone saying his desired utility rather than hope for a good enough number? If it's finite as you say, it can be measured, and we can say (suffering + 1). If it's infinite, well.. we've been there before, and you've stated it again in your example.
But this is an unwinnable scenario, so a perfectly rational agent will just pick a number arbitrarily? Sure you don't get the most utility, but why does this matter?
Again, infinity, different laws of universe, sudden teleportation to an instance of failure.. Neal Stephenson would love this stuff.
What other consequences are there?
#f: maximum recursion depth reached
"But if "perfect theoretical rationality" cannot be achieved, does that mean that the closest value to it is the new perfect theoretical rationality?"
Good question. No matter what number you pick someone else could have done a million times better, or a billion times better or a million, billion times better, so you are infinitely far from being perfectly rational.
"Wouldn't the rational agent postpone saying his desired utility rather than hope for a good enough number?"
The idea is that you don't know how much suffering there is in the universe and so no matter how large a number you picked, there could be more in which case you've lost.
I think your hypothetical universe of infinite suffering is unsuited to the usual sort of utility calculations. For sure the following small variant on it is:
We have infinitely many people -- call them -1, -2, -3, ... -- living more or less equivalent lives of misery. We also have infinitely many people -- call them +1, +2, +3, ... -- living good lives. (The addition of infinitely many good lives is what distinguishes my hypothetical universe from yours, though you didn't actually say there aren't infinitely many good lives in yours to go with the infinitely many bad ones.)
Now Omega comes along and offers to take a million people and make their lives good. So now we have, let's say, -1000001, -1000002, -1000003, ... on the bad side and -1000000, ..., -1 together with +1, +2, +3, ... on the good side. Big improvement, right? Not necessarily, because here's something else Omega could have done instead: just change the names by subtracting 1000000 from the numbers of all the minuses and of all the plusses from +1000001 onwards, and negating the numbers of +1 ... +1000000. This yields exactly the same assignment of numbers to positive/negative, but no one actually got their life improved.
Situations with infinite utilities in them are really hard to reason about. It's possible that there is no way to think about them that respects all our moral intuitions. (Or indeed our intuitions about preferences and rationality. These don't need to be specifically moral questions.)
When the infinite utilities are on only one side -- in your case we have infinite misery but not infinite happiness -- you don't get the exact same problems; I don't see any way to produce results quite as crazy as the one I described -- but my feeling is that there too we have no right to expect that our intuitions are all mutually consistent.
with surreal math ω+10000000 > ω. I am doubtful on whther the two possible options would actually end in the same situation. And in a way I don't care about how people are label, I care abuot peoples lifes. It could just be that the relationship between labels for peoples and people amounts diverge. For example in the infinite realm cardinality and ordinality diverge while in the finite realm they coincicde. It could be that your proof shows that there is no ordinal improvement but what if the thing to be cared about behaves like a cardinality?
And we can as well scale down the numbers. Given that there are infinite bad lives and infinite good lives taking one person from bad to good seems like it ought to improve mattes but technically it seems it results in the same state. I do wonder in that if I have the reals colored white for x>0 and black for 0<=x 0 chould be colored black. Now if I change the color of 0 to white I should be getting the result that now there is more white than previously and less black (even if relatively only infinidesimally so). Now you if you technically take the measure of the point 0 and the white portion of the line one might end up saying things like "measure of a single point is 0". What I would say is that often measures are defined to be reals but here the measure would need to be surreal for it to accomodate questions about infinities. For example in normal form we could write a surreal with "a + b ω^1 + c ω^2 ... " with a,b,c being real factors. Then it woud dmake sense in that a finite countable collection would have b=0 but it doesn't mean the whole sum is 0. While we can't use any small positve real for b for dots I would still say that a point is "pointlike" and not nothing at all. A finite amount of points is not going to be more than no matter how short a line segment. But it doesn't mean they don't count at all. It just means that points "worth" is infinidesimal when compared to lines.
with surreal math
Yup. But I don't see a good reason to apply it that way here, nor a good principled way of doing so that gives the results you want. I mean, how are you going to calculate the net utility in the "after" situation to arrive at ω+10000000 rather than ω?
no ordinal improvement [...] like a cardinality
It looks to me like it's the other way around. Surreal integer arithmetic is much more like ordinal arithmetic than like cardinal arithmetic. That's one reason why I'm skeptical about the prospects for applying it here: it seems like it might require imposing something like an ordering on the people whose utilities we're aggregating.
the measure would need to be surreal
Despite my comments above, I do think it's worth giving more consideration to using a richer number system for utilities.
[EDITED to fix an inconsequential and almost invisible typo.]
It does occur to me that while giving the people an order migth be suspicous utilities are a shorthand of preferences which are defined to be orders of preferring a over b. Therefore there is anyways going to be a conversion to ordinals so surreals should remain relevant.
I don't think I'm convinced. Firstly, because in these cases where we're looking at aggregating the interests of large collections of people it's the people, not the possibilities, that seem like they need to be treated as ordered. Secondly, because having an ordering on preferences isn't at all the same thing as wanting to use anything like ordinals for them. (E.g., cardinals are ordered too -- well-ordered, even -- at least if we assume the axiom of choice. The real numbers are ordered in the obvious way, but that's not a well-ordering. Etc.)
I concede that what I'm saying is very hand-wavy. Maybe there really is a good way to make this sort of thing work well using surreal numbers as utilities. And (perhaps like you) I've thought for a long time that using something like the surreals for utilities might turn out to have advantages. I just don't currently see an actual way to do it in this case.
Is there a practical difference between infinity and "large unknown unbounded number"? It still falls into the issue of "is the number large enough", and if it's not, you missed suffering you could've alleviated. So from an in-universe[1] viewpoint there is no reason not to state googol^googol, alphabetagammadeltaomega-quadtribillion[2], or the total length of of a all possible 4096-bit public keys, or some other horror that might make make reality crash.
[1] I'm assuming we're still in the "timeless celestial beings" universe. [2] I'm making stuff up.
If you create an actual infinity then things get weird. Many intuitive rules don't hold. So I don't want an actual infinity.
But a large, unknown number could easily be some sort of infinity.
Let's look at it another way. Say I choose some unknown number as you described. Any reason I couldn't be enlightened by "well, if you had chosen number+1, you could have saved the universe"?
I definitely am lacking in my mathematical knowledge so if there's a way to deal with un-measured numbers I'd appreciate if someone could enlighten me.
"But a large, unknown number could easily be some sort of infinity." - it could if I hadn't specified that we are assuming it is finite.
Then the best decision is to make some calculations, say, how much suffering per 1m/km2 on average, multiply that by how much of the universe you can observe, then add an incredibly large amount of 9s to it's right side. Use all the excess utility to expand your space travel and observation and save the other planets from suffering.
In order to guarantee being able to deliver whatever utility change the player demands in the way you describe, Omega needs there to be an infinite amount of suffering to relieve.
[EDITED to add:] If whoever downvoted this would like to explain my error, I'd be interested. It looks OK to me. Or was it just Eugine/Azathoth/Ra/Lion punishing me for not being right-wing enough again?
I made no claim that those are the only two possibilities. But, for what it's worth, here are the options I actually see. First, "legitimate" ones where someone read what I wrote, thought it was bad, and voted it down on that ground:
Then there are the options where the downvote was not on the basis of (actual or perceived) problems with the comment itself:
So. It looks to me like there are lots of low-probability explanations, plus "someone thinks I made a dumb mistake", plus "Eugine/Azathoth/Ra wanted to downvote something I wrote", both of which are things that have happened fairly often and good candidates for what's happened here. And if someone thinks I made a dumb mistake, it seems like explaining what would be a good idea (whether the mistake is mine or theirs). Hence my comment.
(This feels like about two orders of magnitude more analysis than this trivial situation warrants, but no matter.)
I made no claim that those are the only two possibilities.
On reflection, I see that you're right; I inferred too much from your comment. What you said was that you'd be interested in an explanation of your error, if and only if you committed one; followed by asking the separate, largely independent question of whether Eugine/Azathoth/Ra/Lion was punishing you for not being right-wing enough again. I erroneously read your comment as saying that you'd be interested in (1) an explanation of your error or (2) the absence of such an explanation, which would prove the Eugine hypothesis by elimination. Sorry for jumping the gun and forcing you into a bunch of unnecessary analysis.
No problem.
Indeed I was not claiming that the absence of an explanation would prove it was Eugine. It might simply mean that whoever downvoted me didn't read what I wrote, or that for whatever reason they didn't think it would be worth their while to explain. Or the actual reason for the downvote could be one of those low-probability ones.
One correction, though: I would be interested in an explanation of my error if and only if whoever downvoted me thinks I committed one. Even if in fact I didn't, it would be good to know if I failed to communicate clearly, and good for them to discover their error.
And now I shall drop the subject. (Unless someone does in fact indicate that they downvoted me for making a mistake and some sort of correction or clarification seems useful.)
Ah, I hadn't taken in that the person complaining rudely that I hadn't considered all the possibilities for why I got downvoted might be the person who downvoted me. In retrospect, I should have.
Anyway (and with some trepidation since I don't much relish getting into an argument with someone who may possibly just take satisfaction in causing petty harm): no, it doesn't look to me as if casebash's arguments are much like 2+2=5, nor do I think my comments are as obvious as pointing out that actually it's 4. The sort of expected-utility-maximizing that's generally taken around these parts to be the heart of rationality really does have difficulties in the presence of infinities, and that does seem like it's potentially a problem, and whether or not casebash's specific objections are right they are certainly pointing in the direction of something that could use more thought.
I do not think I have ever encountered any case in which deliberately making a problem worse to draw attention to it has actually been beneficial overall. (There are some kinda-analogous things in realms other than human affairs, such as vaccination, or deliberately starting small forest fires to prevent bigger ones, but the analogy isn't very close.)
If indeed LW has become irredeemably shit, then amplifying the problem won't fix it (see: definition of "irredeemably") so you might as well just fuck off and do something less pointless with your time. If it's become redeemably shit, adding more shit seems unlikely to be the best way of redeeming it so again I warmly encourage you to do something less useless instead. But these things seem so obvious -- dare I say it, so much like pointing out that 2+2=4? -- that I wonder whether, deep down, under the trollish exterior, there lurks a hankering for something better. Come to the Light Side! We have cookies.
I'll let the rest of your comment stand, but:
no, it doesn't look to me as if casebash's arguments are much like 2+2=5
2 + 2 = 2 + 2 + 0. A number subtracted from itself equals 0, and infinity is a number, so 2 + 2 = 2 + 2 + ∞ - ∞. Infinity plus one is still infinity, so 2 + 2 = 2 + 2 + (∞ + 1) - ∞ = 2 + 2 + (1 + ∞) - ∞ = (2 + 2 + 1) + (∞ - ∞) = 2 + 2 + 1 + 0 = 5. So nyah.
(Sorry, casebash, but it was too good a setup to ignore.)
If perfect rationality doesn't exist in general, then if you want to assume it, you have to prove that a relevant class of problems has a perfectly rational agent
That's not what your other post argues. It argues that perfect ratioanlity doesn't exist in a specific world which is fundamentally different than our own. You haven't shown that there are prolbems in our world for which perfect rationality doesn't exist.
If you want to make a deductive argument based on a bunch of abstract axioms you actually have to care about whether the axioms apply.
+1. Thanks for this feedback, I've edited as follows:
"If perfect rationality doesn't exist in at least some circumstances(edited), then if you want to assume it, you have to prove that a relevant class of problems has a perfectly rational agent."
The point I was making is that you can prove for a large class of real world problems that such a rational agent exists, so most things remain the same, except for the fact that you have to add an additional line in your proofs quoting a general result that proves a perfectly rational agent exists in the particular type of situation.
I've reconsidered, and this is addressed to the author: It's actually difficult to get things quite this degree of perfectly wrong in order to arrive at these conclusions; it's the mathematical equivalent of proving that 1+1=3, and, like said proofs, does a better job confusing people than providing any useful insights. Which is indicative either that you're trolling, or communication issues.
Which is certainly not to rule out communication issues, of which I had considerable issue when I first joined, and which can look similar to such trolling. If this is a communication issue, however, I must advise you to be a little more cautious of doubling down on the wrong conclusions (another issue I had considerable issue with, getting involved in arguments arising entirely from miscommunication) - which this post has a bit of a smell of, in insisting on definitions that have already been challenged.
If the whole of your argument is that a finite number is always smaller than infinity - well, yes. Unbounded utility functions wreck all kinds of havoc in utilitarianism, for exactly this reason, and are generally rejected. Insisting that utility can be unlimited is making an assertion of an unbounded utility - if one of your assumptions going into utilitarianism is that utility functions are unbounded, you're making a foray into an area of utility theory that is acknowledged to be full of weird outcomes. Of a similar nature are your infinitesimal arguments, which avoid the immediate problem of unbounded utility functions, but which you supplement with zero opportunity cost (which results in an infinitesimal number divided by zero when calculating opportunity cost, and further weirdness). Furthermore, not all rationalists are even utilitarians (I'm not, and I consider the concept of "utility" to be so poorly defined as to be worthless).
But regardless of all of that, I'll revert to the more substantive criticism of your criticisms of rationality: Deliberately creating questions without an answer, and using them to criticize an answer-generating system for its inability to arrive at the correct answer, is just sass without substance.
Actually, unbounded utility doesn't necessarily provide too much in the way of problems, unless you start averaging. It's more infinite utility that you have to worry about.
"Unbounded utility functions wreck all kinds of havoc in utilitarianism, for exactly this reason, and are generally rejected" - not because they are invalid, but because they complicate things. That isn't a particularly good reason.
"zero opportunity cost" - Again, that's a problem with your tool, not the situation. If I offer you 10 utility for 0 opportunity cost as a once off offer, you take it. You don't need to divide by 0.
"Deliberately creating questions without an answer, and using them to criticize an answer-generating system for its inability to arrive at the correct answer, is just sass without substance." - that has been more than adequately addressed several times on the comment threads.
Again, that's a problem with your tool, not the situation. If I offer you 10 utility for 0 opportunity cost as a once off offer, you take it. You don't need to divide by 0.
I'm glad you agree you should wait forever in the utilon-trading game, which after all has 0 opportunity cost in exchange for a free utilon.
that has been more than adequately addressed several times on the comment threads.
Then address it. Copy and paste, if necessary.
That's because it is a once off offer. You can't get stuck in a loop by accepting it.
"The argument that you can't choose infinity, so you can't win anyway, is just a distraction. Suppose perfect rationality didn't exist for a particular scenario, what would this imply about this scenario? The answer is that it would imply that there was no way of conclusively winning, because, if there was, then an agent following this strategy would be perfectly rational for this scenario. Yet, somehow people are trying to twist it around the other way and conclude that it disproves my argument. You can't disprove an argument by proving what it predicts"
unbounded utility doesn't necessarily provide too much in the way of problems, unless you start averaging.
??? The whole point of utility is to average it. That's what motivations the decision-theoretic definition of utility.
More of this. I don't find you worth addressing, but as for your audience:
These arguments are -just- ignorant enough to sound plausible while getting subtle, but critical, details wrong. Namely, the use of infinities (period, at all) as if they were real numbers is invalid and produces "undefined" in all the mathematics in which they're (invalidly) inserted, and the concept of "utility" utilized here is leaning heavily on the fact that "utility" isn't actually defined. Additionally, the author imports ideas of rational behavior from this universe and tries to apply them to poorly-self-defined universes where the preconditions that made that behavior rational in this universe don't apply, and believes the failure of rationality to be context-independent is a failure of rationality more broadly.
Summed up, his argument comes down to this: Rationality can't come up with an answer to the question "What's the largest integer", therefore rationality is impossible. All of his problems are some variant on this, as the author works to make sure there -isn't- a correct answer, and if an answer is found anyways, will accuse you of fighting the hypothesis. Which is entirely true, as his hypothesis is that rationality cannot answer every question, and treats this as a failure of rationality, rather than the more fundamental issue of designing questions to have no answer.
For example, if there are a finite number of options, each with a measurable, finite utility, we can assume that a perfectly rational agent exists.
How do you know that to be true? If I remember right Eliezer consider that statement to be false.
It seems to me like you did a lot of work to try to think about a world with special rules and you have a lot of unquestioned axioms.
Why does he consider it to be false?
My understanding was that he argued for a different definition of rationality, not that it didn't exist.
Basically TDT is about the fact that we don't evaluate individual decisions for rationality but agents strategies. For every strategy that an agent has, other agents can punish the agent for having that strategy if they specifically choose to do so.
If there's for example a God/Matrix overlord who rewards people for believing in him even if the person has no rational reason to believe in him that would invalidate a lot of decision theories.
Basically decisions are entangled with the agent that makes them and not independent as you assume in your model. That entangelement forbids perfect decision making strategies that have the property independent of the world and whether other agents decide to specifically discriminate against certain strategies.
Do you have a quote where Elizier says rationality doesn't exist? I don't believe that he'd say that. I think he'd argue that it is okay for rationality to fail when rationality is specifically being punished.
Regardless, I disagree with TDT, but that's a future article.
Do you have a quote where Eliezier says rationality doesn't exist?
No, but I don't think that he shares your idea of perfect theoretical rationality anyway. From that point there no need to say that it doesn't exist.
I think that it's Eliezer position that it's good enough that TDT provides the right answer when the other agent doesn't specifically choose to punish TDT.
But I don't want to dig up a specific quote at the moment.
Regardless, I disagree with TDT, but that's a future article.
You are still putting assumption about nonentanglement into your models. Questions about the actions of nonentangled actors are like asking how many angels can stand of the pin of a nail.
They are far removed from the kind of rationality of the sequences. You reason far away from reality. That's a typical problem with thinking too much in terms of hypotheticals instead of basing your reasoning on real world references.
In order to prove that it is a problem, you have to find a real world situation that I've called incorrectly due to my decision to start reasoning from a theoretical model.
In general that's hard without knowing many real world situations that you have judged.
But there's a recent example. The fact that different people won't come up with different definitions of knowledge if you ask them would be one recent example of concentrating too much on a definition and too little on actually engaging with what different people think.
I was expecting on the assumtinos of the question braking down on increasingly big numbers.
Let me give a try on giving a more real life problem that woudl have charasterisitcs of being a case where absense of a perfectly rational choice comes into play.
You have stumbled upon a perfect counterfeit machine. How much money should you print with it? While for small amounts the obvious way to do better is to just print more once you start stating amounts larger than GDPs of whole countries things start to break down. An addiotional coin doesn't represent a new fresh same portion buying power than the last one. it is also similar in that the more money you print the larger share of the buying power of the money you own but you can only approach but never reach it being 100%. For some arbitrary big number the average worth of a person lifesaving totals 0.0005$ cents when it could have been only 0.000000000005$ if you had named a greater number.
And there is also a good analog about the "ongoer" solution being pretty bad. If you just keep printing money you are never going to offer that money in the local shop because you are busy printing money. Even if you like generate enough money to just buy every real estate in the planet you need to leave the room before you suffocate into your money. But it does mean you don't have to think about how much you should work for salary because you don't need to ever work. But if you had a friend that tried to go to the local shop by candy since you suddenly can afford it would be foolish to forbid him doing so because he needs to keep on printing money and not settle for a finite or smaller amount of money.
If you make an assumtino that cows are spherically symmetric and derive a solution based on that it doesn't really matter if you missed some more optimal solution that strongly depended them on being exactly spherically symmetric. When you keep on making these kind of simplistic assumtions it comes more and more closer to asking "what I should do if I wasn't me?". If you know things like "Gold holders should sell their gold" at some point it is more interesting to answer "Am I a gold owner?" rather than hone gold selling strategies.
Caveats: Dependency (Assumes truth of the arguments against perfect theoretical rationality made in the previous post), Controversial Definition (perfect rationality as utility maximisation, see previous thread)
This article is a follow up to: The Number Choosing Game: Against the existence of perfect theoretical rationality. It discusses the consequences of The Number Choosing Game, which is roughly, that you name the decimal representation of any number and you gain that much utility. It takes place in a theoretical world where there are no real world limitations in how large a number you can name or any costs. We can also assume that this game takes place outside of regular time, so there is no opportunity cost. Needless to say, this was all rather controversial.
Update: Originally I was trying to separate the consequences from the arguments, but it seems that this blog post slipped away from it.
What does this actually mean for the real world?
This was one of the most asked questions in the previous thread. I will answer this, but first I want to explain why I was reluctant to answer. I agree that it is often good to tell people what the real world consequences are as this isn't always obvious. Someone may miss out on realising how important an idea is if this isn't explained to them. However, I wanted to fight against the idea that people should always be spoonfed the consequences of every argument. A rational agent should have some capacity to think for themselves - maybe I tell you that the consequences are X, but they are actually Y. I also see a great deal of value from discussing the truth of ideas separate from the practical consequences. Ideally, everyone would be blind to the practical consequences when they were first discussing the truth of an idea as it would lead to a reduction in motivated reasoning.
The consequences of this idea are in one sense quite modest. If perfect rationality doesn't exist in at least some circumstances(edited), then if you want to assume it, you have to prove that a relevant class of problems has a perfectly rational agent. For example, if there are a finite number of options, each with a measurable, finite utility, we can assume that a perfectly rational agent exists. I'm sure we can prove that such agents exist for a variety of situations involving infinite options as well. However, there will also be some weird theoretical situations where it doesn't apply. This may seem irrelevant to some people, but if you are trying to understand strange theoretical situations, knowing that perfect rationality doesn't exist for some of these situations will allow you to provide an answer when someone hands you something unusual and says, "Solve this!". Now, I know my definition for rationality is controversial, but even if you don't accept it, it is still important to realise that the question, "What would a utility maximiser do?" doesn't always have an answer, as sometimes there is no maximum utility. Assuming perfectly rational agents as defined by utility maximisation is incredibly common in game theory and economics. This is helpful for a lot of situations, but after you've used this for a few years in situations where it works you tend to assume it will work everywhere.
Is missing out on utility bad?
One of the commentators on the original thread can be paraphrased as arguing, "Well, perhaps the agent only wanted a million utility". This misunderstands that the nature of utility. Utility is a measure of things that you want, so it is something you want more of by definition. It may be that there's nothing else you want, so you can't actually receive any more utility, but you always want more utility(edited).
Here's one way around this problem. The original problem assumed that you were trying to optimise for your own utility, but lets pretend now that you are an altruistic agent and that when you name the number, that much utility is created in the world by alleviating some suffering. We can assume in infinite universe so there's infinite suffering to alleviate, but that isn't strictly necessary, as no matter how large a number you name, it is possible that it might turn out that there is more suffering than that in the universe (while still being finite). So let's suppose you name a million, million, million, million, million, million (by its decimal representation of course). The gamemaker then takes you to a planet where the inhabitants are suffering the most brutal torture imaginable by a dictator that is using their planet's advanced neuroscience knowledge to maximise the suffering. The gamemaker tells you that if you had added an extra million on the end, then these people would have had their suffering alleviated. If rationality is winning, does a planet full of tortured people count as winning? Sure, the rules of the game prevent you from completely winning, but nothing in the game stopped you from saving those people. The agent that also saved those people is a more effective agent and hence a more rational agent than you are. Further if you accept that there is no difference between acts and omissions, then there is no moral difference between torturing those people yourself and failing to say the higher number (Actually, I don't really believe this last point. I think this is more a flaw with arguing acts and omissions are the same in the specific case of an unbounded set of options. I wonder if anyone has ever made this argument before, I wouldn't be surprised if it this wasn't the case and if there was a philosophy paper in this).
But this is an unwinnable scenario, so a perfectly rational agent will just pick a number arbitrarily? Sure you don't get the most utility, but why does this matter?
If we say that the only requirement here for an agent to deserve the title of "perfectly rational" is to pick an arbitrarily stopping point, then there's no reason why we can't declare the agent that arbitrarily stops at 999 as "perfectly rational". If I gave an agent the option of picking a utility of 999 or a utility of one million, the agent who picked a utility of 999 would be quite irrational. But suddenly, when given even more options, the agent who only gets 999 utility counts as rational. It actually goes further than this, there's no objective reason that the agent can't just stop at 1. The alternative is that we declare any agent who picks at least a "reasonably large" number to be rational. The problem is that there is no objective definition of "reasonably large". This would create a situation where our definition of "perfectly rational" would be subjective, which is precisely what the idea of perfectly rational was created to avoid. It gets worse than this still. Let's pretend that before the agent plays the game they lose a million utility (and that the first million utility they get from the game goes towards reversing these effects, time travel is possible in this universe). We then get our "perfectly rational" agent a million (minus one) utility in the red, ie. suffering a horrible fate, which they could have easily chosen to avoid. Is it really inconceivable that the agent who gets positive one million utility instead of negative one million could be more rational?
What if this were a real life situation? Would you really go, "meh" and accept the torture because you think a rational agent can pick an arbitrary number and still be perfectly rational?(edit)
The argument that you can't choose infinity, so you can't win anyway, is just a distraction. Suppose perfect rationality didn't exist for a particular scenario, what would this imply about this scenario? The answer is that it would imply that there was no way of conclusively winning, because, if there was, then an agent following this strategy would be perfectly rational for this scenario. Yet, somehow people are trying to twist it around the other way and conclude that it disproves my argument. You can't disprove an argument by proving what it predicts(edit).
What other consequences are there?
The fact that there is no perfectly rational agent for these situations means that any agent will seem to act rather strangely. Let's suppose that a particular agent who plays this game will always stop at a certain number X, say a Googleplex. If we try to sell them this item for more than that, they would refuse as they wouldn't make money on it, despite the fact that they could make money on it if they chose a higher number.
Where this gets interesting is that the agent might have special code to buy the right to play the game for any price P, and then choose the number X+P. It seems that sometimes it is rational to have path-dependent decisions despite the fact that the amount paid doesn't affect the utility gained from choosing a particular number.
Further, with this code, you could buy the right to play the game back off the agent (before it picks the number) for X+P+1. You could then sell it back to the agent for X+P+one billion and repeatedly buy and sell the right to play the game back to the agent. (If the agent knows that you are going to offer to buy the game off it, then it could just simulate the sale by increasing the number it asks for, but it has no reason not to simulate the sale and also accept a higher second offer)
Further, if the agent was running code to choose the number 2X instead, we would end up with a situation where it might be rational for the agent to pay you money to charge it extra for the right to play the game.
Another property is that you can sell the right to play the game to any number of agents, add up all their numbers, and add your profit on top and ask for that much utility.
It seems like the choices for these games obey rather unusual rules. If these choices are allowed to count as "perfectly rational" as per the people who disagree with me that perfect rationality exists, it seems at the very least that perfect rationality is something that behaves very differently from what we might expect.
At the end of the day, I suppose whether you agree with my terminology regarding rationality or not, we can see that there are specific situations where we it seems reasonable to act in a rather strange manner.