All of Heighn's Comments + Replies
"Insofar as your distribution has a faraway median, that means you have close to certainty that it isn't happening soon. And that, I submit, is ridiculously overconfident and epistemically unhumble."
Why? You can say a similar thing about any median anyone ever has. Why is this median in particular overconfident?
"And not only do I not expect the trained agents to not maximize the original “outer” reward signal"
Nitpick: one "not" too many?
I apologize, Said; I misinterpreted your (clearly written) comment.
Reading your newest comment, it seems I actually largely agree with you - the disagreement lies in whether farm animals have sentience.
(No edit was made to the original question.)
Thanks for your answer!
I (strongly) disagree that sentience is uniquely human. It seems to me a priori very unlikely that this would be the case, and evidence does exist to the contrary. I do agree sentience is an important factor (though I'm unsure it's the only one).
"but certainly none of the things that we (legally) do with animals are bad for any of the important reasons why torture of people is bad."
That seems very overconfident to me. What are your reasons for believing this, if I may ask? What quality or qualities do humans have that animals lack that makes you certain of this?
I can, although I indeed don't think it is nonsense.
What do you think our (or specifically my) viewpoint is?
Hmm, interesting. I don't know much about UDT. From and FDT perspective, I'd say that if you're in the situation with the bomb, your decision procedure already Right-boxed and therefore you're Right-boxing again, as logical necessity. (Making the problem very interesting.)
Sorry, I'm having trouble understanding your point here. I understand your analogy (I was a developer), but am not sure what you're drawing the analogy to.
I've been you ten years ago.
Just... no. Don't act like you know me, because you don't. I appreciate you trying to help, but this isn't the way.
Seems to me Yudkowsky was (way) too pessimistic about OpenAI there. They probably knew something like this would happen.
To explain my view more, the question I try to answer in these problems is more or less: if I were to choose a decision theory now to strictly adhere to, knowing I might run into the Bomb problem, which decision theory would I choose?
"But by the time the situation described in the OP happens, it no longer matters whether you optimize expected utility over the whole sample space; that goal is now moot."
This is what we agree on. If you're in the situation with a bomb, all that matters is the bomb.
My stance is that Left-boxers virtually never get into the situation to begin with, because of the prediction Omega makes. So with probability close to 1, they never see a bomb.
Your stance (if I understand correctly) is that the problem statement says there is a bomb, so, that's what's true with...
I see your point, although I have entertained Said's view as well. But yes, I could have done better. I tend to get like this when my argumentation is being called crazy, and I should have done better.
You could have just told me this instead of complaining about me to Said though.
I don't see how it is misleading. Achmiz asked what actually happens; it is, in virtually all possible worlds, that you live for free.
Note that it's my argumentation that's being called crazy, which is a large factor in the "antagonism" you seem to observe - a word choice I don't agree with, btw.
About the "needlessly upping the heat", I've tried this discussion from multiple different angles, seeing if we can come to a resolution. So far, no, alas, but not for lack of trying. I will admit some of my reactions were short and a bit provocative, but I don't appreciate nor agree with your accusations. I have been honest in my reactions.
Interesting. I'm having the opposite experience (due to timing, apparently), where at least it's making some sense now. I've seen it using tricks only applicable to addition and pulling numbers out of its ass, so I was surprised what it did wasn't completely wrong.
If you ask ChatGPT to multiply two 4-digit numbers it writes out the reasoning process in natural knowledge and comes to the right answer.
People keep saying such things. Am I missing something? I asked it to calculate 1024 * 2047, and the answer isn't even close. (Though to my surprise, the first 2 steps are at least correct steps, and not nonsense. And it is actually adding the right numbers together in step 3, again, to my surprise. I've seen it perform much, much worse.)
The scenario also stipulates the bomb isn't there if you Left-box.
What actually happens? Not much. You live. For free.
"So if you take the Left box, what actually, physically happens?"
You live. For free. Because the bomb was never there to begin with.
Yes, the situation does say the bomb is there. But it also says the bomb isn't there if you Left-box.
Agreed, but I think it's important to stress that it's not like you see a bomb, Left-box, and then see it disappear or something. It's just that Left-boxing means the predictor already predicted that, and the bomb was never there to begin with.
Put differently, you can only Left-box in a world where the predictor predicted you would.
I think we agree. My stance: if you Left-box, that just means the predictor predicted that with probability close to 1. From there on, there are a trillion trillion - 1 possible worlds where you live for free, and 1 where you die.
I'm not saying "You die, but that's fine, because there are possible worlds where you live". I'm saying that "you die" is a possible world, and there are way more possible worlds where you live.
I'm not going to make you cite anything. I know what you mean. I said Right-boxing is a consequence, given a certain resolution of the problem; I always maintained Left-boxing is the correct decision. Apparently I didn't explain myself well, that's on me. But I'm kinda done, I can't seem to get my point across (not saying it's your fault btw).
By construction it is not, because the scenario is precisely that we find ourselves in one such exceptional case; the posterior probability (having observed that we do so find ourselves) is thus ~1.
Except that we don't find ourselves there if we Left-box. But we seem to be going around in a circle.
… but you have said, in a previous post, that if you find yourself in this scenario, you Right-box. How to reconcile your apparently contradictory statements…?
Right-boxing is the necessary consequence if we assume the predictor's Right-box prediction is fixed now...
No, that's just plain wrong. If you Left-box given a perfect predictor, the predictor didn't put a bomb in Left. That's a given. If the predictor did put a bomb in Left and you Left-box, then the predictor isn't perfect.
Firstly, there’s a difference between “never” and “extremely rarely”.
That difference is so small as to be neglected.
And in the latter case, the question remains “and what do you do then?”. To which, it seems, you answer “choose the Right box”…? Well, I agree with that! But that’s just the view that I’ve already described as “Left-box unless there’s a bomb in Left, in which case Right-box”.
It seems to me that strategy leaves you manipulatable by the predictor, who can then just always predict you will Right-box, put a bomb in Left, and let you Right-box, causing you to lose $1,000.
"Irrelevant, since the described scenario explicitly stipulates that you find yourself in precisely that situation."
Actually, this whole problem is irrelevant to me, a Left-boxer: Left-boxers never (or extremely rarely) find themselves in the situation with a bomb in Left. That's the point.
The bottom line is: to the actual single question the scenario asks—which box do you choose, finding yourself in the given situation?—we give the same answer. Yes?
The bottom line is that Bomb is a decision problem. If I am still free to make a decision (which I suppose I am, otherwise it isn't much of a problem), then the decision I make is made at 2 points in time. And then, Left-boxing is the better decision.
Yes, the Bomb is what we're given. But with the very reasonable assumption of subjunctive dependence, it specifies what I am saying...
We agree that if I would be there, I would Right-box, but also everybody would then Right-box, as a logical necessity (well, 1 in a trillion trillion error rate, sure). It has nothing to do with correct or incorrect decisions, viewed like that: the decision is already hard coded into the problem statement, because of the subjunctive dependence.
"But you can just Left-box" doesn't work: that's like expecting one calculator to answer to 2 + 2 differently than another calculator.
Alright. The correct decision is Left-boxing, because that means the predictor's model Left-boxed (and so do I), letting me live for free. Because, at the point where the predictor models me, the Bomb isn't placed yet (and never will be).
However, IF I'm in the Bomb scenario, then the predictor's model already Right-boxed. Then, because of subjunctive dependence, it's apparently not possible for me to Left-box, just as it is impossible for two calculators to give a different result to 2 + 2.
Hmmm, I thought that comment might clear things up, but apparently it doesn't. And I'm left wondering if you even read it.
Anyway, Left-boxing is the correct decision. But since you didn't really engage with my points, I'll be leaving now.
One thing we do agree on:
If I ever find myself in the Bomb scenario, I Right-box. Because in that scenario, the predictor's model of me already Right-boxed, and therefore I do, too - not as a decision, per se, but as a logical consequence.
The correct decision is another question - that's Left-boxing, because the decision is being made in two places. If I find myself in the Bomb scenario, that just means the decision to Right-box was already made.
The Bomb problem asks what the correct decision is, and makes clear (at least under my assumption) that the deci...
No, it isn't. In the world that's stipulated, you still have to make your decision.
That decision is made in my head and in the predictor's head. That's the key.
The world you're describing is just as much a possible world as the ones I describe. That's my point.
Inferring that I don't burn to death depends on
- Omega modelling my decision procedure
- Cause and effect from there.
That's it. No esoteric assumptions. I'm not talking about a multiverse with worlds existing next to each other or whatever, just possible worlds.
- That I'm being crazy
- That Left-boxing means burning to death
- That your answer is obviously correct
Take your pick.
You infer the existence of me burning to death from what's stated in the problem as well. There's no difference.
I do have the assumption of subjunctive dependence. But without that one - if, say, the predictor predicts by looking at the color of my shoes - then I don't Left-box anyway.
Yeah you keep repeating that. Stating it. Saying it's simple, obvious, whatever. Saying I'm being crazy. But it's just wrong. So there's that.
My point is that those "other worlds" are just as much stipulated by the problem statement as that one world you focus on. So, you pay $100 and don't burn to death. I don't pay $100, burn to death in 1 world, and live for free in a trillion trillion - 1 worlds. Even if I value my life at $10,000,000,000,000, my choice gives more utility.
(Btw, you can call your answer "obvious" and my side "crazy" all you want, but it won't change a thing until you actually demonstrate why and how FDT is wrong, which you haven't done.)
But of course there isn’t actually a contradiction. (Which you know, otherwise you wouldn’t have needed to hedge by saying “in a way”.)
There is, as I explained. There's 2 ways of resolving it, but yours isn't one of them. You can't have it both ways.
It’s simply that the problem says that if you Left-box, then the predictor predicted this, and will not have put a bomb in Left… usually. Almost always! But not quite always. It very rarely makes mistakes! And this time, it would seem, is one of those times.
Just... no. "The predictor predicted this", yes, so th...
Yes, almost perfectly (well, it has to be “almost”, because it’s also stipulated that the predictor got it wrong this time).
Well, not with your answer, because you Right-box. But anyway.
Why does it matter? We know that there’s a bomb in Left, because the scenario tells us so.
It matters a lot, because in a way the problem description is contradicting itself (which happens more often in Newcomblike problems).
- It says there's a bomb in Left.
- It also says that if I Left-box, then the predictor predicted this, and will not have put a Bomb in Left. (Unless you ass
"Irrelevant, since the described scenario explicitly stipulates that you find yourself in precisely that situation."
It also stipulates the predictor predicts almost perfectly. So it's very relevant.
"Yes, that’s what I’ve been saying: choosing Right in that scenario is the correct decision."
No, it's the wrong decision. Right-boxing is just the necessary consequence of the predictor predicting I Right-box. But insofar this is a decision problem, Left-boxing is correct, and then the predictor predicted I would Left-box.
"No, Left-boxing means we burn to death....
Ah, so your complaint is that the author is ignoring evidence pointing to shorter timelines. I understand your position better now :)