FDT is a decision theory and doesn’t in and of itself value anything. The values need to be given by a utility function.
I explicitly said that this difference in values is meant to reproduce the way that FDT/CDT are usually argued to act in these sorts of scenarios, but is actually orthogonal to decision theory per se.
Psychological Twin Prisoner’s Dilemma
This scenario is a stronger one for the decision theory FDT. But that's not the sort of scenario I was referring to: the argument in my comment applies to scenarios where one of the copies makes itself worse off to benefit the others, like the Bomb or transparent Newcomb. These were the main topic of discussion of the post, and I still think it's accurate to say that the difference in intuitions between CDTists/FDTists here comes down to a values/semantic dispute.
I think the difference between "FDT" and "CDT"[1] in these scenarios can be framed as a difference in preferences. "FDT" values all copies of itself equally; "CDT" has indexical values, only caring about the version of itself that it actually finds itself as. As such the debate over which is more "rational" mostly comes down to a semantic dispute.
quotation marks because this difference in preferences is really orthogonal to CDT/FDT but it reproduces the way they are usually argued to act. ↩︎
Reading this inspired me to go back and try Downwell on the PC again, after trying to beat it for a while and giving up several years ago. I'm attempting to use the skill "when things seem grindy and difficult, try identifying the simplest, most obvious improvement you can make to your strategy instead of just trying harder". So far this has included:
Using the "floating" playstyle instead of the default
Playing slowly and carefully on the first two zones, then rushing through the last two as quickly as possible.
Getting apples and other HP restoration items when at full health(I didn't realize they contributed to upgrading your max health)
Despite playing for much less time, I've gotten further than I did a few years ago! I've made it to the boss(standard difficulty) but died halfway through. Next I'm going to try getting combos in the first two zones, to accumulate more gems early.
Have you tried playing Spelunky? I think it might be better than Downwell as a rationality testing ground since it relies less on twitch reaction skills.
Why not? It's someone empirically investigating their beliefs and how well their terminology cuts reality at the joints, doesn't get much more rationalist than that. Not every post has to be at the highest level of rigor or about topics of the gravest importance(especially since it's currently a 'personal blog post', not frontpage)
In the context of that quote, I was saying why I don’t buy the claim that following LDT gives you advantages over committing to, in future problems, do stuff that’s good for you to commit to do *ex ante *even if it would be bad for you *ex post *had you not been committed.
Yes, but isn't this essentially the same as LDT? It seems to me that different sections of your essay are inconsistent with each other, in that in earlier sections you argue that CDT agents might not adopt LDT-recommended policies and so will have problems with bargaining, but in the last section, you say that CDT agents are not at a competitive disadvantage because they can simply commit to act like LDT agents all the time. But if they so commit, the problems with bargaining won't come up. I think it would make more sense to argue that empirically, situations selecting for LDT simply won't arise(but then will arise and be important later).
What is selected-for is being the sort of agent who, *when others observe you, *they update towards doing stuff that’s good for you
I don't quite understand what you mean here - are you saying that CDT agents will only cooperate if they think it will be causally beneficial, by causing them to have a good reputation with other agents? But we were discussing a case(counterfactual mugging) where they would want to pre-commit to act in ways that would be non-causally beneficial. So I think there would be selection to act non-causally in such cases(unless, again, you just think such situations will never arise, but that's a different argument)
Do you have compelling reasons to think you and your counterpart have been selected to have decision procedures that are so strongly logically linked, that your decision to demand more than a fair bargain implies your counterpart does the same
I don't see why you have to assume that your counterpart is strongly logically-linked with you, there are other reasons that you might not want to demand too much. Maybe you know their source code and can simulate that they will not accept a too-high demand. Or perhaps you think, based on empirical evidence or a priori reasoning that most agents you might encounter will only accept a roughly fair allocation.
My uninformed impression was that it had essentially been superceded by deep learning, SGD, etc., so most of the parts relevant to modern AI are already contained in the standard deep learning curriculum. What’s the most important PDP idea that you think is neglected by this curriculum?
I'm also interested in this topic but it feels very hard to directly make progress. It seems to require solving a lot of philosophy, which has as its subject matter the entire universe and how we know about it, so solving metaphilosophy in a really satisfying way seems to almost require rationally apprehending all of existence and our place within it, which seems really hard, or maybe even fundamentally impossible(or perhaps there are ways of making progress in metaphilosophy without solving most of philosophy first, but finding such ways also seems hard)
That said I do have some more indirect/outside-view theories which make me think we could obtain a good future even if we can't directly solve metaphilosophy before getting AGI. I think we can see philosophy as the process by which "messy"/incoherent agents, which arose from evolution or other non-agentic means, can become more coherent and unified-agent-like. So obtaining an AI which can do philosophy would not consist of hardcoding a 'philosophy algorithm', but creating base agents which are messy in similar ways to us, who will then hopefully resolve that messiness and become more coherent in similar ways also(and thereafter make use of the universe in approximately as good of a way as we would have). A whole brain emulation would obviously qualify, but I think that it's also plausible that we could develop a decent high-level understanding of how human brain algorithms work and create AIs that are similar enough for philosophical purposes without literally scanning peoples' brains(which seems like it will take too long relative to AI) For this reason and others I think creating sufficiently-human-like AI is a promising route for obtaining a good future(but this topic also seems curiously neglected).
As to why few other people are trying to solve metaphilosophy, I think there are just very few people with the temperament to become interested in such things, then the few that do end up deciding that some other topic has a better combination of importance/neglectedness/tractability/personal fit to invest major effort in.
With regard to the overall motivations of the post(technical work to reduce future AI conflict), I don't see why most of the problems listed here can't just be left to the AIs. They basically seem like technical problems in decision theory and bargaining which AIs would in theory be good at solving. It's not clear that any work we do now would be of much use to future superintelligences with very strong motivations to solve the problems(and who will also have direct knowledge of the future strategic landscape)
That's usually true but IMO in this case the heavy downvoting was pretty tribalistic in nature. Something about the subject matter makes people think that opposing views are clearly crazy, and they don't bother trying to understand(admittedly omnizoid's posts suffered from this as well)