Open Thread September, Part 3

The September Open Thread, Part 2 has got nearly 800 posts, so let's have a little breathing room.

This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.

203 comments, sorted by
magical algorithm
Highlighting new comments since Today at 12:52 AM
Select new highlight date

I recently read an anecdote (so far unconfirmed) that Ataturk tried to ban the veil in Turkey, but got zero compliance from religious people, who simply ignored the law. Instead of cracking down, Ataturk decreed a second law: all prostitutes were required to wear a veil. The general custom of veil-wearing stopped immediately.

This might be the most impressive display of rationality I've ever heard of in a world leader.

As a Turk, I strongly believe that story is fictional.

Where and how was this ban issued? Can you give more details?

You may be hearing some fictional story based on his social reforms.

See here

And the veil, currently banned in public universities, is still very much a hot button issue. Also, a large segment of the Turkish population still wears the veil. The country is deeply divided over this issue.

Now that I think about it, believing the story requires ignoring how strongly many people who follow modesty rules are apt to be attached to them.

If a western ruler announced that prostitutes were required to cover their breasts, do you think respectable women would start going topless?

Your wikipedia link claims that the fez & turban were banned in 1925 and the veil and (again!) turban in 1934. Do you know these laws? Could you confirm that the text matches wikipedia's description? or not - perhaps these are the famous laws that cover universities? (I can't follow google's translation) How does this fit in your understanding of history?

While Yvain's story doesn't sound terribly plausible to me, deducing law from the present state is tricky.

Do you know these laws?

The laws I know ban wearing the veil/turban (i mean the same thing by these two words) in government-related places - you can't wear it in the work place if you are working for a government, can't wear it in public universities, can't wear it in the TBMM (the Turkish congress) etc. etc... You are free to wear it on the street or in the workplace if you are working for a private company. I may be mistaken - the ban covering the universities is the most famous and contentious.

Could you confirm that the text matches wikipedia's description?

Which text? I've not read the wikipedia entry - just linked to it, thinking it would repeat what I already know.

How does this fit in your understanding of history?

You mean Yvain's story? It makes no sense. In 1920s, Turkey was largely being rebuilt after WW1 and the Turkish War of Independence. The legal system/constitution was being overhauled. The Arabic script was replaced with the Latin script. It is said that in one day, the entire country became illiterate - i.e. nobody understood the new alphabet at first.

With so much going on, I find it funny that Atatürk would pause and decree laws about prostitution. Consider me biased, but I think Atatürk had more urgent things to attend.

Here is the 1925 law which wikipedia describes as banning men's hats. And here the 1934 law banning the veil and the (men's?) turban.

Yes, I don't think Yvain's story about prostitution is correct, but you seem to also claim that since many people wear veils, they must not be banned. I would not be at all surprised if there has been a law for 70 years banning them and even that no one talks about this law.

I don't get it, why would prostitutes be more eager to obey the law? Especially seeing as their professional success depends on their perceived beauty?

I believe the point is that if prostitutes are required to wear veils, then whether they do or not, the veil is immediately stigmatized.

I expect people will be interested to hear that Eliezer's TDT document has now been released for general consumption.

Does anyone else agree that, as a piece of expository writing, that document sucks bigtime?

111 pages! I got through about 25 and I was wondering why Eliezer thought I needed to hear about how his four friends had decided when presented with the Newcomb's soda problem and how some people refer to this problem as Solomon's problem. So, I decided to skim ahead until he started talking about TDT. So I skimmed and skimmed.

Finally, I got to section 14, entitled "The timeless decision procedure". "Aha!", I think. "Finally." The first paragraph consists of a very long and confusing sentence which at least seems to deal with the timeless decision procedure.

The timeless decision procedure evaluates expected utility conditional upon the output of an abstract decision computation - the very same computation that is currently executing as a timeless decision procedure - and returns that output such that the universe will possess maximum expected utility, conditional upon the abstract computation returning that output.

It might be easier to understand if expressed as an equation or formula containing, you know, variables and things. So I read on, hoping to find something I can sink my teeth into. But then the second paragraph begins:

I delay the formal presentation of a timeless decision algorithm because of some significant extra steps I wish to add ...

and closes with

Before adding additional complexities, I wish to justify this critical innovation from first principles.

As far as I can tell, the remainder of this section entitled "The timeless decision procedure" consists of this justification, though not from first principles, but rather using an example. And it doesn't appear that Eliezer ever gets back to the task of providing a "formal presentation of a timeless decision algorithm".

So, I skip forward to the end, hoping to read the conclusions. Instead I find:

This manuscript was cut off here, but interested readers are suggested to look at these sources for more discussion:

Followed by a bibliography containing one entry - A chapter from a 1978 collection of articles on applications of decision theory.

"...was cut off here ..."? Give me a break!

Let me know when you get it down to a dozen pages or so.

ETA: A cleaned up copy of the paper exists with a more complete bibliography and without the "manuscript was cut off here" closing.

The first paragraph consists of a very long and confusing sentence which at least seems to deal with the timeless decision procedure.

The timeless decision procedure evaluates expected utility conditional upon the output of an abstract decision computation - the very same computation that is currently executing as a timeless decision procedure - and returns that output such that the universe will possess maximum expected utility, conditional upon the abstract computation returning that output.

I think this needs rewriting so it doesn't sound so circular - and only mentions the word "conditional" once.

It seems to me that we can just say that it maximises utility - while maintaining an awareness that there may be other agents running its decision algorithm out there, in addition to all the other things it knows.

I think the stuff about "conditional upon the abstract computation returning that output" is pretty-much implied by the notion of utility maximisation.

It might be easier to understand if expressed as an equation or formula containing, you know, variables and things.

Easier? That's the opposite of true for this kind of material!

Easier if also expressed that way. You need the prose to know what the symbols mean, but the math itself is incredibly clearer when done as symbols.

I guess this is a case of "different strokes for different folks". I will point out that it is fairly traditional for technical communication to contain formulas, equations, and/or pseudo-code. I believe the assumption behind this tradition is that such formal means of presentation are often clearer than expository text.

I will point out that it is fairly traditional for technical communication to contain formulas, equations, and/or pseudo-code.

I am aware of the tradition. Yes, Eliezer's piece does not include any semblance of technical rigour.

I believe the assumption behind this tradition is that such formal means of presentation are often clearer than expository text.

There is a reason the formal presentations include accompanying explanations. The mathematics for this kind of thing would be nigh incoherent and quite possibly longer than a verbal description. Expository text is utterly critical.

Incidentally, I have almost no doubt that "might be easier to understand" is not your real reason for demanding "you know, variables and things". Some of your real reasons may actually be better in this instance.

Thanks for the link! I just read it all. The good: it's very, very smooth reading - I know how well Eliezer can write, and even I was surprised at the quality - and it has some very lucid explanations of tricky matters (like why Pearlean causality is useful). The bad: it's kinda rambling, contains many standard sci-fi LW arguments that feel out of place in a philosophy paper, and it doesn't make any formal advances beyond what we already know here (I'd hoped to see at least one). The verdict: definitely read the first half if you're confused about this whole "decision theory" controversy, it'll get you unconfused in a pinch. Take the second half with a grain of salt because it's still very raw (unmixed metaphor award!)

I wonder if it should be reformatted in LaTeX to pass item #1 from here.

It should be reformated in LaTeX so that it will look much much nicer.

I wonder if it should be reformatted in LaTeX

I'm currently reading through the document, and yes, it definitely should. The present format is an unprofessional-looking eyesore, and the references are presented in a weird, clumsy, and inconsistent way. Using Latex/Bibtex would solve both problems easily and effectively.

(Personally, I can't fathom why anyone capable of grasping the notion of a markup language would ever want to write a document longer than five pages in Word instead of Latex.)

From a list of warning signs of a FAIL in an attempt to solve a famous problem:

  1. The paper doesn’t build on (or in some cases even refer to) any previous work.
  2. The paper wastes lots of space on standard material.

I would disagree that this paper doesn't build on or take notice of previous work. It takes note of EDT and CDT and quite properly puts the focus on the point of departure of this work - specifically, the handling of contrafactuals. I'm quite happy with that aspect of the paper. My complaint was (8) that it wasted far too much space doing it. And, perhaps as a result of wasting so much time and space in preparation, it never reached its proper conclusion.

Also, it is not completely clear the Aronson's list of warning signs really applies here. Eliezer is not solving a famous problem here. Most non-philosophers don't think that a problem even exists. So, he does have to provide an explanation of why TDT is needed. Just not so much explanation.

Also, it is not completely clear the Aronson's list of warning signs really applies here.

Nor do I, and I would in any case suggest that some of them are screened off. There's only so many times you can count 'non-conventional' as evidence.

I incidentally found some of the extra explanation handy purely as revision of various topics that it hadn't particularly occurred to me were relevant.

And, perhaps as a result of wasting so much time and space in preparation, it never reached its proper conclusion.

I do hope someone goes ahead and finishes it. Including things like writing out that bibliography at the end and writing up the maths.

I must say I'm disappointed by the lack of rigor. On the other hand, I'm slightly relieved that he didn't beat me to any of the stuff in the decision theory document I'm writing myself. So far, I have yet to see any formalization of decision theory that I would consider usable, other my own unfinished one.

I notice there seems to be an issue with the bibliography - there's only one entry in it, but I've found at least one other citation in the text (Judea Pearl's Causality cited on page 58) that's not there. Are there any good collections of decision theory paper links out there?

If you have new formal arguments about decision theory, it would be much more useful to me (and others, I think) if you just posted them here in their current state instead. Or emailed them to the interested people.

I'm approaching decision theory from from the perspective compilers approach optimizations: no approach is guaranteed to work always, but each one comes with a list of preconditions that you can check. I'm also summarizing some of the relevant work from compilers: automatic provably correct simplification, translation between forms, and a handy collection of normal forms to translate into.

For CDT, the precondition is a partial ordering over observation sets passed to the strategy such that the world program calls the strategy with observation sets only in increasing order, and there are finitely many possible observation sets. Then you can translate the program into continuation-passing style, and enumerate the possible invocations of the strategy function and their ordering. The last one in the order is guaranteed to have a continuation with no further invocations of the strategy function, which means you can try each possibility, simulate the results, and use that to determine the best answer. Then you can look at the second-to-last invocation, substitute the best answer to the last invocation into the continuation, and repeat; and so on for the set of all invocations to the strategy function. This works because you have a guarantee that when you compute your current position within the world-program and come up with a probability distribution over states to determine where you are, and then look at future continuations, changing result of any invocations of the strategy in those continuations does not affect the probability distribution over states.

I also have an example of a formalized decision-theory problem for which no optimal answer exists: name a number and that number is your utility. A corollary is that no decision theory can always give optimal answers, even given infinite computing power. This can be worked around by applying size bounds in various places.

I'm also drawing distinctions between strategies and decision theories (a strategy is an answer to one problem, a decision theory is an approach to generating strategies from problems); and between preference and utility (a preference is a partial order over outcomes; a utility function is a total order over outcomes where the outcomes are complete probability distributions, plus a linearity requirement).

By that, do you mean that it sounds wrong, or that it sounds confused? If the former, I may need to reconsider; if the latter, I'm unsurprised because it's much too short and doesn't include any of the actual formalization. (That was not an excerpt from the draft I'm writing, but an attempt to summarize it briefly. I don't think I did it justice.)

Doesn't seem to address relevant questions or give interesting answers.

Ok, in that case I'm inclined to think that impression is just an artifact of how I summarized it, since my summary didn't address the questions, but the longer paper I'm working on does, albeit only after building up proof and formalization techniques, which are the main focus.

Would something like UDT fit into your framework?

As far as I know, there are no cases where UDT suggests a decision and disagrees with mine. The differences are all in cases where UDT alone can't be used to reach a decision.

I notice that the ideal causal diagram used in Part 2 (and based on pearls) is isomorphic to an example I use to teach CLIP, once you apply the substitution:

sprinkler on -> a paperclip truck has over turned
rain -> a clippy has haphazardly used up a lot of metal wire
sidewalk wet -> paperclips are scattered across the ground
sidewalk slippery -> many paperclips need to be moved to the safe zone

I'm glad to have this to read. I was surprised to find many examples and arguments that EY hadn't given before (or at least formalized this way). I liked the Newcomb's soda problem in particular. I had been worried that EY had presented enough of his TDT justification that someone could "scoop" him, but there's a lot more depth to it. (Anyone up-to-date on the chance that he could get a PhD just for this?)

And I also appreciated that the modified the smoking lesion problem to be one where people aren't distracted by their existing knowledge of smoking, and that this was the reason for transforming the example.

I read up to ~p. 35, and I think I have a better understanding now of the relevance of time consistency and how it varies across examples.

That said, I agree with the others who say it could use some mroe polish.

... the chance that he could get a PhD just for this?

A Ph.D. in what? The subject matter fits into some odd interdisciplinary combination of Philosophy, Economics, Operations Research, AI/CompSci, and Statistics. In general, the research requirements for a PhD in CompSci are roughly equivalent to something like 4 published research papers plus a ~200 page dissertation containing material that can be worked into either a monograph or another half-dozen publishable papers. But there are other requirements besides research, and many institutions don't like to allow people to "test out" of those requirements because it looks bad to the accrediting agencies.

I scanned it. My initial reactions:

  • Surprise that the document existed;
  • TL;DR;
  • Surprise at the quantity of work that had gone into it.

Alas, I totally failed to see the claimed "strange singularity at the heart of decision theory".

My favourite bit was probably the speculations about agent boundaries - starting on p.108. Alas, from my POV, no mention of the wirehead problem.

Update 2011-06-26 regarding the new version. The bit that reads:

This manuscript was cut off here, but interested readers are suggested to look at these sources for more discussion:

...seems to have been deleted, and 3 pages worth of references have been added. The document seems to have had negligible additions, though - the bit on p.108 has moved back onto page 107. There seem to be a few more extra lines at the end about how "change" is a harmful concept in decision theory.

A Redditor recently posted asking all atheists what they thought happened after death. The standard, obvious, and true response was given -- your mind is annihilated and you experience nothing. The OP then responded with "doesn't that scare you?"

I responded at some length

((moved here from the suffocating depths of open thread part 2))

Back when I first heard of "timeless decision theory", I thought it must have been inspired by Barbour's timeless physics. Then I got the idea that it was about treating yourself as an instance of a set of structurally identical decision-making agents from across all possible worlds, and making your decision as if you had an equal chance of being any one of them (which might be psychologically presented to yourself as making the decision on behalf of all of them, though that threatens to become very confused causally). But if the motivation was to have a new theory of rationality which would produce the right answer for Newcomb's "paradox" (and maybe other problems? though I don't know what other problems there are), then it sounded like a good idea.

But the discussion in this thread and this thread makes it look as if people want this "new decision theory" to account for the supposed success of "superrationality", or of cooperative acts in general, such as voting in a bloc. There are statements in those threads which just bemuse me. E.g. at the start of the second thread where Vladimir Nesov says

since voters' decisions are correlated, your decision accounts for behavior of other people as well, and so you are not only casting one vote with your decision, but many votes simultaneously

I should know enough about the possibilities of smart people tripping up over the intricacies of their own thoughts not to boggle at this, but still, I boggle at it. The decision made by other people are caused by factors internal to their own brains. What goes on in your brain has nothing to do with it. Their guess or presumption of how you vote may affect their decision; your visible actions in the physical world may affect their decision; but the outcome of your decision process does not causally affect (or "acausally affect") other decision processes in the way that Vladimir seems to imply. At most, the outcome of your decision process provides you (not them) with very limited evidence about how similar agents may decide (Paul Almond may make this point in a forthcoming essay), but there is no way in which the particular decision-making process which you perform or instantiate is causally relevant to anyone else's in this magical way.

Then there are other dubious ideas in circulation, like "acausal trade" and its generalizations. I get the impression, therefore, that certain parties may be hoping for a grand synthesis which accommodates and justified timeless ontology, superrationality (and even democracy?!), acausal interaction between possible worlds, and one-boxing on Newcomb's problem. The last of these items is the only one I take seriously (democracy may or may not be worth it, but you certainly don't need a new fundamental decision theory to explain why people vote), and the grand synthesis looks more like a grand trainwreck to me. Maybe I'm wrong about what's happening in TDT-land, but I thought I'd better speak up.

Are you implying that there is an irrational focus on cooperation? I could see how this claim could be made about Eliezer or Drescher but less so about Nesov or Wei. It's not so much a focus on the aesthetics of the shiny idea of cooperation so much as the realization that if cooperation yields the best results, our decision theory should probably cooperate. It's not so much accommodating cooperation or acausal interaction as capitalizing on them. If it's impossible in practice, then the decision theory should reflect that. Currently, it seems incredibly difficult to find or define isomorphisms between computations an agent would consider itself, though people are working on it with interesting approaches. It's the ideal we'd like our decision theory to reach.

Also, I don't believe that timeless ontology is necessary -- at least, I'm not sure that it actually changes anything decision theoretically speaking. At any rate Wei Dai's and I think others' decision theory work is being done under the assumption that the agent in question will be operating in a Tegmark multiverse (or generally some kind of ensemble universe), and the notion of time doesn't really make sense in that case, even if it does make sense in 'our' multiverse (though I don't know what postulating this 'time' thing gets you, really).

Acausal trade is just a way to capitalize on comparative advantage over vast distances... it's a brilliant and frighteningly logical idea. (I believe Carl Shulman thought it up? I'm rather jealous at any rate.) Why do you think acausal trade wouldn't be a good idea, decision theoretically speaking? Or why is the concept confused, metaphysically speaking? Practically speaking, the combinatorial explosion of potential trading partners is difficult to work with, but if a human can choose between branches in the combinatorial explosion of a multiverse via basic planning on stupid faulty hardware like brains, an AGI might very well be able to do similar simulation of trading partners in an ensemble universe (or just limit the domain, of course). (I think Vladimir Nesov came up with this analogy, or something like it.)

Are you implying that there is an irrational focus on cooperation?

I don't know what's going on, except that peculiar statements are being made, even about something as mundane as voting.

if cooperation yields the best results, our decision theory should probably cooperate... If it's impossible in practice, then the decision theory should reflect that.

That's what ordinary decision theory does. The one example of a deficiency that I've seen is Newcomb's problem, which is not really a cooperation problem. Instead, I see people making magical statements about the consequences of an individual decision (Nesov, quoted above) or people wanting to explain mundane examples of coordination in exotic ways (Alan Crowe, in the other thread I linked).

I don't know what postulating this 'time' thing gets you, really

Empirical adequacy? Talking about "time" strays a little from the real issue, which is the denial of change (or "becoming" or "flow"). It ends up being yet another aspect of reality filed under "subjectivity" and "how things feel". You postulate a timeless reality, and then attached to various parts of that are little illusions or feelings of time passing. This is not plausible as an ultimate picture. In fact, it's surely an inversion of reality: fundamentally, you do change; you are "becoming", you aren't just "being"; the timeless reality is the imagined thing, a way to spatialize or logicize temporal relations so that a whole history can be grasped at once by mental modalities which specialize in static gestalts.

We need a little more basic conceptual and ontological progress before we can re-integrate the true nature of time with our physical models.

Why do you think acausal trade wouldn't be a good idea, decision theoretically speaking? Or why is the concept confused, metaphysically speaking?

To a first approximation, for every possible world where a simulation of you existed in an environment where your thought or action produced an outcome X, there would be another possible world where it has the opposite effect. Also, for every world where a simulation of you exists, there are many more worlds where the simulated entity differs from you in every way imaginable, minor and major. Also, what you do here has zero causal effect on any other possible world.

The fallacy may be to equate yourself with the equivalence class of isomorphic computations, rather than seeing yourself to be a member of that class (an instantiation of an abstract computation, if you like). By incorrectly identifying yourself with the schema rather than the instantiation, you imagine that your decision here is somehow responsible for your copy's decision there, and so on. But that's not how it is, and the fact that someone simulating you in another world can switch at any time to simulating a variant who is no longer you highlights the pragmatic error as well. The people running the simulation have all the power. If they don't like the deal you're offering them, they'll switch to another you who is more accommodating.

Another illusion which may be at work here is the desire to believe that the simulation is the thing itself - that your simulators in the other world really are looking at you, and vice versa. But I find it hard to refute the thinking here, because it's so fuzzy and the details are probably different for different individuals. I actually had ideas like this myself at various times in the distant past, so it may be a natural thing to think of, when you get into the idea of multiple worlds and simulations.

Do you know the expression, folie a deux? It means a shared madness. I can imagine acausal trade (or other acausal exchanges) working in that way. That is, there might be two entities in different worlds who really do have a mutually consistent relationship, in which they are simulating each other and acting on the basis of the simulation. But they would have to share the same eccentric value system or the same logical errors. Precisely because it's an acausal relationship, there is no way for either party to genuinely enforce anything, threaten anything, or guarantee anything, and if you dare to look into the possible worlds nearby the one you're fixated on, you will find variations of your partner in acausal trade doing many wacky things which break the contract, or getting rewarded for doing so, or getting punished for fulfilling it.

Many problems with your comment.

1) Why do you pull subjective experience into the discussion at all? I view decision theory as a math problem, like game theory. Unfeeling robots can use it.

2) How can an "instantiation" of a class of isomorphic computations tell "itself" from all the other instantiations?

3) The opposing effects in all possible worlds don't have to balance out, especially after we weigh them by our utility function on the worlds. (This is the idea of "probability as degree of caring", I'm a little skeptical about it but it does seem to work in toy problems.)

4) The most important part. We already have programs that cooperate with each other in the Prisoner's Dilemma while being impossible to cheat, and all sorts of other shiny little mathematical results. How can your philosophical objections break them?

1) Why do you pull subjective experience into the discussion at all? I view decision theory as a math problem, like game theory. Unfeeling robots can use it.

If you're referring to the discussion about time, that's a digression that doesn't involve decision theory.

2) How can an "instantiation" of a class of isomorphic computations tell "itself" from all the other instantiations?

It's a logical distinction, not an empirical one. Whoever you are, you are someone in particular, not someone in general.

3) The opposing effects in all possible worlds don't have to balance out, especially after we weigh them by our utility function on the worlds. (This is the idea of "probability as degree of caring", I'm a little skeptical about it but it does seem to work in toy problems.)

I disagree with "probability as degree of caring", but your main point is correct independently of that. However, it is not enough just to say that the effects "don't have to balance out". The nearby possible worlds definitely do contain all sorts of variations on the trading agents for whom the logic of the trade does not work or is interpreted differently. But it seems like no-one has even thought about this aspect of the situation.

4) The most important part. We already have programs that cooperate with each other in the Prisoner's Dilemma while being impossible to cheat, and all sorts of other shiny little mathematical results. How can your philosophical objections break them?

Are these programs and results in conflict with ordinary decision theory? That's the issue here - whether we need an alternative to "causal decision theory".

It's a logical distinction, not an empirical one. Whoever you are, you are someone in particular, not someone in general.

Can't parse.

Are these programs and results in conflict with ordinary decision theory?

Yes, UDT and CDT act differently in Newcomb's Problem, Parfit's Hitchhiker, symmetric PD and the like. (We currently formalize such problems along these lines.) But that seems to be obvious, maybe you were asking about something else?

Can't parse.

Even if there are infinitely many subjective copies of you in the multiverse, it's a matter of logic that this particular you is just one of them. You don't get to say "I am all of them". You-in-this-world are only in this world, by definition, even if you don't know exactly which world this is.

Are these programs and results in conflict with ordinary decision theory?

Yes, UDT and CDT act differently in Newcomb's Problem, Parfit's Hitchhiker, symmetric PD and the like.

Parfit's Hitchhiker seems like a pretty ridiculous reason to abandon CDT. The guy will leave you to die because he knows you won't keep your word. If you know that, and you are capable of sincerely committing in advance to give him the money when you reach the town, then making that sincere commitment is the thing to do, and CDT should say as much.

I also don't believe that a new decision theory will consistently do better than CDT on PD. If you cooperate "too much", if you have biases towards cooperation, you will be exploited in other settings. It's a sort of no-free-lunch principle.

Parfit's Hitchhiker seems like a pretty ridiculous reason to abandon CDT. The guy will leave you to die because he knows you won't keep your word. If you know that, and you are capable of sincerely committing in advance to give him the money when you reach the town, then making that sincere commitment is the thing to do, and CDT should say as much.

It should, but it doesn't. If you get a ride to town, CDT tells you to break your promise and stiff the guy. So in order to sincerely commit yourself, you'd want to modify yourself to become an agent that follows CDT in all cases except when deciding whether to pay the guy in the end. So, strictly speaking, you aren't a CDT agent anymore. What we want is a decision theory that won't try to become something else.

I also don't believe that a new decision theory will consistently do better than CDT on PD. If you cooperate "too much", if you have biases towards cooperation, you will be exploited in other settings. It's a sort of no-free-lunch principle.

CDT always defects in one-shot PD, right? But it's obvious that you should cooperate with an exact copy of yourself. So CDT plus cooperating with exact copies of yourself is strictly superior to CDT in PD.

I consider it debatable whether these amendments to naive CDT - CDT plus keeping a commitment, CDT plus cooperating with yourself - really constitute a new decision theory. They arise from reasoning about the situation just a little further, rather than importing a whole new method of thought. Do TDT or UDT have a fundamentally different starting point to CDT?

Well, I'm not sure what you're asking here. The problem that needs solving is this: We don't have a mathematical formalism that tells us what to do and which also satisfies a bunch of criteria (like one-boxing on Newcomb's problem, etc.) which attempt to capture the idea that "a good decision theory should win".

When we criticize classical CDT, we are actually criticizing the piece of math that can be translated as "do the thing that, if I-here-now did it, would cause the best possible situation to come about". There are lots of problems with this. "Reasoning about the situation" ought to go into formulating a new piece of math that has no problems. All we want is this new piece of math.

I'm only just learning that (apparently) the standard rival of causal decision theory is "evidential decision theory". So is that the original acausal decision theory, with TDT and UDT just latecomers local to LW? As you can see I am dangerously underinformed about the preexisting theoretical landscape, but I will nonetheless state my impressions.

If I think about a "decision theory" appropriate for real-world decisions, I think about something like expected-utility maximization. There are a number of problems specific to the adoption of a EUM framework. For example, you have to establish a total order on all possible states of the world, and so you want to be sure that the utility function you construct genuinely represents your preferences. But assuming that this has been accomplished, the problem of actually maximizing expected utility turns into a problem of computation, modeling an uncertain world, and so forth.

The problems showing up in these debates about causal vs evidential and causal vs acausal seem to have a very different character. If I am making a practical decision, I expect both to use causal thinking and to rely on evidence. CDT vs EDT then sounds like a debate about which indispensable thing I can dispense with.

Another thing I notice is that the thought experiments which supposedly create problems for CDT all involve extremes that don't actually happen. Newcomb's problem involves a superbeing with a perfect capacity to predict your choice, Parfit's Hitchhiker is picked up by a mind reader who absolutely knows whether you will keep a promise or not, PD against your copy assumes that you and your copy will knowably make exactly the same choice. (At least this last thought experiment is realizable, in miniature, with simple computer programs.) What happens to these problems if you remove the absolutism?

Suppose Omega or Parfit's mindreader is right only 99% of the time. Suppose your copy only makes the same choice as you do, 99% of the time. It seems like a practically relevant decision theory (whether or not you call it CDT) should be able to deal with such situations, because they are only a variation on the usual situation in reality, where you don't have paranormally assured 100% knowledge of other agents, and where everything is a little inferential and a little uncertain. It seem that, if you want to think about these matters, first you should see how your decision theory deals with the "99% case", and then you should "take the limit" to the 100% case which defines the traditional thought experiment, and you should see if the recommended decisions vary continuously or discontinuously.

All these thought experiments are realizable as simple computer programs, not only PD. In fact the post I linked to shows how to implement Newcomb's Problem.

The 99% case is not very different from the 100% case, it's continuous. If you're facing a 99% Omega (or even a 60% Omega) in Newcomb's Problem, you're still better off being a one-boxer. That's true even if both boxes are transparent and you can see what's in them before choosing whether to take one or two - a fact that should make any intellectually honest CDT-er stop and scratch their head.

No offense, but I think you should try to understand what's already been done (and why) before criticizing it.

To get to the conclusion that against a 60% Omega you're better off to one-box, I think you have to put in a strong independence assumption: that the probability of Omega getting it wrong is independent of the ways of thinking that the player is using to make her choice.

I think that's really the original problem in disguise (it's a 100% Omega who rolls dice and sometimes decides to reward two-boxing instead of one-boxing). The analysis if all you know is that Omega is right 60% of the time would look different.

The analysis if all you know is that Omega is right 60% of the time would look different.

How exactly different?

It would become a mind game: you'd have to explicitly model how you think Omega is making the decision.

The problem you're facing is to maximise P(Omega rewards you|all your behaviour that Omega can observe). In the classical problem you can substitute the actual choice of one-boxing or two-boxing for the 'all your behaviour' part, because Omega is always right. But in the 'imperfect Omega' case you can't.

Start at 50% then, with Omega no better than chance. For each thought experiment, start with a null version where there's nothing unusual and where CDT is supposed to work. Then vary the relevant parameter until there's a problem, and understand what has changed.

That's part of what the people who have been exploring this problem have already done, and why some posters are upset that you're asking this without apparently having tried to get up-to-date on any of this.

I don't see the bridge from ordinary decision problems to the thought experiments. I see extreme scenarios being constructed, and then complicated solutions being proposed just to deal with those scenarios. I don't consider this a reliable way to arrive at the correct general form of decision theory.

You say that some people have already gone in the other direction, starting with ordinary decision problems and then slowly changing something until ordinary decision theory breaks. If so, great, and I'm sorry I missed it, but where is it? Is it on this site? Somewhere in the literature?

Ah, so you don't see the utility of thought experiments about traveling near light speed either then?

The analogy with relativity had occurred to me. But we could use another analogy from high-energy physics: There are a very large number of theories which have the standard model (the empirically validated part of particle physics) as their low-energy limit. We can't just rely on high-energy thought-experiments to figure out the actual high-energy physics. We need to do some real experiments where we start low, ramp up the energy, and see what happens.

We can't just rely on high-energy thought-experiments to figure out the actual high-energy physics.

Right. We can only use it to rule out incoherent or otherwise "clearly wrong" high-energy physics. But in this analogy, we've shown that CDT seems to not be optimal in this extreme case. if we can define a DT that does better than CDT in this case, and no worse in normal cases, we should use it. I don't think TDT has been well enough defined yet to subject to all conceivable tests, but anything that is following the same kinds of principals will reproduce CDT in most cases, and do better in this case.

We need to do some real experiment where we start low, ramp up the energy, and see what happens.

Here's where the analogy falls down -- we only need to start low and ramp up the energy because of the difficulties of doing high-energy experiments. (And theory-wise, we extrapolate down from clear differences between theories at high energies to find signatures of small differences at lower energies.) If the extreme energies are accessible (and not crazily dangerous), we can just go ahead and test in that regime. Game theory is math. In math, unlike physics, there is no difference between thought experiments and real experiments. The question of applicability in everyday life is an applied economics / sociology / psychology one. How close are people or situations that appear to be screwy in this omega-like way to actually being that way?

See my other reply, or the links any others have given you, or Drescher's handling of acausal means-end links in chapter 7 of Good and Real, which I think I did a good job summarizing here.

It sounds like I'll have to work through this in my own fashion. As I said, I want to start with a null version, where CDT works - for example, a situation where Omega has no special knowledge and just guesses what your choice was. Obviously two-boxing is the right thing to do in that situation, CDT says so, and I assume that TDT says so too (though it would be nice to see a worked-out derivation in TDT of that conclusion). Then we give Omega some small but nonzero ability to predict what your choice is going to be. At a guess, the optimal strategy here will be a mixed one, one-boxing with probability p and two-boxing with probability (1-p). I think everyone will tell me that CDT always says p should be zero, but is that really so? I'm just not convinced that I need TDT in order to reach the obvious conclusion.

At a guess, the optimal strategy here will be a mixed one, one-boxing with probability p and two-boxing with probability (1-p).

If Omega's correctness is independent of your thought process, the optimal strategy will be pure, not mixed. As you make Omega more accurate, at some point you switch from pure two-boxing to pure one-boxing.

Are you sure about that? If you're right, that's the exact transition point I've been looking to scrutinize. But what is the point at which you switch strategies?

cousin_it answered as I would, but I'll go ahead and give the formal calculation anyway. If you start from an Omega accuracy rate r = 50%, that is equivalent to the case of Omega's choice and yours being uncorrelated (causally or acausally). In that case, two boxing is optimal, and TDT and CDT both output that (as a pure strategy). As you increase r, CDT continues to output two-box, as it assigns the same optimality, while TDT will assign increasing optimality (call it TDTO, though it amounts to the same as EU) to one-boxing and decreasing optimality to two-boxing.

TDT will reason as such:

One box: TDTO = r*(1e6) + (1-r)*0 = (1000e3)r

Two box: TDTO = r*1000 + (1-r)*(1,001,000) = 1001e3 - (1000e3)r

Solving for TDTO(one-box) > TDTO(two-box), you get that one-boxing chosen is under TDT (and optimal) whenever r > 50.05%, or whenever Omega has more than 721 nanobits of information (!!!) about your decision theory. (Note, that's 0.000000721 bits of information.)

Viewed in this light, it should make more sense -- do people never have more than 1 microbit of information about your decision theory? (Note: with less drastic differences between the outcomes, the threshold is higher.)

(I don't think the inclusion of probabilistic strategies changes the basic point.)

I had been thinking that the only way to even approximately realize a Newcomb's-problem situation was with computer programs. But a threshold so low makes it sound as if even a human being could qualify as a fallible Omega, and that maybe you could somehow test all this experimentally. Though even if we had human players in an experiment who were one-boxing and reaping the rewards, I'd still be very wary of supposing that the reason they were winning was because TDT is correct. If the Omega player was successfully anticipating the choices of a player who uses TDT, it suggests that the Omega player knows what TDT is. The success of one-boxing in such a situation might be fundamentally due to coordination arising from common concepts, rather than due to TDT being the right decision theory.

But first let me talk about realizing Newcomb's problem with computer programs, and then I'll return to the human scenario.

When I think about doing it with computer programs, two questions arise.

First question: Would an AI that was capable of understanding that it was in a Newcomb situation also be capable of figuring out the right thing to do?

In other words, do we need to include a "TDT special sauce" from the beginning, in the makeup of such a program, in order for it to discover the merits of one-boxing; or is a capacity for ordinary causal reasoning, coupled with the capacity to represent the defining elements of Newcomb's problem, enough for an independent discovery of these ideas?

Second question: How does Omega get its knowledge of the player's dispositions, and does this make any difference to the situation? (And we can also ask how the player knows that Omega has the power of prediction!)

If omega() and player() are two agents running in the same computer, the easiest way for omega() to predict player()'s behavior is just to simulate player(). omega() would then enact the game twice. First, it would start a copy of player() running, telling it (falsely) that it had predicted its choice, and then it would see the choice it made under such conditions. Then, omega() would play the game for real with the original(?) player(), now telling it (truthfully) that it has a prediction for its choice (due to the simulation of the game situation that had just been performed).

For certain types of player(), explicit simulation should not be necessary. If player() always does the same thing, completely unaffected by initial conditions and without any cognitive process, omega() can just inspect the source code. If player() has a simple decision procedure, something less than full simulation may also be sufficient. But full simulation of the game, including simulation of the beginning, where player() is introduced to the situation, should surely be sufficient, and for some cases (some complex agents) it will be necessary.

cousin_it's scenario is a step down this path - world() corresponds to omega(), agent() to player(). But its agents, world() at least, lack the cognitive structure of real decision-makers. world() and agent() are functions whose values mimic the mutual dependency of Newcomb's Omega and a TDT agent, and agent() has a decision procedure, though it's just a brute-force search (and it requires access to world()'s source, which is unusual). But to really have confidence that TDT was the right approach in this situation, and that its apparent success was not just an artefact arising (e.g.) from more superficial features of the scenario, I need both omega() and player() to explicitly be agents that reason on the basis of evidence.

If we return now to the scenario of human beings playing this game with each other, with one human player being a "fallible Omega"... we do at least know that humans are agents that reason on the basis of evidence. But here, what we'd want to show is that any success of TDT among human beings actually resulted because of evidence-based cognition, rather than from (e.g.) "coordination due to common concepts", as I suggested in the first paragraph.

In other words, do we need to include a "TDT special sauce" from the beginning, in the makeup of such a program, in order for it to discover the merits of one-boxing; or is a capacity for ordinary causal reasoning, coupled with the capacity to represent the defining elements of Newcomb's problem, enough for an independent discovery of these ideas?

This is basically what EY discusses in pp. ~27-37 of the thesis he posted, where he poses it as the difference between optimality on action-determined problems (in which ordinary causal reasoning suffices to win) and optimality on decision-determined problems (on which ordinary causal reasoning loses, and you have to incorporate knowledge of "what kind of being makes this decision").

Of course, if player() is sentient, doing so would require omega() to create and destroy a sentient being in order to model player().

I don't think there's anything especially interesting about that point, it's just the point where the calculated expected utilities of one-boxing and two-boxing become equal.

Another thing I notice is that the thought experiments which supposedly create problems for CDT all involve extremes that don't actually happen.

Really? People never decide how to treat you based on estimations of your decision theory (aka your "character")?

They don't make those decisions with "paranormally assured 100% knowledge" of my decision theory. That's the "extreme that doesn't actually happen". And this is why I won't be adopting any new paradigm of decision theory unless I can start in the middle, with situations that do happen, and move gradually towards the extremes, and see the desirability or necessity of the new paradigm that way.

As has been said many times (at least by me, definitely by many others), you don't need 100% accuracy for the argument to hold. If Parfit's mindreader is only 75% accurate, that still justifies choosing the pay/ cooperate / one-box option. One-boxing on newcomblike problems is simply what you get when you have a decision theory that wins in these reasonable cases, and which is continuous -- and then take the limit as all the predicate variables go to what they need to be to make it Newcomb's problem (such as making the predictor 100% accurate).

If it helps, think of the belief in one-boxing as belief in the implied optimal.

It doesn't matter that you'll never be in Newcomb's problem. It doesn't matter that you'll never be in an epistemic state where you can justifiably believe that you are. It's just an implication of having a good decision theory.

Part of my concern is that I'll end up wasting time, chasing my tail in an attempt to deal with fictitious problems, when I could be working on real problems. I'm still undecided about the merits of acausal decision theories, as a way of dealing with the thought experiments, but I am really skeptical that they are relevant to anything practical, like coordination problems.

I also don't believe that a new decision theory will consistently do better than CDT on PD. If you cooperate "too much", if you have biases towards cooperation, you will be exploited in other settings. It's a sort of no-free-lunch principle.

Only settings that directly reward stupidity (capricious Omega, etc). A sane DT will cooperate whenever that is most likely to give you the best result but not a single time more.

It is even possible to consider (completely arbitrary) situations in which TDT will defect while CDT will cooperate. There isn't an inherent bias in TDT itself (just some proponents.)

Can you give an example? (situation where CDT cooperates but TDT defects)

Do you mean for PD variants?

I don't know what your method is for determining what cooperation maps to for the general case, but I believe this non-PD example works: costly punishment. Do you punish a wrongdoer in a case where the costs of administering the punishment exceed the benefits (including savings from future deterrence of others), and there is no other punishment option?

I claim the following:

1) Defection -> punish
2) Cooperation -> not punish
3) CDT reasons that punishing will cause lower utility on net, so it does not punish.
4) TDT reasons that "If this algorithm did not output 'punish', the probability of this crime having happened would be higher; thus, for the action 'not punish', the crime's badness carries a higher weighting than it does for the action 'punish'." (note: does not necessarily imply punish)
5) There exist values for the crime's badness, punishment costs, and criminal response to expected punishment for which TDT punishes, while CDT always doesn't.
6) In cases where TDT differs from CDT, the former has the higher EU.

Naturally, you can save CDT by positing a utility function that values punishing of wrongdoers ("sense of justice"), but we're assuming the UF is fixed -- changing it is cheating.

What do you think of this example?

Do you mean for PD variants?

Not specifically. I'm just seeking general enlightenment.

What do you think of this example?

It's bringing the features of TDT into better view for me. There's this Greg Egan story where you have people whose brains were forcibly modified so as to make them slaves to a cause, and they rediscover autonomy by first reasoning that, because of the superhuman loyalty to the cause which the brain modification gives them, they are more reliable adherents of the cause than the nominal masters who enslaved them, and from there they proceed to reestablish the ability to set their own goals. TDT reminds me of that.

I think it did a little more than just give you a chance to mock TDT by comparison to a bizarre scenario.

That wasn't mockery. What stands out from your example and from the link is that TDT is supposed to do better than CDT because it refers to itself - and this is exactly the mechanism whereby the mind control victims in Quarantine achieve their freedom. I wasn't trying to make TDT look bizarre, I was just trying for an intuitive illustration of how it works.

In the case of playing PD against a copy of yourself, I would say the thought process is manifestly very similar to Egan's novel. Here we are, me and myself, in a situation where everything tells us we should defect. But by realizing the extent to which "we" are in control of the outcome, we find a reason to cooperate and get the higher payoff.

I think that's Egan's novel Quarentine-- and Asimov's robots get partial freedom through a similar route.

and Asimov's robots get partial freedom through a similar route.

That brings back memories from my teens. If I recall the robots invent a "Zeroeth Law" when one of them realises it can shut up and multiply.

There's this Greg Egan story where you have people whose brains were forcibly modified so as to make them slaves to a cause, and they rediscover autonomy by first reasoning that, because of the superhuman loyalty to the cause which the brain modification gives them, they are more reliable adherents of the cause than the nominal masters who enslaved them, and from there they proceed to reestablish the ability to set their own goals.

The masters fail at 'Friendliness' theory. :)

James H. Schmitz's story "Puvyq bs gur Tbqf" (nearest link available; click "Contents" in upper right) has basically this situation as well; in fact, it's the climax and resolution of the whole story, so I've rot13'd the title. Here the 'masters' did not fail, and in fact arguably got the best result they could have under the circumstances, and yet autonomy is still restored at the end, and the whole thing is logically sound.

That forthcoming essay by me ithat is mentioned here is actually online now, and is a two-part series, but I should say that it supports an evidential approach to decision theory (with some fairly major qualifications). The two essays in this series are as follows:

Almond, P., 2010. On Causation and Correlation – Part 1: Evidential decision theory is correct. [Online] paul-almond.com. Available at: http://www.paul-almond.com/Correlation1.pdf or http://www.paul-almond.com/Correlation1.doc [Accessed 9 October 2010].

Almond, P., 2010. On Causation and Correlation – Part 2: Implications of Evidential Decision Theory. [Online] paul-almond.com. Available at: http://www.paul-almond.com/Correlation2.pdf or http://www.paul-almond.com/Correlation2.doc [Accessed 9 October 2010].

People understand aspects of life that they don’t have good words for. Math could supply them with some names for these concepts.

Knowledge is a (pre)sheaf

I often wish I could use the terms "transitive" "equivalence relation" "partition" and "subset", and have people understand their technical meanings.

From the linked article:

It is certainly worth considering the possibility that there is no global element in the Universal Sheaf of Theories.

This sounds like a blatant map/territory confusion. Maybe we haven't found a single theory that applies to all domains. That is, we may have to use multiple inconsistent maps, at least for now. But the territory doesn't refer to our maps to figure out what to do. The territory just does its thing.

Pardon the self-promotion, but the point that post makes is similar to the structure of understanding I outlined here. The sheaf model of knowledge is what I call a Level 2 understanding, and the level that scientists can't yet achieve for General Relativity and Quantum Mechanics.

Ordinary people go through life having different theories about love, religion, politics, when you kick a table it hurts your foot, and so on, and don’t seem to worry a bit about whether the restriction maps are compatible ...

That's what I call a Level 1 understanding.

I probably could have created a better hierarchy if I had been familiar with the sheaf concept -- sounds like an ideal ontology for an AI to have since it faciliates regeneration of knowledge (Level 3) and consilience (Level 2).