A sorcerer has two ways to manipulate people:
1) Move things around in the world.
2) Directly influence people's minds.
I'm not going to talk about option 2 because it stops people from being perfect reasoners. (If there's a subset of option 2 that still lets people be perfect reasoners, I'd love to hear it - that might be the most interesting part of the puzzle). That leaves option 1.
Here's a simple model of option 1. Nature shuffles a deck of cards randomly, then a sorcerer (if one exists) has a chance to rearrange the cards somehow, then the deck is shown to an observer, who uses it as Bayesian evidence for or against the sorcerer's existence. We will adopt the usual "Nash equilibrium" assumption that the observer knows the sorcerer's strategy in advance. This seems like a fair idealization of "moving things around in the world". What would the different types of sorcerers do?
Note that if both Bright and Dark might exist, the game becomes unpleasant to analyze, because Dark can try to convince the observer that Bright exists, which would mean Dark doesn't exist. To simplify the game, we will let the observer know which type of sorcerer they might be playing against, so they only need to determine if the sorcerer exists.
A (non-unique) best strategy for Bright is to rearrange the cards in perfect order, so the observer can confidently say "either Bright exists or I just saw a very improbable coincidence". A (non-unique) best strategy for Dark is to leave the deck alone, regardless of the observer's prior. Invisible has the same set of best strategies as Dark. I won't spell out the proofs here, anyone sufficiently interested should be able to work them out.
To summarize: if sorcerers can only move things around in the world and cannot influence people's minds directly, then Bright does as much as possible, Invisible and Dark do as little as possible, and the observer only looks at things in the world and doesn't do anything like "updating on the strength of their own beliefs". The latter is only possible if sorcerers can directly influence minds, which stops people from being perfect reasoners and is probably harder to model and analyze.
Overall it seems like your post can generate several interesting math problems, depending on how you look at it. Good work!
some models of 2:
Every person in faeri has a Phsycomagical Intuition. This has a 50% probability of always detecting if there are any sorcerers within a light minute or so, and a 50% probability of giving random results (that don't change or otherwise can be distinguished from having a functioning one). A sorcerer can expend some effort to set the non-functioning ones to whatever it wants.
The sorcerer cannot affect existing minds, but can alter any faerians priors before birth, in a probabilistic manner within normal human variation.
Hmm, the first case seems reducible to "moving things around in the world", and the second sounds like it might be solvable by Robin Hanson's pre-rationality.
How about, if Bob has a sort of "sorcerous experience" which is kind of like an epiphany. I don't want to go off to Zombie-land with this, but let's say it could be caused by his brain doing its mysterious thing, or by a sorcerer. Does that still count as "moving things around in the world"?
Well, it seems possible to set up an equivalent game (with the same probabilities etc) where the sorcerer is affecting a card deck that's shown to you.
Maybe I should have drawn the distinction differently. If the sorcerer can only affect your experiences, that's basically the same as affecting a card deck. But if the sorcerer can affect the way you process these experiences, e.g. force you to not do a Bayesian update where you normally would, or reach into your mind and make you think you had a different prior all along, that's different because it makes you an imperfect reasoner. We know how to answer questions like "what should a perfect reasoner do?" but we don't know much about "what should such-and-such imperfect reasoner do?"
I see what you mean now, I think. I don't have a good model of dealing with a situation where someone can influence the actual updating process either. I was always thinking of a setup where the sorcerer affects something other than this.
By the way, I remember reading a book which had a game-theoretical analysis of games where one side had god-like powers (omniscience, etc), but I don't remember what it was called. Does anyone reading this by any chance know which book I mean?
You might be thinking of Superior Beings by Steven Brams.
(My favourite result of this kind is that if you play Chicken with God, then God loses.)
A (non-unique) best strategy for Dark is to leave the deck alone, regardless of the observer's prior
If I were a Dark, I would try to rearrange the cards so they look random to an unsophisticated observer. No long runs of same color, no obvious patterns in numbers (people are bad random number generators, they think that random string is string without any patterns, not string without big patterns, 17 is the most random number, blah blah blah).
(It's possible that the variation of it can be a good strategy even against more sophisticated agents, because if by a pure chance string of cards has low Kolmogorov complexity, agent is going to take this as evidence for Bright, and I don't want him to believe in Bright)
I think I have a proof that the only Nash equilibrium strategies for Dark playing against a perfect reasoner are those that lead to a uniform distribution over observed decks. K-complexity doesn't seem to come into it. What Dark should do against an imperfect reasoner is a different question, which we can't solve because we don't have a good theory of imperfect reasoning.
Sorcerers could influence people's minds to just change their goals/utility functions without stopping them from being perfect reasoners. This is why people worry so much about AI friendliness, because assuming infinite computational capacity there is still (we assume) no privileged utility function.
Looks like my argument leads to a mildly interesting result. Let's say a Bayesian is playing a game against a Tamperer. The Bayesian is receiving evidence about something, and tries to get more accurate beliefs about that thing according to some proper scoring rule. The Tamperer sits in the middle, tampering with the evidence received by the Bayesian. Then any Nash equilibrium will give the Bayesian an expected score that's at least as high as they would've got by just using their prior and ignoring the evidence. (The expected score is calculated according to the Bayesian's prior.) In other words, you cannot deliberately lead a Bayesian away from the truth.
The proof is kinda trivial: the Bayesian can guarantee a certain score by just using the prior, regardless of what the Tamperer does. Therefore in any Nash equilibrium the Bayesian will get at least that much, and might get more if the Tamperer's opportunities for tampering are somehow limited.
That rather depends on whether the Bayesian (usually known as Bob) knows there is a Tamperer (usually known as Mallory) messing around with his evidence.
If the Bayesian does know, he just distrusts all evidence and doesn't move off his prior. But if he does not know, then the Tamperer just pwns him.
I think your objection is kinda covered by the use of the term "Nash equilibrium" in my comment. And even if the universe decides to create a Tamperer with some probability and leave the evidence untouched otherwise, the result should still hold. The term for that kind of situation is "Bayes-Nash equilibrium", I think.
In this case what's special about Bayesians here?
Bob is playing a zero-sum game against Mallory. All Bob's information is filtered/changed/provided by Mallory and Bob knows it. In this situation Bob cannot trust any of this information and so never changes his response or belief.
I don't see any reason to invoke St.Bayes.
The result also applies if Mallory has limited opportunities to change Bob's information, e.g. a 10% chance of successfully changing it. Or you could have any other complicated setup. In such cases Bob's best strategy involves some updating, and the result says that such updating cannot lower Bob's score on average. (If you're wondering why Bob's strategy in a Nash equilibrium must look like Bayesian updating at all, the reason is given by the definition of a proper scoring rule.) In other words, it's still trivial, but not quite as trivial as you say. Also note that if Mallory's options are limited, her best strategy might become pretty complicated.
If there's a subset of option 2 that still lets people be perfect reasoners, I'd love to hear it - that might be the most interesting part of the puzzle
Make them forget about some piece of evidence they already updated on!
Let's say that evidence A moves B in some direction, and that P(B|A) has already been computed. If you forget A, by encountering A again you would get P(B|A,A) <> P(B|A), while still executing the perfect reasoner's algorithm.
People can counteract that trick and other similar tricks by constantly regenerating their beliefs from their original prior and remembered evidence. Can you make a more watertight model?
I think we can combine your [cousin_it's] suggestion with MrMind's for an Option 2 scenario.
Suppose Bob finds that he has a stored belief in Bright with an apparent memory of having based it on evidence A, but no memory of what evidence A was. That does constitute some small evidence in favor of Bright existing.
But if Bob then goes out in search of evidence about whether Bright exists, and finds some evidence A in favor, he is unable to know whether it's the same evidence as before that he had forgotten, or if it's different evidence. Another way of saying that is that Bob can't tell whether or not A and A are independent. I suppose the ideal reasoner's response would be to assign a probability density distribution over a range from full independence to full dependence and proceed with any belief updates taking that distribution into account.
The distribution should be formed by consideration of how Bob got the evidence. If Bob found his new evidence A in some easily repeatable way, like hearing it from Bright apologists, then Bob would probably think dependence on A is much more likely than independence, and so he would take into account mostly just A and not A. But if Bob got A by some means that he probably wouldn't have had access to in the past, like an experiment requiring brand new technology to perform, then he would probably think independence was more likely, and so he would take into account A and A mostly separately.
But I wonder whether you could manipulate them this way arbitrarily far from rational behavior (at least from the subjective view of an external observer) by ridding them (possibly temporarily) of key facts.
And then there is the question of whether they may notice this as some inferences are more likely to be detected when you already have some other facts. I'd guess that you'd quickly notice if you should suddenly have forgotten that you were repeatedly told that Bright exists.
But I wonder whether you could manipulate them this way arbitrarily far from rational behavior
Surely I can construct such a model. But whether this is generally the case depends too much on the details of the implementation to give a complete answers.
whether they may notice this as some inferences are more likely to be detected when you already have some other facts.
... especially logical inferences: logical deductions of true facts are true, even if you don't know/remember them. But then again, that depends too much on the implementation of the agent to have a general answer, in this case also its computational power would matter.
That's a very interesting analysis. I think you are taking the point of view that sorcerers are rational, or that they are optimizing solely for proving or disproving their existence. That wasn't my assumption. Sorcerers are mysterious, so people can't expect their cooperation in an experiment designed for this purpose. Even under your assumption you can never distinguish between Bright and Dark existing: they could behave identically, to convince you that Bright exists. Dark would sort the deck whenever you query for Bright, for instance.
The way I was thinking about it is that you have other beliefs about sorcerers and your evidence for their existence is primarily established based on other grounds (e.g. see my comment about kittens in another thread). Then Bob and Daisy take into account the fact that Bright and Dark have these additional peculiar preferences for people's belief in them.
The assumption of rationality is usually used to get a tractable game. That said, the assumption is not as restrictive as you seem to say. A rational sorcerer isn't obliged to cooperate with you, and can have other goals as well. For example, in my game we could give Dark a strong desire to move the ace of spades to the top of the deck, and that desire could have a certain weight compared to the desire to stay hidden. In the resulting game, Daisy would still use only the information from the deck, and wouldn't need to do Bayesian updates based on her own state of mind. Does that answer your question?
Like cousin_it, I'm assuming no mind tampering is involved, only evidence tampering
Is it sane for Bob and Daisy to be in such a positive or negative feedback loop? How is this resolved?
I don't think the feedback loops exist. Bob saying "the fact that I believe in bright is evidence of Bright's existence" is double-counting the evidence; deducing "and therefore, Bright exist" doesn't bring in any new information.
It's not that different from saying "I believe it will rain tomorrow, and the fact that I believe that is evidence that it is rain tomorrow, so I'll increase my degree of belief. But wait, that makes the evidence even stronger!".
If Bob and Daisy took the evidence provided by their belief into account already, how does this affect my own evidence updating? Should I take it into account regardless, or not at all, or to a smaller degree?
Just ignore the whole "belief in dark is evidence against dark" thing, Daisy already took that information into account when determining her own belief, you don't want to double count it.
Treat it the same way as you'd treat hearing Bob tell you that in Faery, the sky is Blue, and Daisy telling you that in Faery, the sky is Green.
It's not that different from saying "I believe it will rain tomorrow, and the fact that I believe that is evidence that it is rain tomorrow, so I'll increase my degree of belief. But wait, that makes the evidence even stronger!".
This is completely different. My belief about the rain tomorrow is in no way evidence for actual rain tomorrow, as you point out - it's already factored in. Tomorrow's rain is in no way able to affect my beliefs, whereas a sorcerer can, even without mind tampering. He can, for instance, manufacture evidence so as to mislead me, and if he is sufficiently clever, I'll be misled. But I am also aware that my belief state about sorcerers is not as reliable because of possible tampering.
Here, by me, I mean a person living in Faerie, not "me" as in the original post.
He can, for instance, manufacture evidence so as to mislead me, and if he is sufficiently clever, I'll be misled. But I am also aware that my belief state about sorcerers is not as reliable because of possible tampering.
Seems you can calculate P(evidence | Dark) by taking Dark's tampering into account (basically he'll try to get that value as close as possible to P(evidence | no Dark) ), and update based on that. Your belief may not be reliable in that you may still be wrong, but it still already takes all the information you have (i.e. P(evidence | Dark) ) into account.
Daisy isn't in a loop at all. There's apparently evidence for Dark and that is tempered by the fact that its existance indicates a failing on Dark's part.
For Bob, to make an analogy, imagine Bob is wet. For you, that is evidence that it is raining. It could be argued that being wet is evidence that it's raining for Bob as well. But generally speaking Bob will know why Bob is wet. Given the knowedge of why Bob is wet, the wetness itself is masked off and no longer relevant. If Bob has just had a bath, then being wet no longer constitutes any evidence of rain. If Bob was outside and water fell on him from the sky, it probably did rain, but his being wet no longer constitutes any additional evidence in that case either (well, ok, it has some value still as confirmation of his memory, but it's orders of magnitude less relevant).
Similarly Bob should ask "Why do I believe in Bright?". The answer to that question contains all the relevant evidence for Bright's existance, and given that answer Bob's actual belief no longer constitutes evidence either way. With that answer, there is no longer a loop for Bob either.
One final point, you have to consider the likelihood of belief in case 4. If you would expect some level of belief in sorcerors in Faerie even when there are no sorcerors, then case 4 doesn't fall behind as much as you might think. Once you've got both Bob and Daisy, case 4 doesn't just break even, it's actually way ahead.
It seems like, at this level, thinking of things in terms of "evidence" and "priors" at all is no longer really relevant - Bayesian updating just a way of computing and maintaining "belief caches", which are a highly compressed map of our "evidence" (which is itself a highly compressed map of our phenomenal experience).
The apparent paradox is resolved as long as you note that P(Daisy thinks Dark does exist|Dark does exist) > P(Daisy thinks Dark doesn't exist|Dark does exist).
That is, even if Dark does exist and does want to hide his existence, his less-than-100%-effective attempts to hide will produce non-zero evidence for his existence and make the probability that Daisy will believe in Dark go up by a non-zero amount.
P(Daisy thinks Dark does exist|Dark does exist) > P(Daisy thinks Dark doesn't exist|Dark does exist), then Dark does a very poor job of causing people to believe he doesn't exist- he would do better by not existing!
Adding some structure to this hypothetical: At time t=0, Bob and Daisy have certain priors for their beliefs on sorcery, which they have not adjusted for this argument. Bob's belief was Position 1, with reduced strength, and Daisy's was Position 3, with greater strength.
I'll call your argument A0.
At time t=1, Bob and Daisy are both made aware of A0 and its implications for adjusting their beliefs. They update; Bob's belief in 1 increases, and Daisy's belief in 3 decreases.
More arguments:
A1: If Position 1 is true, then Bright is likely to cause you to increase your belief in him, therefore increasing your belief in Bright is evidence for Position 1.
A1': Corollary: Decreasing your belief in Bright is evidence against Position 1.
A2:If Position 3 is true, then Dark is likely to cause you to decrease your belief in him, therefore decreasing your belief in Dark is evidence for Position 3.
A2': Corollary: Increasing your belief in Dark is evidence against Position 3.
At time t=2, Bob and Daisy are exposed to A1 and A2, and their converses A1' and A2'. If they believe these, they should both increase credence for Positions 1 and 3, following A1 and A2, then increase credence for Position 1 and decrease it for Position 3, following A1 and A2', then follow A1 and A2 again, etc. This might be difficult to resolve, as you mention in your first question.
However, there is a simple reason to reject A1 and A2: Their influence is totally screened off! Bob and Daisy know why they revised their beliefs, and it was because of the valid argument A0. Unless Bright and Dark can affect the apparent validity of logical arguments (in which case your thoughts can't be trusted anyway), A0 is valid independent of which position is true. This action moves them to begin a feedback loop, but stop after a single iteration.
There is a valid reason they might want to continue a weaker loop.
A3: That you have encountered A0 is evidence for the sorcerers whose goals are served by having you be influenced by A0.
A3': That you have encountered A3 is evidence for the sorcerers whose goals are served by having you be influenced by A3.
A3'': That you have encountered A3'' is evidence for the sorcerers whose goals are served by having you be influenced by A3''.
etc.
But this is only true if they didn't reason out A0 or A3 for themselves, and even then A3', A3'', etc. should be considered obvious implications of A3 for a well-reasoned thinker. (In fact, A3 is properly more like "That you have encountered a valid argument is evidence for the sorcerers whose goals are served by having you be influenced by that argument.") So that adds at most one more layer, barring silly Tortoise-and-Achilles arguments.
Given all that, for your second question, you still should take their beliefs into account, but possibly to a slightly lesser degree.
A point I'm confused on: when you, based on A0, update based on their A0-updated belief, are you double-counting A0? If so, you should update to lesser degree. But is that so?
I don't think I completely follow everything you say, but let's take a concrete case. Suppose I believe that Dark is extremely powerful and clever and wishes to convince me he doesn't exist. I think you can conclude from this that if I believe he exists, he can't possibly exist (because he'd find a way to convince me otherwise), so I conclude he can't exist (or at least the probability is very low). Now I've convinced myself he doesn't exist. But maybe that's how he operates! So I have new evidence that he does in fact exist. I think there's some sort of paradox in this situation. You can't say that this evidence is screened off, since I haven't considered the result of my reasoning until I have arrived at it. It seems to me that your belief oscillates between 2 numbers, or else your updates get smaller and you converge to some number in between.
LOL.
The argument goes something like this: "I refuse to prove that I exist,'" says God, "for proof denies faith, and without faith I am nothing."
"But," says Man, "The Babel fish is a dead giveaway, isn't it? It could not have evolved by chance. It proves you exist, and so therefore, by your own arguments, you don't. QED."
"Oh dear," says God, "I hadn't thought of that," and promptly vanishes in a puff of logic.
"Oh, that was easy," says Man, and for an encore goes on to prove that black is white and gets himself killed on the next zebra crossing.”
The scenario you outlined is exactly the same as Daisy's half of the piece ending in A3. The result of your reasoning isn't further evidence, it's screened off by the fact that it's your reasoning, and not the actions of an outside force.
If people believing in Dark is the only possible evidence against Dark, with what certainty should you believe in Dark?
Is it sane for Bob and Daisy to be in such a positive or negative feedback loop? How is this resolved?
It is not sane.
If you use a belief (say A) to change the value of another belief (say B), then depending on how many times you use A, you arrive at different values. That is, if you use A or A,A,A,A as evidence, you get different results.
It would be as if:
P(B|A) <> P(B|A,A,A,A)
But the logic underlying bayesian reasoning is classical, so that A <-> A+A+A+A, and, by Jaynes' requirement IIIc (see page 19 of The logic of science):
P(B|A) = P(B|A,A,A,A)
But in this case we do not have a constant A but an A dependent on someone updating on A.
With notation A[P] for "A believed by P" e.g.
B = Bright exists
B[Bob] = Bob believes Bright exists
B[Bob][Bob] = Bob believes that Bob believes that Bright exists
we can represent the expansion of the updating on belief B = Bright exists as
P(B|X(Bob)) with
X(Bob) where X(Person) = B0(Person) & X[forall P: P!=Person][Person]
that is a belief in a) an aprioi B0 of a person and b) a belief of other persons into this kind of belief. This requires a fixed point to solve. Using the "belief distribution" approximation (A&B)[P] ~= A[P]&B[P] this could be approximated for Bob with
B0(Bob)
B0(Bob) & B0(Alice)[Bob]
B0(Bob) & B0(Alice)[Bob] & B0(Bob)[Alice][Bob]
B0(Bob) & B0(Alice)[Bob] & B0(Bob)[Alice][Bob] & B0(Bob)[Nob][Alice][Bob]
and so on...
And this could be further approximated as B0(Bob) assuming plausibly that other persons have non-stricter priors than yourself (P(X0[Q])<=P(X0[Yourself])). With these two approximations we are back where we began. I didn't work out yet how to get a better bound but it seems plausible that X doesn't diverge but probably converges given suitable B0s.
It is of course correct to state that one can only update once toward
P(B|X) for X=X(P) for all P in Person
but this implies infinitely many expansions in X (the loop I think implied by irrational).
In order to respond... better, in order to understand what you wrote up here I need you to clarify some notation.
What is B[a][b]? b believes that a believes B or a believes that b believes that B?
What is B0(a)? It is the same as B[a]?
What is X0(a)? It is the same as X[a], so that X is a relational variable?
Is X(a) different from X[a]?
Sorry for the confusion, but I couldn't recast your argument in any formal language whatsoever.
Sorry for the confusion, but I couldn't recast your argument in any formal language whatsoever.
Sorry too. That was my risk of inventing a notion on the spot. The original comment started using excessive "believes X" which I basically replaced. I'm not aware of/trained in a notation to compactly write nested probability assertions.
I'm still trying to resolve the expansion and nesting issues I totally glossed over.
What is B[a][b]? b believes that a believes B or a believes that b believes that B?
[] is like a parameterized suffix. It could be bracketed as X[a][b] = (X[a])[b] if that is more clear. I just lent this from programming languages.
Note: There seems to be a theory of beliefs that might applicable but which uses a different notion (looks like X[a] == K_aX): http://en.wikipedia.org/wiki/Epistemic_modal_logic
So what does B[a] mean? B[a] means are are reasoning about the probability assignment P_a(B) of the actor a and we ask for variants of P(P_a(B)=p).
First: I glossed about a lot of required P(...) assuming (in my eagerness to address the issue) that that'd be clear from context. In general instead of writing e.g.
P((A & B)[p]) ~= P(A[p] & B[p])
I just wrote
(A & B)[P] ~= A[P] & B[P]
What is B0(a)? It is the same as B[a]?
No. the 0 was meant to indicate an apriori (which was hidden in the fragment "a) an aprioi B0 of a person"). Instead of writing the needed probability that Bobs prior probability of B is b (needed in the orig post) as
P_{Bob}(B) = b
I just wrote
B0(Bob)
That is informally I represented my belief in the prior p of another actor in some fact F as a fact in itself (calling it F0) instead of representing all beliefs of the represented actor as relative to that (P(F)=p).
This allowed me to simplify the never written out long form of P(B|X(Bob)). On this I'm still working.
What is X0(a)? It is the same as X[a], so that X is a relational variable?
Yes. For all prior belief expressions X0 it is plausible to approximate other persons prior probability to be less strict than your own.
Is X(a) different from X[a]?
Yes. X(a) is the X of person a. This is mostly releant for the priors.
What I now see after trying to clean up all the issues glossed over is that this possibly doesn't make sense. At least not in this incomplete form. Please stay tuned.
Please stay tuned.
I will!
The main problem (not in your post, in the general discussion) seems to me that there's no way to talk about probabilities and beliefs clearly and dependently, since after all a belief is the assignment of a probability, but they cannot be directly targeted in the base logic.
Puh. And I feared that I came across as writing totally unintellegible stuff. I promise that I will put some more effort into the notation.
I am not certain that it's the same A. If I say to you, here's a book that proves that P=NP. You go and read it, and it's full of Math, and you can't fully process it. Later, you come back and read it again, this time you actually able to fully comprehend it. Even later you come back again, and not only comprehend it, but are able to prove some new facts, using no external sources, just your mind. Those are not all the same "A". So, you may have some evidence for/against a sorcerer, but are not able to accurately estimate the probability. After some reflection, you derive new facts, and then update again. Upon further reflection, you derive more facts, and update. Why should this process stop?
I think we are talking about different things.
I proved only that Bob cannot update his belief in Bright on the sole evidence "Bob believes in Bright". This is a perfectly defined cognitive state, totally accessible to Bob, and unique. Therefore Bob cannot update on it.
On the other hand, if from a belief Bob gathers new evidence, then this is clearly another cognitive state, well different from the previous, and so there's no trouble in assigning different probabilities (provided that "Bob believes in Bright" doesn't mean that he assigns to Bright probability 1).
Please give numeric probabilities for Bob and Daisy's beliefs, and examples of (private) evidence that would justify them. I'm not sure you can construct a scenario where they're rational, truthful, have similar priors, and disagree by all that much about sorcery.
I am not assuming they are Bayesians necessarily, but I think it's fine to take this case too. Let's suppose that Bob finds that whenever he calls upon Bright for help (in his head, so nobody can observe this), he gets unexpectedly high success rate in whatever he tries. Let's further suppose that it's believed that Dark hates kittens (and it's more important for him than trying to hide his existence), and Daisy is Faerie's chief veterinarian and is aware of a number of mysterious deaths of kittens that she can't rationally explain. She is afraid to discuss this with anyone, so it's private. For numeric probabilities you can take, say, 0.7, for each.
I think your degree of belief in their rationality (and their trustworthiness in terms of not trying to mislead you, and their sanity in terms of having priors at least mildly compatible with yours) should have a very large effect on how much you update based on the evidence that they claim a belief.
The fact that they know of each other and still have wildly divergent beliefs indicates that they don't trust in each other's reasoning skills. Why would you give them much more weight than they gave each other?
For this experiment, I don't want to get involved in the social aspect of this. Suppose they aren't aware of each other, or it's very impolite to talk about sorcerers, or whatever. I am curious about their individual minds, and about an outside observer that can observe both (i.e. me).
With respect to question 1, Aumann's Agreement Theorem would require that if they are acting rationally as you stated and with common knowledge, they would have to agree. That being the case, according to the formalism, question 2 is ill-posed.
Your proposed state of affairs could hold if they lack common knowledge (including lack of common knowledge of internal utility functions despite common knowledge of otherwise external facts and including differing prior probabilities). To resolve question 2 in that case you would have to assign probabilities to the various forms that the shared and unshared knowledge could take, to determine which state of affairs most probably prevails. For example, you may use your best estimation and determine that the state of affairs that prevails in Faerie is similar to the state of affairs that prevails in your own world/country/whatever, in which case you should weight the evidence provided by each person similar to how you would rate it if you were surveying your fellow countrymen. This is all fairly abstract because approaching such a thing formally is currently well outside our capabilities.
Thanks. I am of course assuming they lack common knowledge. I understand what you are saying, but I am interested in a qualitative answer (for #2): does the fact they have updated their knowledge according to this meta-reasoning process affect my own update of the evidence, or not?
I read this as the two consultants being basically rational, but not perfect Bayesians, and not sharing common priors.
This article is going to be in the form of a story, since I want to lay out all the premises in a clear way. There's a related question about religious belief.
Let's suppose that there's a country called Faerie. I have a book about this country which describes all people living there as rational individuals (in a traditional sense). Furthermore, it states that some people in Faerie believe that there may be some individuals there known as sorcerers. No one has ever seen one, but they may or may not interfere in people's lives in subtle ways. Sorcerers are believed to be such that there can't be more than one of them around and they can't act outside of Faerie. There are 4 common belief systems present in Faerie:
This is completely exhaustive, because everyone believes there can be at most one sorcerer. Of course, some individuals within each group have different ideas about what their sorcerer is like, but within each group they all absolutely agree with their dogma as stated above.
Since I don't believe in sorcery, a priori I assign very high probability for case 4, and very low (and equal) probability for the other 3.
I can't visit Faerie, but I am permitted to do a scientific phone poll. I call some random person, named Bob. It turns out he believes in Bright. Since P(Bob believes in Bright | case 1 is true) is higher than the unconditional probability, I believe I should adjust the probability of case 1 up, by Bayes rule. Does everyone agree? Likewise, the probability of case 3 should go up, since disbelief in Dark is evidence for existence of Dark in exactly the same way, although perhaps to a smaller degree. I also think the case 2 and case 4 have to lose some probability, since it adds up to 1. If I further call a second person, Daisy, who turns out to believe in Dark, I should adjust all probabilities in the opposite direction. I am not asking either of them about the actual evidence they have, just what they believe.
I think this is straightforward so far. Here's the confusing part. It turns out that both Bob and Daisy are themselves aware of this argument. So, Bob says, one of the reasons he believes in Bright, is because that's positive evidence for Bright's existence. And Daisy believes in Dark despite that being evidence against his existence (presumably because there's some other evidence that's overwhelming).
Here are my questions:
I am looking forward to your thoughts.