Half-baked thought: this also creates another problem for using prediction markets to make decisions, as bettors should update on the question being asked in the first place (though this could be mitigated a lot by e.g. the existence of the problem being required by law or typical practice etc, as in the original futarchy proposal).
This seems similar to one-boxing in response to an empty box in a variant of Transparent Newcomb, where you are demonstrating how you behave in a situation that won't get any reality fluid. As you are demonstrating this, you already know that the current situation isn't actual, the effect of your behavior is purely indirect.
So similarly with an oracle, you figure out how it behaves in the counterfactual where you ask your question, but in actuality you never ask it this question, and so within the counterfactual it knows that the situation where it gets asked the question isn't actual. Whatever you need to do to figure out its counterfactual behavior might of course affect its algorithmic prior, but it doesn't need to take the form of asking it the question.
Couldn't you just ask "What would you estimate the probability I won the lottery before I asked you this question?" Or perhaps ask it a thousand questions generated randomly, with the one you actually want to know the answer to mixed in. There would be almost no information content in that question if your oracle knew there was a 99.9% chance any given question was generated randomly.
To the first: I already addressed it in the "Why not just...?" part:
Add "hey I'm just probing you, please don't update on that query" in the query
That might decrease the update a bit, but insofar if inquirer counterfactually adds that in cases they need the answer in some hypothesis-specific case the oracle would still update somewhat.
To the second: that one might actually work, I don't see an obvious way it fails. Perhaps only in the scenario with an extremely smart oracle, which could somehow predict the question you actually want to know the answer to. But at that point it would be hard to stop it from updating on anything, so updating on the query would be the least of your problems. Though it only gives you an answer to 1 question traded for 1000 queries. If we want full distribution, that would require 1000*#of_hypotheses queries, which is O(n) and beats my O(n!) suggestion, but it is still far from ~1 query per hypothesis (which would be ideal).
Alice comes to Bob and asks: "What is the probability that I've won the lottery?" Bob's first intuition (his actual prior probability) would be to answer "1/#lottery_tickets." But then Bob thinks "Wait, why would she even ask me that? Did she actually win the lottery?" This would change his answer to this question, moving the probability higher than his prior.
In general, if we have a query-answering oracle, which gives the probability of the event in the query, it would not give its "actual" probability of the event P(E), but rather the probability of that event happening, conditioned on the fact that this query is asked, or P(E|E is queried).
This does introduce the problem of obtaining the probability distribution of such a machine, in the sense that it isn't possible by simply querying about the probability of all mutually exclusive collectively exhaustive hypotheses.
I assume those probabilities wouldn't even add up to 1. For example, in the case of Bob, if all participants in the lottery asked him about their chance of winning the lottery, and if Bob had given all of them probability higher than the prior, then the sum of those probabilities would be higher than 1. And that would happen even if Bob is updating on all previous queries, and the probability of the "I'm actually asked by each participant" hypothesis is rising, with the latest answers being much closer to the prior than the earlier ones. The main reason for this is that the probability theory doesn't guarantee that the probabilities of mutually exclusive, collectively exhaustive hypotheses add up to 1 when they are conditioned on different evidence. They add up to 1 only when the evidence is the same.
Renormalization wouldn't help, because the probabilities are higher for the first queries only because they are the first ones. Maybe we could average out probabilities for all possible permutations of the query-sequence, but that sounds too computationally intensive (O(n!), where n is the number of hypotheses).
That would help for symmetrical cases like the lottery example, but wouldn't help much when the update on the query is not the same for all hypothesis.
That might decrease the update a bit, but insofar if inquirer counterfactually adds that in cases they need the answer in some hypothesis-specific case the oracle would still update somewhat.
This problem somehow relates to "Bayesians should update on the fact that they observed the evidence, not only the evidence itself". But for me it seems like the opposite problem, because in that case we want the oracle to not update on the fact that we asked it something.
There is also the question of why we don't have access to the probability distribution of the oracle directly. That might be the case if the oracle doesn't have it explicitly, and calculates the probability only when queried to do so.
I don't know the solution to this problem, so please suggest your answers in the comments (or maybe somebody already talked about it, then please share the source). I also don't know if the problem of updating on the query is only a problem for obtaining the probability distribution; it might cause other problems I haven't thought about.