In the course of studying how subjective probabilities can be defined, I read A Definition of Subjective Probability (1963) by Anscombe and Aumann. My notes may be of interest to the Less Wrong community, and have pasted them below.

The authors define two types of lotteries:

• A "roulette lottery" which is a game of chance with "physical" probabilities attached to outcomes, where each outcome is associated with a prize. The authors are vague about what they mean by "physical" probabilities, but they seem to mean probabilities that it's possible to generate via frequentist inference.
• A "horse lottery," which is a game of chance where physical probabilities are unavailable.

The paper's goal is to give a definition of subjective probabilities attached to outcomes in a horse lottery.

Intuitively, the idea seems to be as follows. Suppose that you have an event E, that you desire to happen, and a choice between the following options:

1. A horse lottery occurs, and event E occurs if and only if the outcome of the horse lottery is O.
2. A roulette lottery occurs, and event E occurs if and only if the outcome is O', where O' has probability q.

Consider the set T of values of q such that you'd prefer #2 over #1. Then your subjective probability p of the horse lottery having outcome O is defined to be the greatest lower bound of T.

The authors begin by assuming that one has a preference ordering over the prizes awarded in lotteries, with the best prize strictly favored over the worst prize. Here the prizes include tickets to other lotteries.

The authors convert this preference ordering to a utility function u where the best prize is assigned utility 1 and the worst prize is assigned utility 0. The authors assume that the function u has the property that u of a roulette lottery is the expected utility (sum of utilities of the outcomes weighted by the probabilities of the outcomes). The authors also convert a preference ordering over horse lotteries to a utility function u^*. We know that u^* is 1 for the lottery that gives the best prize with probability 1 and u^* is 0 for the horse lottery that gives the worst prize with probability 1. At this point it's not meaningful to say that that u^* of a horse lottery is the expected utility, because the probabilities associated with outcomes of the horse lottery have not been defined.

The authors then consider the set of horse lotteries h with the same a priori possible outcomes O_i and the same actual outcome, where the prizes are tickets for roulette lotteries R_i.The main theorem of the paper is that there exist nonnegative numbers p_i summing to 1 such that u^* of the horse lottery is given by the the sum of p_iu(R_i), independently of the roulette lotteries R_i. Then p_i is taken to be the definition of the subjective probability that one assigns to outcome O_i. 

The candidate for p_i is u^*(k_i), where k_i is the horse lottery where outcome O_i is associated with the best prize and the other outcomes are associated with the worst prize. 

To prove the theorem, let c be the sum of the numbers u(R_i). For the sake of clarity, suppose that c < 1 (where the idea of the proof is most evident). We have

• Claim 1: u^*(h) = cu^*(h'), where h' is the horse race associating outcome O_i to a roulette lottery S_i with u(S_i) = u(R_i)/c. (These utilities are admissible since u(R_i)/c is no larger than 1, from the definition of c).
• Claim 2: u^*(h') = u(S), where S is the roulette lottery that with probability u(S_i) gives a ticket to the horse lottery k_i. (Here S is well defined since the u(S_i)'s sum to 1, from the definition of c.) This is the most subtle step, and the core of the theorem. Unpackaging it in words: a horse lottery where each outcome O_i is associated with a utility u(S_i) is equivalent to (for each i) there being a u(S_i) chance of getting a horse lottery where outcome O_i is associated with utility 1 . 
• Claim 3: We can write u(S) = Sum of u(S_i) x u^*(k_i) 

Combining these gives the desired theorem.