Can you give some more intuitions as to why allowing finite support is among your criteria?

I can imagine a definition which, lacking this criterion, is still useful, and requiring to have infinite support might be a useful reminder that 0 and 1 are not probabilit(y densities). Further, whereas requiring infinite support might risk analyzing absurd outcomes, it may also allow us to consider, and thus reach maximally great futures.

Reality is inherently bounded - I can confidently assert that there is no possible risk today that would endanger a trillion lives, because I am confident the number of people on the planet is well below that.

Consider that the number of animal lives is probably greater than one trillion, and you didn't specify *human* lives. You could also consider future lives, or abstruse moral realism theories. Your definition of personhood (moral personhood?) could change. Having finite support considered harmful (?).

Here is another point by @jacobjacob, which I'm copying here in order for it not to be lost in the mists of time:

Though just realised this has some problems if you expected predictors to be better than the evaluators: e.g. they’re like “one the event happens everjacobyone will see I was right, but up until then no one will believe me, so I’ll just lose points by predicting against the evaluators” (edited)

Maybe in that case you could eventually also score the evaluators based on the final outcome… or kind of re-compensate people who were wronged the first time…

Another point in favor of such a set-up would be that aspiring superforecasters get much, much more information when they see ~[the prediction of a superforecaster would have made having their information]; a point vs a distribution. I'd expect that this means that market participants would get better, faster.

This is somewhat solved if you have a forecaster that you trust that can make a prediction based on Sophia's seeming ability and honesty. The naive thing would be for that forecaster to predict their own distribution of the log-loss of Sophia, but there's perhaps a simpler solution. If Sophia's provided loss distribution is correct, that would mean that she's calibrated in this dimension (basically, this is very similar to general forecast calibration). The trusted forecaster could forecast the adjustment made to her term, instead of forecasting the same distribution. Generally this would be in the direction of adding expected loss, as Sophia probably had more of an incentive to be overconfident ( which would result in a low expected score from her) than underconfident. This could perhaps make sense as a percentage modifier (-30% points), a mean modifier (-3 to -8 points), or something else. Is it actually true that forecasters would find it easier to forecast the adjustment?> This is somewhat solved if you have a forecaster that you trust that can make a prediction based on Sophia's seeming ability and honesty. The naive thing would be for that forecaster to predict their own distribution of the log-loss of Sophia, but there's perhaps a simpler solution. If Sophia's provided loss distribution is correct, that would mean that she's calibrated in this dimension (basically, this is very similar to general forecast calibration). The trusted forecaster could forecast the adjustment made to her term, instead of forecasting the same distribution. Generally this would be in the direction of adding expected loss, as Sophia probably had more of an incentive to be overconfident ( which would result in a low expected score from her) than underconfident. This could perhaps make sense as a percentage modifier (-30% points), a mean modifier (-3 to -8 points), or something else.

Is it actually true that forecasters would find it easier to forecast the adjustment?

We could also say that if we took a probability distribution of the chances of every possible set of findings being true, the differential entropy of that distribution would be 0, as smart forecasters would recognize that inputs_i s correct with ~100% probability.

In that paragraph, did you mean to say "findings_i is correct"?

***

Neat idea. I'm also not sure whether the idea is valuable because it could be implementable, or from "this is interesting because it gets us better models".

In the first case, I'm not sure whether the correlation is strong enough to change any decisions. That is, I'm having trouble thinking of decisions for which I need to know the generalizability of something, and my best shot is measuring its predictability.

For example, in small foretold/metaculus communities, I'd imagine that miscellaneous factors like "is this question interesting enough to the top 10% of forecasters" will just make the path predictability -> differential entropy -> generalizability difficult to detect.

No state will secede from the US. 95%

This seems underconfident?

I have different intuitions for both:

No one will have won a Nobel Prize in Physics for their work on string theory. 80%

and

No US President will utter the words "Existential risk" in public during their term as president. 65%

But this is such that I'd expect that looking into either for a couple of hours would change my mind. For the second one, the Google ngram page for existential risk is interesting, but it sadly only reaches up to the year 2008.

Is anyone accepting bets on their predictions?

My method was reading the Wikipedia page and answering the following questions:

1. Was the movement succesful as a community?

• 0: nope
• 1: to some extent / ambiguous.
• 2: clearly yes.

2. Did the movement produce the change in the world which it said it wanted?

• 0: nope
• 1: not totally a failure / had some minor victories / ambiguous.
• 2: clearly yes.

3. Was it succesful at changing laws? | Was that its intent?

4. Is it fringe (0), minority (1) or mainstream (2)?

5. Bias: how sympathetic am I to this movement?

• 0: I am unsympathetic.
• 1: I am not unsympathetic
• 2: I like them a lot.

I feel that for the amount of effort I'm spending on this, I'm going to have to rely on my gut feeling at some point, and that the pareto principle thing to do is to have well defined questions.

In case I or someone else wants to develop this further, a way to improve on question 2 would be:

• a) Identify the three most important objectives the movement claims to have.
• b) For each, to what extent has it been achieved?

I excluded "Salt March" because I saw it as doublecounting "Nonviolence", and excluded "Reform movements in the United States" because it was too broad a category. I kept "Student Movements", though.

Anyways, you can find a .csv table with the results here or a Google Drive link here. I might play around with the results further, but for the moment:

Socially, the average movement does pretty well, with an average of 1.3/2, distributed as: 16% are 0s, 36% are 1s and 48% are 2s . With regards to effectiveness, the average is 0.72/2, distributed as: 44% are 0s, 40% are 1s and 16% are 2s.

I'm on this.

Is that talk worth listening to in full?