Good epistemic calibration of a prediction source is not impressive.
I see people being impressed by calibration charts, for example https://x.com/ESYudkowsky/status/1924529456699641982 , or stronger: https://x.com/NathanpmYoung/status/1725563206561607847
But it's trivial to have a straight-line calibration graph, if it's not straight just fix it for each probability by repeatedly predicting a one-sided coin's outcome as that probability.
If you're a prediction market platform where the probability has to be decided by dumb monkeys, just make sure that the vast majority of questions are of the form "will my p-weighted coin land heads".
---
If a calibration graph isn't straight, that implies epistemic free lunch - if things that you predict at 20% actually happen 30% of the time, just shift those predictions. This is probably the reason why actual prediction markets are calibrated, since incalibration leads to an easy trading strategy. But the presence of calibration is not a very interesting property.
Hmm yea gameability might not be so interesting of a property of metrics as I've expressed.
(though I still feel there is something in there. Fixing your calibration chart after the fact by predicting one-sided coins dice is maybe a lot like taking a foot off the bathroom scale. But, for example, predicting every event as a constant p%, is that even cheating in the calibration game? Though neither of these directly applies to the case of prediction market platforms)