Thanks for the reply, 3 seams very automatable, record all text before the image, if that's 4 minuts then then put the image in after 4 min. But i totally get that stuff is more complicated than it initially seems, keep up the good work!

I agree tails are important, but for callibration few of your predictions should land in the tail, so imo you should focus on getting the trunk of the distribution right first, and the later learn to do overdispersed predictions, there is no closed form solution to callibration for a t distribution, but there is for a normal, so for pedagogical reasons I am biting the bullet and asuming the normal is correct :), part 10 in this series 3 years in the future may be some black magic of the posterior of your t predictions using HMC to approximate the 2d posterior of sigma and nu ;), and then you can complain "but what about skewed distributios" :P

The text to speech is phenomenal!, Only math and tables suck

Suggestions for future iterations:

1. allow filtering of the RSS by hacking the url, for example ?sources=LW,EA&quality=curated would give the curated posts from only EA and LW ignoring alignment forum
2. somehow allow us to convert our "read later" to an RSS :)
3. when there are figures in the post, then put them in the podcast "image" like SolenoidEntity does for the Astral Codex 10 podcast
4. put the text in the show notes so we can pause and look at tables

It would be nice if you wrote a short paragraph for each link, "requires download", "questions are from 2011", or you sorted the list somehow :)

Yes, You can change future $\mu$ by being smarter and future $\sigma$ by being better calibrated, my rule assumes you don't get smarter and therefore have to adjust only future $\sigma$.

If you actually get better at prediction you could argue you would need to update $\sigma$ less than the RMSE estimate suggests :)

I agree with both points

If you are new to continuous predictions then you should focus on the 50% Interval as it gives you most information about your calibration, If you are skilled and use for example a t-distribution then you have $\sigma$ for the trunk and $\nu$ for the tail, even then few predictions should land in the tails, so most data should provide more information about how to adjust $\sigma$, than how to adjust $\nu$

Hot take: I think the focus 95% is an artifact of us focusing on p<0.05 in frequentest statistics.

Our ability to talk past each other is impressive :)

would have been an easier way to illustrate your point). I think this is actually the assumption you're making. [Which is a horrible assumption, because if it were true, you would already be perfectly calibrated].

Yes this is almost the assumption I am making, the general point of this post is to assume that all your predictions follow a Normal distribution, with $\mu$ as "guessed" and with a $\sigma$ that is different from what you guessed, and then use $X^2$ to get a point estimate for the counterfactual $\sigma$ you should have used. And as you point out if (counterfactual) $\sigma =1$ then the point estimate suggests you are well calibrated.

In the post counter factual $\sigma$ is ${\hat{\sigma }}_z$

Thanks!, I am planing on writing a few more in this vein, currently I have some rough drafts of:

• 30% Done, How to callibrate normal predictions

• defence of my calibration scheme, and an explanation of how metaculus does.

• 10% Done, How to make overdispersed predictions

• like this one for the logistic and t distribution.

• 70% Done, How to calibrate binary predictions

• like this one + but gives a posterior over the callibration by doing an logistic regression with your predictions as "x" and outcome as "y"

I can't promise they will be as good as this one, but if they are not terrible then I would like them to be turned into a sequence :), how do I do this?

Yes you are right, but under the assumption the errors are normal distributed, then I am right:

If:

$$p\sim Bern\left(0.78\right)\sigma =p\times N(0,1.1)+(p-1)N(0,0.5)$$

Then $E\left[{\sigma }^2\right]\approx 0.37$ Which is much less than 1.

proof:

import scipy as sp

x1 = sp.stats.norm(0, 0.5).rvs(22 * 10000)
x2 = sp.stats.norm(0, 1.1).rvs(78 * 10000)
x12 = pd.Series(np.array(x1.tolist() + x2.tolist()))
print((x12 ** 2).median())

I am making the simple observation that the median error is less than one because the mean squares error is one.