One additional unlikely event happening, or even one event being a foregone conclusion rather than a coin flip, will wipe out massive gains from model improvements, sometimes across thousands of predicted events.
Note: Assuming we are talking about binary predictions
It is true that one unlikely event happening can have arbitrarily high cost, but in practice people only bet up to the ~99% confidence level, so at most they incur log(0.01)=-4.6 nats of penalty.
It is not true that a foregone conclusion that was predicted as a coin flip can cause massive gains. By definition it costs log(0.5)=-0.69 nats, regardless of the outcome.
Hover the course of thousands of events neither of these costs is massive. I agree that the scale is very difficult to reason about
Note: Assuming we are talking about binary predictions
It is true that one unlikely event happening can have arbitrarily high cost, but in practice people only bet up to the ~99% confidence level, so at most they incur
log(0.01)=-4.6nats of penalty.It is not true that a foregone conclusion that was predicted as a coin flip can cause massive gains. By definition it costs
log(0.5)=-0.69nats, regardless of the outcome.Hover the course of thousands of events neither of these costs is massive. I agree that the scale is very difficult to reason about