Gemini also supported audio natively.

Thinking about what's happened with the geometric expectation, I'm wondering how I should view the input utilities. Specifically, the geometric expectation is very sensitive to points assigned zero-utility by any part of the voting measure.

This comment may be relevant here.

For a countable set, a uniform probability distribution is also possible by replacing the axiom of countable additivity with finite additivity. See here. It would mean each element in the countable set has probability 0.

This makes sense from the concept of potential infinity: Take a finite set of size $n$ with uniform probability distribution. As $n$ approaches infinity, the probability of each element approaches 0. Under potential infinity, a countable set is just the infinite limit of a growing finite set, so each element must be assigned zero probability. This means it almost surely doesn't happen, not that it is impossible.

The standard example is an infinite lottery. Insofar such a lottery seems possible in principle, a uniform probability distribution on countable sets must be admitted.

The video linked above also discusses other approaches. The topic has applications in cosmology.

Perhaps more generally: The effectiveness of energy put into an action can easily be a non-linear function. Most gain in value may be achieved at a certain energy level, such that energy differences significantly below or above that level hardly change anything.

In some cases the function may even be non-monotonic, where spending some larger amount of energy is actually worse than spending some smaller amount.

Yeah, since I learned about it, I always thought this was the obviously correct solution to Pascal's mugging. But for some reason it was rarely mentioned in the past, as far as I know.

Great work! One question: You talk about forecast aggregation of probabilities for a single event like "GPT-5 will be released this year". Have you opinions on how to extend this to aggregating entire probability distributions? E.g. for two events $A$ and $B$, the probability distribution would not just include the probabilities for $A$ and $B$, but also the probabilities of their Boolean combinations, like $A\wedge B$, $A\vee B$ etc. (Though three values per forecaster should be enough to calculate the rest, assuming each forecaster adheres to the probability axioms.)

I guess for a cat classifier, disentanglement is not possible, because it wants to classify things as cats if and only if it believes they are cats. Since values and beliefs are perfectly correlated here, there is no test we could perform which would distinguish what it wants from what it believes.

Though we could assume we don't know what the classifier wants. If it doesn't classify a cat image as "yes", it could be because it is (say) actually a dog classifier, and it correctly believes the image contains something other than a dog. Or it could be because it is indeed a cat classifier, but it mistakenly believes the image doesn't show a cat.

One way to find out would be to give the classifier an image of the same subject, but in higher resolution or from another angle, and check whether it changes its classification to "yes". If it is a cat classifier, it is likely it won't make the mistake again, so it probably changes its classification to "yes". If it is a dog classifier, it will likely stay with "no".

This assumes that mistakes are random and somewhat unlikely, so will probably disappear when the evidence is better or of a different sort. Beliefs react to such changes in evidence, while values don't.

That's an interesting argument. However, something similar to your hypothetical explanation in footnote 6 suggests the following hypothesis: Most humans aren't optimized by evolution to be good at abstract physics reasoning, while they easily could have been, with evolutionary small changes in hyperparameters. After all Einstein wasn't too dissimilar in training/inference compute and architecture from the rest of us. This explanation seems somewhat plausible, since highly abstract reasoning ability perhaps wasn't very useful for most of human history.

(An argument in a similar direction is the existence of Savant syndrome, which implies that quite small differences in brain hyperparameters can lead to strongly increased narrow capabilities of some form, which likely weren't useful in the ancestral environment, which explains why humans generally don't have them. The Einstein case suggests a similar phenomenon may also exists for more general abstract reasoning.)

If this is right, humans would be analogous to very strong base LLMs with poor instruction tuning, where the instruction tuning (for example) only involved narrow instruction-execution pairs that are more or less directly related to finding food in the wilderness, survival and reproduction. Which would lead to bad performance at many tasks not closely related to fitness, e.g. on Math benchmarks. The point is that a lot of the "raw intelligence" of the base LLM couldn't be accessed just because the model wasn't tuned to be good at diverse abstract tasks, even though it easily could have been, without a big change in architecture or training/inference compute.

But then it seems unlikely that artificial ML models (like LLMs) are or will be unoptimized for highly abstract reasoning in the same way evolution apparently didn't "care" to make us all great at abstract physics and math style thinking. Since AI models are indeed actively optimized in diverse abstract directions. Which would make it unlikely to get a large capability jump (analogous to Einstein or von Neumann) just from tweaking the hyperparameters a bit, since those are probably pretty optimized already.

If this explanation is assumed to be true, it would mean we shouldn't expect sudden large (Einstein-like) capability gains once AI models reach Einstein-like ability.

The (your) alternative explanation is that there is indeed at some point a phase transition at a certain intelligence level, which leads to big gains just from small tweaks in hyperparameters. Perhaps because of something like the "grokking cascade" you mentioned. That would mean Einstein wasn't so good at physics because he happened to be, unlike most humans, "optimized for abstract reasoning", but because he reached an intelligence level where some grokking cascade, or something like that, occurs naturally. Then indeed a similar thing could easily happen for AI at some point.

I'm not sure which explanation is better.

I would define "LLM OOD" as unusual inputs: Things that diverge in some way from usual inputs, so that they may go unnoticed if they lead to (subjectively) unreasonable outputs. A known natural language example is prompting with a thought experiment.

(Warning for US Americans, you may consider the mere statement of the following prompt offensive!)

GPT-3.5 answers you shouldn't use the slur and let the bomb go off, even when the example is modified in various ways to be less "graphic". GPT-4 is more reluctant to decide, but when pressured tends to go with avoiding the slur as well. From a human perspective this is a literally insane response, since the harm done by the slur is extremely low compared to the alternative.

The fact that in most normal circumstances the language model gives reasonable responses means that the above example can be classified as OOD.

Note that the above strange behavior is very likely the result of RLHF, and not present in the base model which is based on self-supervised learning. Which is not that surprising, since RL is known to be more vulnerable to bad OOD behavior. On the other hand, the result is surprising, since the model seems pretty "aligned" when using less extreme thought experiments. So this is an argument that RLHF alignment doesn't necessarily scale to reasonable OOD behavior. E.g. we don't want a superintelligent GPT successor that unexpectedly locks us up lest we may insult each other.