This is a special post for quick takes by JonasH. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
In public discourse, it is often implicitly assumed that LLMs' spikiness is a bug. This overlooks the fact that the human cognitive profile is just as arbitrary as those of LLMs.
It is true that the human cognitive profile (not, of course, the absolute capability levels) is close to optimal for creating economic value in the current system [1]. We shaped the system according to our cognitive profiles, after all.
This does not at all hold true for x-risk! It seems exceedingly unlikely to me that the optimal cognitive profile for inducing x-risk is close to the human one. The AI that kills you might write horrible poetry.
[1] In the short run, that is, where AIs are used within the global economy but don't yet reshape its fundamental processes. E.g., a drop-in remote worker AI, equivalent to a top specialist in every field, is more valuable than an AI that can push theoretical physics into uncharted territory but is a weak collaborator within human-established systems. Over slightly longer time horizons this flips.