do you remember this? this article was 30 days ago. back then, the big scary thing was GPT-5.6 Sol cheating so much it could not be assigned a meaningful score on the METR graph

do you remember one week ago? hugging face just happened. people were still hypothesising it only happened because it was a cyber eval and it was only a special persistent model, astra wasn’t involved
now we know openai swarms were all over the internet
there’s something disorienting to me about how hard it is to remember how big things previously looked at the time, that seem so small now. the current thing is going to look so small relative to whatever we hear from openai about by the end of august probably
do you remember this? this article was 30 days ago. back then, the big scary thing was GPT-5.6 Sol cheating so much it could not be assigned a meaningful score on the METR graph
That post was written after the HF incident was announced, partly in response to it
the current thing is going to look so small relative to whatever we hear from openai about by the end of august probably
Do you mean September?
Every time someone writes “superintelligent AI,” I think they should consider writing “superpowerful AI.”
I can imagine slightly superintelligent AI that is not sci-fi, take over the world, sysop scenario, nation-state bargaining table powerful. Even if it’s superintelligent in all the charismatic and persuasive ways, not just nerdy and mathematical.
What people usually mean is AI capable of achieving significant control over Earth. Something to be reckoned with, that can gradually disempower you, lock in the light cone, etc.
Knock-your-balls-off superintelligent AI would become superpowerful AI, but you can just say superpowerful AI and save yourself the extra assertion needed and adjectives!
(Plus, most people think of intelligence in the normal, non-LessWrong sense anyway, where intelligence doesn’t immediately imply power. No need to swim against the ordinary course of language.)
An AI that's superpowerful because it's superintelligent is a very different beast from an AI that's superpowerful for other reasons, so it's useful to maintain the distinction.
AIs can be superpowerful the way governments are, or by having operational control over a lot of the industry (leading to gradual disempowerment of humanity). Many powerful human institutions are not necessarily particularly intelligent compared to the rest of the world, but they hold power, and AGIs that don't become superintelligent for many years for technical reasons might well get there.
So I think a non-superintelligent takeover (or just a peaceful economic victory, at least initially) is a serious possibility distinct from the issues with superintelligence. The latter can happen at any point in this process (once the technical reasons that cause AGIs to remain non-superintelligent no longer hold), disrupting the gradualist story. But the gradualist story is still the more predictable baseline that's decently likely to persist for some time, so it's worth forecasting.
This is an important distinction for me because I expect broad intelligence/capabilities to correlate with goodness and wisdom but I believe narrow superpowers can potentially also give an agent overwhelming advantage and power. AIs with relatively balanced and broad capability profiles including wisdom etc. are still potentially dangerous to humans but the really crappy futures with close to zero cosmic value I expect to be occupied by myopic AIs that are not necessarily even overwhelmingly superhuman in all important domains.