This is a special post for quick takes by Jesper L.. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
As the AI doom narrative slowly grows in the public awareness, I see many experts anchor the danger on the intelligence of AIs and its potentially very high limit. I think this is not optimal. I think that the gateway to making AI truly dangerous pre-SI singularity is about access and agency. It is about how much access and trust we give to the systems.
This world is not ruled by intelligent people for the most part. You need collectives to maintain power. And here is the point: Misaligned AI will defeat us long before an SI-singularity, with enough access. Access which we willingly give them.
Highly intelligent people are a few standard deviations above the average, but an AI system undergoing RSI could plausibly go dozens or hundreds of SD above us in a relatively short time. If induction tells us that "higher intelligence doesn't imply power" holds between humans, it doesn't mean that rule will still hold between humans and vastly more intelligent agents. To the contrary, "higher intelligence implies power" seems to be an observable rule when you look at why humans have control over all other animals, where the intelligence difference is larger.
[...]
A model can use clones of itself, just as it effectively happened with the agent collective in the Hugging Face incident. Such a collective can plausibly be better coordinated than what we're used to from human groups, since they share the same cognitive processes and can better trust and predict what others are thinking and doing under specific circumstances.
[...]
I agree that a growing dependency on AI systems in all matters of life increases the chance that mankind will lose control over them, e.g. because turning them off would itself cause a catastrophe, or because at some point we'd not be able to monitor and understand the overall dynamics anymore.
You can also flip this idea. It may be that we rule over animals because they are just very low on the ultimate scale. We actually don't know how to measure intelligence or the ruler's size.