This is a special post for quick takes by Linch. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Here's my rough order (descending) of the most scary near-future or medium-future AI capabilities, in terms of something like catastrophic danger x potential of happening soon x expected realistic magnitude.
AI research -- clearly the scariest thing because among others it can unlock everything else on the list, massively shorten the feedback loop before we build defenses, etc.
(large gap)
Autonomy/agency/long-term planning -- seems like a likely but-for cause of most takeover stories. Most takeover stories I can think about entail superhuman long-term planning, or at minimum peak human planning + coordination across instances. Takeovers without that always seem either flimsy or entail seemingly a "magical" technological edge in one or multiple other domains.
human persuasion/manipulation/political strategy -- I've written about this elsewhere but if I'm imagining a takeover without ASI the easiest story is 3+4.
(known unknowns) -- things not explicitly on this list (or most other people's top 10 lists) but people are tracking at least a little. Essentially this is a bet on "the field" over the listed categories.
robotics (including military drones) -- Besides being directly scary, among other things, robotics lets the AIs service the data centers after a takeover. Especially relevant if #4 isn't there.
(unknown unknowns) -- capabilities basically out of left field.
bio uplift -- a bio catastrophe takeover without #4 or #7 seems flimsy to me. Though the story to non-agentic catastrophe is in some sense the "cleanest" -- I just don't think the total probability mass for AI x bio uplift extremal outcomes is very high compared to other disaster stories. (tbc by bio uplift I mostly mean lifting terrorists and others up to bio expert levels rather than superhuman levels, though my current guess is that near-future superhuman bio capabilities won't change the ordering much)
cybersecurity -- I just don't see it. Any cyber-only takeover seems cartoonish to me, and I can't make even superhuman planning + superhuman offensive cybersecurity really work. And the "pure destruction" story (eg turning off critical infrastructure, or somehow hacking the nukes) is weaker than for bio. Curious if other people disagree, and if so, why?
I know it's a bit of a weird thing to say in this of all weeks, but I do think there's a realistic upper bound for the badness of cyber.
(Self-exfiltration does seem kinda bad but mostly I don't expect it to be world-ending by itself, or even as a but-for factor in conjunction with other things)
I'm curious whether people disagree with the order, and if so, why. I'm also curious if there are specific key capabilities I missed!
To the extent this ordering is roughly correct, it does suggest ex-risk-focused people should probably have different orderings and prioritizations of evaluations and defenses than they currently work on.
(I'm also currently assuming that analysis at this high level of abstraction is not infohazardous but feel free to privately lmk if I'm wrong)
I agree both are possible and bad! I think the self-exfiltration case for badness seems somewhat bounded. I've thought less about the relative value-add of rogue internal deployments from marginal gain in cybersec skills.
Personally, I'd put bio capabilities higher up. I would agree with your ranking if "an AGI/ASI decides to kill people and uses bio to do so" were the only mechanism of x-risk to worry about. However, I think much of bio x-risk comes from human-misuse scenarios (i.e. providing uplift to humans wishing to create bioweapons).