This is a special post for quick takes by robo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
You can smell a chess bot by how quickly they change plans[1]. Human players act like they have a couple of attack strategies in mind and stick with them. Chess bots change tacks constantly, one move looking like they're moving towards this goal and next move switching to something totally different.
I'm guessing this it how it would be with real-world takeover. Human need a simple grand strategy they can coordinate around ("An amphibious invasion of northern France"). AIs, even very weak ones[2], have far more working memory and excel at breadth. Their plan could be 1000 plans at once.
1a3orn made what I thought was a good reply in a DM. My digestion/riff would be "You're implicitly assuming that planning will be especially cheap for an LLM agent compared to humans. But even medium term planning / resilience to plans going awry is a particular weakness for LLM agents. The first, weakest agents that could pose a risk to humanity might be differentially bad at planning, find it differentially expensive to make good plans, do differentially less planning."
Yeah, when I think about "AI takeover", I am imagining a very strong a smart AI, the one for which the success is more plausible. But before we get strong AIs, we will have weak AIs, so the first takeover attempts will be made by them. Maybe even the first successful takeover attempt.
A very strong and smart AI would however do thousand different things at the same time. Unlike the chess bot, which only plays on one chessboard, the AI could e.g. have separate plans to taking over each specific country. Many plans to take over one specific country would not ... (read more)