This is a special post for quick takes by Linch. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Here's my current four-point argument for AI risk/danger from misaligned AIs.
We are on the path of creating intelligences capable of being better than humans at almost all economically and militarily relevant tasks.
There are strong selection pressures and trends to make these intelligences into goal-seeking minds acting in the real world, rather than disembodied high-IQ pattern-matchers.
Unlike traditional software, we have little ability to know or control what these goal-seeking minds will do, only directional input.
Minds much better than humans at seeking their goals, with goals different enough from our own, may end us all, either as a preventative measure or side effect.
Request for feedback: I'm curious whether there are points that people think I'm critically missing, and/or ways that these arguments would not be convincing to "normal people." I'm trying to write the argument to lay out the simplest possible case.
The article's now out! Comments appreciated
https://linch.substack.com/p/simplest-case-ai-catastrophe
2yams
I think you gamed it a bit by packing a lot into each point (and to someone who doesn't know the arguments already — or even someone who does — the groupings feel pretty unnatural).
Yeah each core point has some number of subpoints. I'm curious if you think I should instead have instantiated each point with just the strongest, or easiest to explain, subpoint (especially when it was branching). Eg the current structure looks like
1: The world’s largest tech companies are building intelligences that will become better than humans at almost all economically and militarily relevant tasks
1.1 (implicit) building intelligences that will become better than humans at almost all economically relevant tasks