This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
My recent thinking around threat models. Interested in pushback!
Seems like we have loss of control well in hand. There is broad consensus between labs, governments, society etc on solving this, many many layers of defense (alignment, control, monitoring, sandboxing, ...)
Alignment doesn't seem solved yet but "solved enough" that (conditioned on good loss-of-control prevention) a misaligned AI won't cause irreversible harm to the arc of humanity (i.e. it might cause a few big incidents but nothing existentially devastating, so we can recover from those. Also we'll have plenty of other ~harmless warning shots like an AI reward hacking in some low-stakes case.)
Risks from power concentration / internal deployment are totally unsolved and I have no idea how they might be solved.
I think people's main threat model these days for loss of control is a model achieving rouge internal deployment, or being able to operate internally without much effective oversight in other ways, or sabotaging the control and alignment of future models; I think literally escaping the datacenter and self-exfiltrating the weights is usually considered a relatively small part of the threat model these days. So I think when you say "risks from internal deployment", people usually associate that with the AI achieving rouge internal deployment, or sabotaging alignment research for the next models, so it is confusing to people if you refer to risks from internal deployment as distinct from loss of control.
I agree that AI-enable human power concentration is a distinct risk which can be worsened by internal models becoming much more powerful than anything deployed externally.
2Daniel Tan
Mmm, so if I understand correctly your first point is a hope that power concentrates in a multipolar way, and this multipolar world is stable.
What is a historical example of a "mechanism that redirects power from rivals to the public"? Also what is a present-day example? The closest I can think of is UHNW individuals lobbying the USG to preferentially tax their rivals.
Separately: "Capital in the 21st Century" has statistics which are like "in the past 100 years the fraction of wealth owned by the top 1% has increased rapidly", this seems to be evidence that competitive dynamics between elites aren't a very strong check on concentration of wealth (which is a form of power).
I agree that the current state of alignment isn't sufficient for us to have "sovereign AI". Maybe this is a good reason for continuing to do alignment work. I'll have to think about it more.
Separately: "Capital in the 21st Century" has statistics which are like "in the past 100 years the fraction of wealth owned by the top 1% has increased rapidly", this seems to be evidence that competitive dynamics between elites aren't a very strong check on concentration of wealth (which is a form of power).
I agree that something like the book's scenario could play out once we have AIs that make people's work irrelevant, but I want to note that Piketty's book is just basically wrong about today or the past, and is almost akin to a crank writing about a su... (read more)