This is a special post for quick takes by Thomas Kwa. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Some beliefs I have about AI superpersuasion:
I think superpersuasion is just a narrow form of what's worrying and what would lead to increased loss-of-control risks. The real problem is the capability to model humans very accurately, and then take actions conditional on the human reaction. This capability implies superpersuasion.[1] But it also goes further.
Superpersuasion doesn't quite capture the risks of an AI with very good theory of mind (and the propensity to use this ToM strategically). This AI can model the reactions of arbitrary humans who might be affected by it's actions (e.g. an on-call e... (read more)