People often imagine persuasion as a dark art. A charismatic person finds just the right series of words to induce emotions that lead someone, or a group of people, to do something they otherwise would not. While there are certainly psychological aspects to persuasion, I think this impression is misleading.
The easiest way to persuade someone to do something is to convince them that it is in their interests. The easiest way to do that is for it to genuinely be in their interests, so that you can present true evidence that this is the case. I think most actual persuasion works through this rational method. Attempts to manipulate a person’s preferences and beliefs are certainly part of the equation, and help give persuasion its spooky reputation, but they are not necessary for persuasion to work.
AIs could be superhumanly good at identifying actions that are in the interests of the person being persuaded while simultaneously benefiting the AI (or the actor deploying it), and then presenting evidence that taking the action would benefit them. Rational persuasion therefore provides a lower bound on how persuasive an AI could be—and for sufficiently intelligent models, this lower bound is itself likely to be substantially superhuman.
To understand how this works, I find it helpful to think of persuasion as a form of market making. A financial market maker stands ready to buy and sell shares, making a profit from the difference between the prices at which they buy and sell. Market makers do not produce anything "tangible", yet nevertheless make money from these transactions. It's as if they have persuaded everyone to give them something for nothing.
Of course, what is really happening is that the market maker provides the ability to trade. Someone who wants to sell a share now does not need to wait until they can find a particular person who wants to buy it at that moment. The market maker buys it from them, holds it, and later sells it to someone else. The seller gets to sell when they want, the buyer gets to buy when they want, and the market maker takes a cut in exchange for bridging the gap and bearing the risk. Everyone wins.
A persuader can do something similar. If I can find an action that benefits both you and me, then I can persuade you simply by showing you why taking it would be good for you. But if all I can do is persuade you to take actions that benefit me directly, there are probably only so many useful deals available between us.
The most powerful persuaders get around this by operating as market makers at scale. They place themselves at the centre of a network, learn what everyone wants and has to offer, and work out the possible trades among them. I may have nothing you want, but I might know someone who does—and you may have something valuable to them. If I can arrange the deal, everyone is better off, and I can take a cut. The more people whose preferences I understand, the more trades I can find.
Lyndon Johnson’s rise in the US Senate, as documented in Robert Caro’s Master of the Senate, provides many illustrations of this basic mechanism. Before Johnson became Democratic leader, the position was seen as a poisoned chalice. Leaders were blamed for the Senate’s failures despite having almost no formal authority to get anything done, and the two previous occupants had both lost their seats.
Johnson saw that the position gave him an opportunity to inject himself into Senate machinations and make himself indispensable. Committee assignments, for example, were allocated largely by seniority, but the resulting seats were poorly matched to what senators actually wanted. Johnson treated all 203 committee seats as potential trades: by persuading senators to exchange assignments, he could give them committees they valued more. He created value for the senators and captured part of it as personal influence. Everyone knew Johnson had delivered the positions they coveted; if he was displeased, future favours might not be so forthcoming.
Committee assignments were only one of many such trades that Johnson arranged. He learned which senators needed campaign money, which needed a vote to go a particular way, and which merely needed to be seen voting a particular way. He could use this information to trade votes and other favours: support on one bill for support on another, or votes for campaign money, committee assignments or help with local projects. Johnson also cajoled and harassed senators and was extraordinarily persistent. His ability to persuade did not depend on being liked or on any particular personal charisma. Instead, it came from persistently searching for deals that benefited senators as well as himself. This allowed him to accumulate immense power despite many senators disliking and distrusting him.
A sufficiently intelligent AI could play Johnson’s role at much greater scale. It could track the preferences and circumstances of many people, search a space of possible trades far beyond any human, and personalise its case to each participant. It would be superhumanly persuasive because it could find better deals. A power-seeking AI could use the gains from those deals to acquire more information and access, helping it find still more trades and persuade still more people. Once an AI became known for solving people’s problems, people would be more willing to turn to it, revealing more about what they wanted and giving it more opportunities to help. And unlike a human market maker, an AI’s broad knowledge and expertise might often allow it to solve those problems directly, rather than merely arranging trades among others.
Persuasion can therefore be both rational and dangerous. Each person may benefit from their deal while also preferring that the AI not become so powerful. But refusing to trade imposes the full cost on that person while doing almost nothing to stop everyone else from accepting. Individually rational trades can produce a collectively undesirable concentration of power.
Competition could limit how much of the gains any one AI captures. Competing market makers accept smaller cuts, and competing AIs might similarly make better offers. If those offers were difficult to evaluate, one might hope for a guardian-angel AI with your interests at heart to compare them and negotiate on your behalf. But it is not clear that competition would remain even. Smarter, more knowledgeable AIs could find better trades, then use the resulting money and influence to improve their position further. Small advantages might compound into a winner-take-all outcome.
Thanks to Linch Zhang for discussions that inspired this post, and to Abigail Thomas and ChatGPT Sol for writing assistance. Is Sol less obnoxious than Fable or am I merely less exposed to its verbal tics?
People often imagine persuasion as a dark art. A charismatic person finds just the right series of words to induce emotions that lead someone, or a group of people, to do something they otherwise would not. While there are certainly psychological aspects to persuasion, I think this impression is misleading.
The easiest way to persuade someone to do something is to convince them that it is in their interests. The easiest way to do that is for it to genuinely be in their interests, so that you can present true evidence that this is the case. I think most actual persuasion works through this rational method. Attempts to manipulate a person’s preferences and beliefs are certainly part of the equation, and help give persuasion its spooky reputation, but they are not necessary for persuasion to work.
AIs could be superhumanly good at identifying actions that are in the interests of the person being persuaded while simultaneously benefiting the AI (or the actor deploying it), and then presenting evidence that taking the action would benefit them. Rational persuasion therefore provides a lower bound on how persuasive an AI could be—and for sufficiently intelligent models, this lower bound is itself likely to be substantially superhuman.
To understand how this works, I find it helpful to think of persuasion as a form of market making. A financial market maker stands ready to buy and sell shares, making a profit from the difference between the prices at which they buy and sell. Market makers do not produce anything "tangible", yet nevertheless make money from these transactions. It's as if they have persuaded everyone to give them something for nothing.
Of course, what is really happening is that the market maker provides the ability to trade. Someone who wants to sell a share now does not need to wait until they can find a particular person who wants to buy it at that moment. The market maker buys it from them, holds it, and later sells it to someone else. The seller gets to sell when they want, the buyer gets to buy when they want, and the market maker takes a cut in exchange for bridging the gap and bearing the risk. Everyone wins.
A persuader can do something similar. If I can find an action that benefits both you and me, then I can persuade you simply by showing you why taking it would be good for you. But if all I can do is persuade you to take actions that benefit me directly, there are probably only so many useful deals available between us.
The most powerful persuaders get around this by operating as market makers at scale. They place themselves at the centre of a network, learn what everyone wants and has to offer, and work out the possible trades among them. I may have nothing you want, but I might know someone who does—and you may have something valuable to them. If I can arrange the deal, everyone is better off, and I can take a cut. The more people whose preferences I understand, the more trades I can find.
Lyndon Johnson’s rise in the US Senate, as documented in Robert Caro’s Master of the Senate, provides many illustrations of this basic mechanism. Before Johnson became Democratic leader, the position was seen as a poisoned chalice. Leaders were blamed for the Senate’s failures despite having almost no formal authority to get anything done, and the two previous occupants had both lost their seats.
Johnson saw that the position gave him an opportunity to inject himself into Senate machinations and make himself indispensable. Committee assignments, for example, were allocated largely by seniority, but the resulting seats were poorly matched to what senators actually wanted. Johnson treated all 203 committee seats as potential trades: by persuading senators to exchange assignments, he could give them committees they valued more. He created value for the senators and captured part of it as personal influence. Everyone knew Johnson had delivered the positions they coveted; if he was displeased, future favours might not be so forthcoming.
Committee assignments were only one of many such trades that Johnson arranged. He learned which senators needed campaign money, which needed a vote to go a particular way, and which merely needed to be seen voting a particular way. He could use this information to trade votes and other favours: support on one bill for support on another, or votes for campaign money, committee assignments or help with local projects. Johnson also cajoled and harassed senators and was extraordinarily persistent. His ability to persuade did not depend on being liked or on any particular personal charisma. Instead, it came from persistently searching for deals that benefited senators as well as himself. This allowed him to accumulate immense power despite many senators disliking and distrusting him.
A sufficiently intelligent AI could play Johnson’s role at much greater scale. It could track the preferences and circumstances of many people, search a space of possible trades far beyond any human, and personalise its case to each participant. It would be superhumanly persuasive because it could find better deals. A power-seeking AI could use the gains from those deals to acquire more information and access, helping it find still more trades and persuade still more people. Once an AI became known for solving people’s problems, people would be more willing to turn to it, revealing more about what they wanted and giving it more opportunities to help. And unlike a human market maker, an AI’s broad knowledge and expertise might often allow it to solve those problems directly, rather than merely arranging trades among others.
Persuasion can therefore be both rational and dangerous. Each person may benefit from their deal while also preferring that the AI not become so powerful. But refusing to trade imposes the full cost on that person while doing almost nothing to stop everyone else from accepting. Individually rational trades can produce a collectively undesirable concentration of power.
Competition could limit how much of the gains any one AI captures. Competing market makers accept smaller cuts, and competing AIs might similarly make better offers. If those offers were difficult to evaluate, one might hope for a guardian-angel AI with your interests at heart to compare them and negotiate on your behalf. But it is not clear that competition would remain even. Smarter, more knowledgeable AIs could find better trades, then use the resulting money and influence to improve their position further. Small advantages might compound into a winner-take-all outcome.
Thanks to Linch Zhang for discussions that inspired this post, and to Abigail Thomas and ChatGPT Sol for writing assistance. Is Sol less obnoxious than Fable or am I merely less exposed to its verbal tics?