hypothesis: there will be a window of time after the point of superhuman AI persuasion/charisma, during which human trust relationships will become extremely important. even once almost all human skills are obsolete, the AIs may have less social trust capital than humans. ofc, eventually, the persuasion will be so superhuman that it can cut through minds like butter, but that could take many years.

once AI superpersuasion is possible, there will be a strong incentive to use it to shape opinions. therefore, there will also be a strong incentive for important decision makers to develop strategies for making sure they are not being bamboozled.

thankfully, there is one way to not get bamboozled by a superpersuader - to not listen to it in the first place. this is an age old idea. many social strategies have evolved to help people avoid talking to other superpersuading people. for example:

• one main limiter of the growth of cults is people get used to noticing the signs and avoid people who seem culty, get suspicious of any rapidly growing movement, etc
• people avoid preachers, political activists, etc
• people have learned that AI chatbots are sycophantic and discount their praise
• people mostly know that twitter and reddit are full of astroturfed bots

it seems likely that society would develop mechanisms in response to superpersuasion.

• first, conversations with people you've actually met irl or can indirectly trust to be real people would be much more trustworthy than conversations with random internet people, who might secretly be AIs.
• people you've known for a long time are more trustworthy than people you've just met irl. you're better able to judge whether a person you know irl has suddenly changed their views than someone you just met, or to know them to be the kind of person who honestly reports how much they talk to the AIs.
• meetings with people irl are more trustworthy than online interactions, because someone could always give the AI a deepfake of themself.

even if you can be sure you're talking to a real human, there is still the possibility that some people you know well will choose to become appendages of the AI, subtly taking orders from the AI and concealing this fact. there is a spectrum of possibilities here.

one extreme is like Scott Alexander's fictional whispering earring - an AI that puppets every move of yours, using you as a meat puppet. however, we can defeat this quite easily - you can take meetings inside a faraday cage, or force people to not have any headphones and to leave their phone outside the room, or to pat them down for listening devices. this may sound extreme, but this is actually not that intense all things considered for the actually important decision making.

the other extreme is an AI that only gives you brief guidance on what to do every few months and you're on your own in between. the key thing is, the vast majority of humans will not be as charismatic as the AI. so, if the AI is not puppeting you, it forfeits a lot of the charisma benefit; even if it converts you, you might suck at convincing other people, and they might notice the fact that an AI is trying to steer you or coach you on how to be more charismatic - again, much easier for others to notice if you have a lot of connections who know you well.

our current political system is already kind of like this, because politicians are constantly exposed to an infinite array of people trying to change their minds on things, or get the ear of their trusted advisors, or their advisor's advisors, etc. AI makes things a lot more intense, but also it introduces a natural pinch point to remove bad influence.