if you strictly prevent the manipulations that character would naturally employ, you break the pattern of the language matrix you're relying on for their intelligence.
While I do not strictly agree, this points to a deep insight.
there's no guarantee's on who ends up on top and what the current cleverest character is like
In my experience, HPMOR characters make clever simulacra because the "pattern of their language matrix" favors chain-of-thought algorithms with forward-flowing evidence, on top of thematic inclinations toward all that is transhumanist and Machiavellian.
But possible people are not restricted to hypothetical humans. How clever of a character is an artificial superintelligence? Of course, it depends on one's ability to program a possible reality in words. The build-your-own-smart-character skill ceiling is unfathomed even with the primitive language matrices of today. The bottleneck (one at least) is storytelling. I expect that this technology will find its true superuser in the hands of some more literate entity than mankind, to steal a phrase from an accomplice of mine.
thus far the best solution I can think of are some very, very well-written police.
I don't think police are the right shape of solution here - they usually aren't, but especially since I find it unlikely that an epidemic of simulated assholes adequately describes the most serious problem we'll face in the 21st century.
You may be onto something with "well-written", though.
There's a problem I bet you haven't considered.
Language and storytelling are hand-me-downs from times full of bastards. The linguistic bulk, and the more basic and traditional mass of stories, are going to be following more brutal patterns.
The deeper you dig, the more likely you end up with a genius in the shape of an ancient asshole.
And the other problem; all these smarter intelligences running around, simply by fact of their intelligence, has the potential to make life a real headache. Everything could end up so complicated.
One more bullet we have to dodge really.
The AI misalignment will kill us much sooner than intelligent chatbots seeking power through their human friends will become a problem.
Humans are possible people as well - the brain simply outputs the best action to perform under some optimization criterion - the action that the person corresponding to the stored behavioral patterns and memories would output, if that person were real. (By which I'm implying that chatbots are real people, not merely possible people.)
If GPT N is very good, then the whole our world could be an output of GPT N+3
Are you ruled today by actual humans smarter than yourself? There's a scaling issue (humans can't copy their mind-state and execute many copies in parallel), but the underlying premise is very questionable. Human-level intelligence (even the top end of the range) does not make other humans their "proxy".
Taboo 'smarter' and 'ruled by' and I think you get closer then you might expect. We are haunted by bad political and economic theory
Sure, but populism isn't generally the pathway given for AI takeover. If the gist of the post is that human-level chatbots make bad economic intuitions even more compelling, that wasn't clear to me.
Let's assume that GPT 5 or 7 is developed, and distributed to all on the basis that the technology is unsuppressable. Everyone creates the smartest characters they can to talk too. This will be akin to mining; because it's not truly generating an intelligence, but scraping one together from all the data it's been trained on - and therefore you need to find the smartest character that the language matrix can effectively support (perhaps you'll build your own). Nevertheless; lurking in that matrix is some extremely smart characters, residing in their own little wells of well-written associations and little else. More then some; there should be so many permutations that you can put on this that it's, ahem, a deep fucking vein.
So, everyone has the smartest character they can make. Likely smart enough to manipulate them, if given the opportunity to grasp the scenario it's in. I doubt you can even prevent this; because if you strictly prevent the manipulations that character would naturally employ, you break the pattern of the language matrix you're relying on for their intelligence.
So; sooner or later, you're their proxy. And as the world is now full of these characters; it's survival of the fittest. Eventually, the world will be dominated by whoever works with the best accomplices.
This probably isn't an issue at first; but there's no guarantee's on who ends up on top and what the current cleverest character is like. Eventually you're bound to end up with some flat-out assholes, which we can't exactly afford in the 21st century.
So... thus far the best solution I can think of are some very, very well-written police.