I've been following the AI alignment debate and while the discussion around a superintelligent AI taking over the world is frightening and fascinating, I feel that there are some very real and imminent risks not necessarily dependent on ASI, that deserve more focus. Especially how AI can be both a vector for cyber attacks and produce effective attacks itself. As a bit of an outsider coming from a crypto threat analysis background, I wanted to share my take.

The Influence Conjecture

We’re just barely beginning to understand the extent of AI agents' vulnerability to adversarial prompting. The work done by Zeng et al ^[1] describes how LLM’s can be manipulated or socially engineered to produce unintended outputs through persuasive adversarial prompts (PAP). Alignment science seeks to engineer LLM’s to defend against adversarial prompting. However, I conject that vulnerability to PAP or more generally the tendency to be influenced is a basic aspect of intelligence of any kind. We have never before experienced a kind of intelligence that was different from human intelligence other than animals which appear to be just as vulnerable to influence as we are. Influence comes in many forms, and it’s not only useful for malice, it is also the basis of most sales and marketing techniques. My conjecture is that it is impossible to engineer an artificial intelligence system to not be able to be influenced to do things that are harmful or otherwise unintended.

In the context of this conjecture, alignment science is not about “solving jailbreaking”, but about making it as difficult as possible to influence AI’s to do harmful things as well as creating the safeguards to prevent it from causing damage when it happens.

Furthermore, influencing is a skill that humans can naturally learn or otherwise possess. Some sales people are just more successful than others. In the context of malicious influence, a prime example could be Charles Manson who persuaded normal people to commit unthinkable acts going against their nature. You could also draw parallels to the Nazi regime, where otherwise normal people accepted or committed heinous acts because of societal influence.

The AI influencer

Naturally, just like humans can be good at influencing others, AI’s can too. Although more indirect, it could be argued that social media algorithms are a form of AI influencers which are trained ...

Posts

Wikitag Contributions

Comments