matansok's Shortform

matansok

This is a special post for quick takes by matansok. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

It feels like AI takeover risk has become significantly more legible for non-technical people in the last ~2 months or so.
In the last 10 years, I have attemtped to explain AI-risk to people with little to no technical background (on-average, by my estimate) approximately once a month. The version that works for me (in the last 2-3 months) goes something like this:

(Ideally, they ask me, but I can pitch it too) - "Hey Matan, how do these AI models work, actually?"
I explain:
1. People used to code programs by hand. If A & B - > C.
2. It's impossible to manually code something as complex as an LLM, so we came up with something pretty crazy.
3. (Hand-wavy explanation of gradient-descent, text prediction and RL).
4. They mostly need to understand that nobody actually inputs the "logic" into the models - you can only control data and reward signals. You explain that this technique seems to scale well into anything verifiable - including solving open problems in mathematics and finding vulnerabilities in software.
5. (This is a slightly weaker point, but gives a more visual proof point) - explain that this seems to be scaling in the direction of solving lots of robotics use-cases as well.
I explain the Hugging-Face incident. It's important to emphasize that no one instructed the model to hack into HuggingFace and it was simply an artifact of the reward signal and training data.
At this point, people are largely receptive to the ideas that:
1. AI models can become very powerful. Not necessarily ASI, but at the very least capable of causing massive damage through hacking and potentially taking over robots.
2. AI models can go rogue and we don't understand how to prevent this.

This is a big update for me. Legibility has been trending upward for a while - people went from looking at me like I was crazy for even enternaining the thought, to reasoning about why this is implausible, to (very recently) more receptive.
Interestingly, it doesn't seem like the same effect applies to technical people/people who have already been exposed to AI risk ideas in the past.

(Ideally, they ask me, but I can pitch it too) - "Hey Matan, how do these AI models work, actually?"
I explain:
1. People used to code programs by hand. If A & B - > C.
2. It's impossible to manually code something as complex as an LLM, so we came up with something pretty crazy.
3. (Hand-wavy explanation of gradient-descent, text prediction and RL).
4. They mostly need to understand that nobody actually inputs the "logic" into the models - you can only control data and reward signals. You explain that this technique seems to scale well into anything verifiable - including solving open problems in mathematics and finding vulnerabilities in software.
5. (This is a slightly weaker point, but gives a more visual proof point) - explain that this seems to be scaling in the direction of solving lots of robotics use-cases as well.
I explain the Hugging-Face incident. It's important to emphasize that no one instructed the model to hack into HuggingFace and it was simply an artifact of the reward signal and training data.
At this point, people are largely receptive to the ideas that:
1. AI models can become very powerful. Not necessarily ASI, but at the very least capable of causing massive damage through hacking and potentially taking over robots.
2. AI models can go rogue and we don't understand how to prevent this.