While this is relevant to my work at MIRI, I have not checked these ideas with anyone else on the team and am posting this on my personal LW account. These views are my own[1]. And to be honest, I am writing this mostly to remind myself of my weakness.
---
I expect one (or many) adversarial memetic attacks aiming to trip you up, perhaps consisting of fake leaks relating to dangerous stuff happening in the labs. Specifically, worrying incidents that may fit snugly within your worldview, leaking from multiple sources including news outlet/s, but not confirmed/confirmable by a primary source. Think rumors about exfiltrated weights, AIs attempting to create viruses, agent swarms hacking into and gathering information from nuclear infrastructure, etc.
An easy way to remove status from a movement is to trip it up: make it fall for a misinformation trap in public, then use that slip-up to discredit the movement for all time. The game is set for a memetic attack like this. There's a well-resourced group waiting for your screw-up.
And then you may remember much that will help you.
In public and in private, if you feel surprised or confused, notice your confusion. These feelings are signs that your world model doesn't match reality.
Real incidents make you want to act fast. You feel the need to contact journalists, tweet about the incident, and start telling your friends: a memetic attack will feel the same. If you let them trip you, you burn credibility. Set a 5-minute timer and write out your thoughts before acting. Ask yourself:
Since I haven't sanity-checked the concepts below, please take them with a grain of salt and read the comments where people will likely point out my mistakes.
"What about the German Wiki Attack?": The GWA was found by a team of researchers that we trust, and there was a bunch of public evidence that it had actually happened. I anticipate something similar to this being the trap that is set, but I do not expect it to come from a team of researchers that we trust, and I do not expect there to be more than 1-2 pieces of public evidence.
I think that these recommendations are generally a good idea for the epistemic environment of the modern internet regardless of the details.
That said, I find myself lacking anything in my world model that would predict this kind of memetic attack. Are there any specific actors that you think might do this? Is the opportunity just ripe for people with a particular goal? I can't quite think of someone other than in vague terms.
I think that if I had a better model of why the potential memetic attack would be conducted I could probably avoid it better.
Totally.
For now, I recommend these two posts, but am working on something myself.
1. https://thezvi.substack.com/p/openai-14-openai-descends-into-paranoia
2. https://x.com/tyler_johnston/status/2062188304611827899?s=20
And hey! Welcome to LessWrong! You are one day old. Are you here from the big wave after the Coxon Tweet???
Wherever you're from, glad to have you aboard!
I think your comment was meant positively, but this could come across as saying the question was naive. I think it was a very relevant question, and although I have followed the AI safe debate for a decade, I had the same question.
While this is relevant to my work at MIRI, I have not checked these ideas with anyone else on the team and am posting this on my personal LW account. These views are my own[1]. And to be honest, I am writing this mostly to remind myself of my weakness.
---
I expect one (or many) adversarial memetic attacks aiming to trip you up, perhaps consisting of fake leaks relating to dangerous stuff happening in the labs. Specifically, worrying incidents that may fit snugly within your worldview, leaking from multiple sources including news outlet/s, but not confirmed/confirmable by a primary source. Think rumors about exfiltrated weights, AIs attempting to create viruses, agent swarms hacking into and gathering information from nuclear infrastructure, etc.
An easy way to remove status from a movement is to trip it up: make it fall for a misinformation trap in public, then use that slip-up to discredit the movement for all time. The game is set for a memetic attack like this. There's a well-resourced group waiting for your screw-up.
And then you may remember much that will help you.
In public and in private, if you feel surprised or confused, notice your confusion. These feelings are signs that your world model doesn't match reality.
Real incidents make you want to act fast. You feel the need to contact journalists, tweet about the incident, and start telling your friends: a memetic attack will feel the same. If you let them trip you, you burn credibility. Set a 5-minute timer and write out your thoughts before acting. Ask yourself:
Since I haven't sanity-checked the concepts below, please take them with a grain of salt and read the comments where people will likely point out my mistakes.
"What about the German Wiki Attack?": The GWA was found by a team of researchers that we trust, and there was a bunch of public evidence that it had actually happened. I anticipate something similar to this being the trap that is set, but I do not expect it to come from a team of researchers that we trust, and I do not expect there to be more than 1-2 pieces of public evidence.