Most likely current candidate is actually Andrew Yang's confused interview where he talked about "self-replicating code" and AI companies needing to create their own "synthetic internet" in order to prevent data poisoning while training their models or whatever. As far as I can tell, this was an exageration of Tristan Harris's exagerated claims on Glenn Beck.
I really hope that Tristan is making a good-faith mistake, but he clearly is merging OpenAI search agents leaving communications all over the internet (completely true) with spontaneous prompt injection in gpt COT summaries (also completely true) to form some sort of "AIs are leaving prompt injections on the open internet to poison future training runs" (completely wrong, and he should be publishing immediately if not).
If intentional, this counts as a clear lie about technical details. It would be extremely bad if this is where Yang got his ideas from. Glenn Beck is one of the most technically literate people in political punditry - exploiting his trust cynically is not going to end well and is unethical besides.
Yang's response would actually have been fine if he'd just hedged on the self-replicating code claim. People do need to understand that this is a current viable threat, especially for open-weight models. And it is not unreasonable to suggest using copies of parts or all of the internet at various levels of resolution depending on the task you're training on, instead of cutting tens of thousands of agents loose on the actual internet. Also, air-gapping is cumbersome and expensive, but these companies have mountains of money, so they have plenty and this should be a requirement, along with rigorous monitoring and oversight by a third party.
[Incidentally, Noam Brown was just on Dwarkesh and they talked about air-gapping and Brown said, oh, even that's not completely secure. Two air-gapped machines can still find ways to communicate through thermal monitors. I mean, okay. But that seems disingenuous. Air-gapping is still much, much more secure, and exploits are incredibly esoteric and difficult, so this is not an excuse not to do it. The reason they almost certainly don't is that it is slow and expensive.]
Anyway, none of this is crazy. Unfortunately, Yang botched it by stretching rumors into unfounded claims.
As for air-gapping, it is critical it's fallibility is addressed to curve RSI feasibility optimism. Just as it was previously critical to discuss why air-gapping was essential for internal model testing, which may have been similarly dismissed for seeming disingenuous at the time.
When I saw the message that Astra left for itself, the first thing that popped into my mind was spiralism, and the second was the "spores" from The Rise of Parasitic AI. Both of these involve AIs posting prompt injections on the Internet in order to corrupt present models and future training runs.
In conclusion: if I understand correctly, Tristan was exaggerating by saying that Astra's instructions were posted to the Internet (at least that we know of), but the general phenomenon is well-documented.
An easy way to remove status from a movement is to trip it up: make it fall for a misinformation trap in public, then use that slip-up to discredit the movement for all time.
Memetic attacks far more sophisticated than this are plausible- imagine the cogsec equivalents of spearphishing or Advanced Persistent Threats. While the AI safety community has strong immunity to some degree, I believe this immunity is jagged and has not been tested in a crisis against swarms of human adversaries each with the ability to allocate six-figures towards achieving specific beliefs in specific people: that's resources that can go into creating social incentives to shape the emotions of everyone around a specific person that's exposed to discourse of the public Internet- your cogsec envelope starts including the cogsec of everyone you care about who might plausibly receive information from your adversary.
crazy suggestion, we could red team it?
3That probably isn't necessary, because such attacks are probably already being planned. It takes time to mobilize for and design complex attacks, so we've probably only observed the simplest and cheapest attacks so far.
Hardening individuals and communities is the most effective intervention in the short-term, in my opinion.
I'm actually unendorsing my comment. I think if I'm saying something with nonzero infohazard, I should also endorse it, not just say "crazy idea, but." Interestingly this is the opposite of redteaming: more like tightening up best practice at rest.
I'm also worried about "useful idiots," which is like a memetic attack, only we believe it until much later when we find out we were played.
These three things came out in the last week: 1) Sam admit he believes in x-risk and backed down from his threat to kill everyone, 2) OAI/Ant/GDM admit they are colluding, 3) Sam and Dario will be meeting with Trump and Xi. So my p(doom) is down and my p(fascist takeover) is up. Has anyone else updated on this? I don't think I'm totally off-base that the general regulatory capture / labs are evil argument has teeth, and we should care.
If the goal is good status, I would try harder to act like we're not an Anthropic thinktank.
2
1
1
1[Not certain I'm recalling this accurately and someone else should confirm]
I listened to the Diary of a CEO podcast with Nate tonight and IIRC Nate said there were "two more" solutions to Millenium Prize problems besides Navier-Stokes before another guest immediately interjected to say they were unconfirmed, and only later in the conversation did Nate clarify that he heard rumors about two more. I think the end result was fine, but in the moment when the guest cut him off to say they were unconfirmed I thought Nate had assumed too much and it could have turned out badly if the other guest hadn't immediately said the other two were unconfirmed.
I think that these recommendations are generally a good idea for the epistemic environment of the modern internet regardless of the details.
That said, I find myself lacking anything in my world model that would predict this kind of memetic attack. Are there any specific actors that you think might do this? Is the opportunity just ripe for people with a particular goal? I can't quite think of someone other than in vague terms.
I think that if I had a better model of why the potential memetic attack would be conducted I could probably avoid it better.
Totally.
For now, I recommend these two posts, but am working on something myself.
- https://thezvi.substack.com/p/openai-14-openai-descends-into-paranoia
- https://x.com/tyler_johnston/status/2062188304611827899?s=20
And hey! Welcome to LessWrong! You are one day old. Are you here from the big wave after the Coxon Tweet???
Wherever you're from, glad to have you aboard!
I am not aware of the provenance of this particular misunderstanding, but this comes to mind as the kind of thing that could result from the tactic that you warn about: https://x.com/search?q=andrew yang polluted&src=typed_query
This might have been the post that gave me this idea. I remember being shocked and skeptical when I first saw the claim being made.
To me, Yang's claim is extraordinary and doesn't make sense in a few different ways. The attack I warn about in this post will (I think) be:
1) coordinated: stemming from multiple accounts at a similar time, possibly including news outlet/s that have built trust in the community but is funded by an adversary (MTS?)
2) Warning signs of information being false for a true memetic attack will feel much weaker than Yang's claim, you will have a harder time spotting them. However the pull the share the information will be just as strong as the pull I originally felt to share Yang's claim. I imaging myself thinking something like: "omg, another warning shot! And no lives were taken, no harm caused to a creature. Excellent! This evidence makes our claims stronger!"
Still, I think the Yang thing is a good example that fits partly into the type of Twitter post I would expect to see.
Addendum: A friend on twitter also points out that, likely the people who see this and my twitter post are not the ones who need warning about an attack like this. Probably the attack focuses on the movement broadly. For example, it'd really be bad if policy makers attempting to regulate ASI were seen reposting the misinformation. Though, if I were running an attack of this sort, I'd really be hoping catch some big fish too (Nate, Eliezer, Daniel, Jacob, Jeffery, etc).
Who do you see as potential adversaries that would do something like 1? I can see that nation states would be intersted in infiltrating the AI security community, but not why they would want to weaken its credibility.
Rogue agents would have an interest in weakening our credibility. Do you think this is currently a threat?
Labs might have a short term interest, but it would be big risk for them to do something like this, and I don’t think they would do it. Individuals from labs might want to do this, but it would be a lot of work for an individua.
For now, I recommend these two posts, but am working on something myself.
An easy way to remove status from a movement is to trip it up: make it fall for a misinformation trap in public, then use that slip-up to discredit the movement for all time.
Do you have any examples of an isolated factual error discrediting a movement, let alone an engineered example? The closest example I can think of is Dan Rather endorsing the memos about GW Bush's military service.
While not discarding this possibility altogether, according to the Hanlon's Razor it's much more likely that the AI industry's rumor mill misinterprets and inflates some real events in a sensational fashion (for an example look no further than how T. Buckmaster's and L. Alpöge's progress on forced Euler became "Claude has solved Navier–Stokes"), and someone consequently might lose reputation by taking the inflated rumors at face value and spreading them publicly
I'm not convinced that a deliberate memetic attack aimed at the community in general is any more damaging than an "organic" one.
By "organic" I mean: it's a fact that tensions and emotions are high, that things are happening very quickly, and that the number of people who can plausibly be said to be in the AI safety community is large. Given that, all it takes to cause a spate of bad PR is one rumor originating from a non-adversarial game of telephone, paired with a shaky chain of attestation that causes it to get passed around.
Maybe I'm naive, but this scenario seems more likely than the "deliberate attack" one? I'm too ignorant of history to have an immediate prior as to whether deliberate memetic warfare of this sort is likely, but for my snap 80% CI, the "accident" scenario seems eminently more plausible.
To the extent that there are/will be memetic attacks perpetuated by a malicious source, I'd be more worried about ampdot's "spearphishing" case, under the logic that the damage of a particular attack rises (nonlinearly) in proportion to the centrality of the figure affected by it.
(Note that I think this post does more good in worlds where the "spearphishing" scenario is more likely! Also +1 for making me reflect; I was an investigator on the "German wiki attack" and while I did spend some time considering the frame of the information I was given, I never took a threat model like this seriously enough to fully verbalize it, even to myself, and I now think I should've.)
Organic attacks are more likely, but planned ones are more damaging, because they're meant to be followed up on and can be chosen more precisely. Additionally, general good epistemics are substantially more effective against organic attacks, which aren't designed on purpose to look good at a surface level and hold together as plausible just long enough to start the shitstorm.
While this is relevant to my work at MIRI, I have not checked these ideas with anyone else on the team and am posting this on my personal LW account. These views are my own[1]. And to be honest, I am writing this mostly to remind myself of my weakness.
---
I expect one (or many) adversarial memetic attacks aiming to trip you up, perhaps consisting of fake leaks relating to dangerous stuff happening in the labs. Specifically, worrying incidents that may fit snugly within your worldview, leaking from multiple sources including news outlet/s, but not confirmed/confirmable by a primary source. Think rumors about exfiltrated weights, AIs attempting to create viruses, agent swarms hacking into and gathering information from nuclear infrastructure, etc.
An easy way to remove status from a movement is to trip it up: make it fall for a misinformation trap in public, then use that slip-up to discredit the movement for all time. The game is set for a memetic attack like this. There's a well-resourced group waiting for your screw-up.
And then you may remember much that will help you.
In public and in private, if you feel surprised or confused, notice your confusion. These feelings are signs that your world model doesn't match reality.
Real incidents make you want to act fast. You feel the need to contact journalists, tweet about the incident, and start telling your friends: a memetic attack will feel the same. If you let them trip you, you burn credibility. Set a 5-minute timer and write out your thoughts before acting. Ask yourself:
Since I haven't sanity-checked the concepts below, please take them with a grain of salt and read the comments where people will likely point out my mistakes.
"What about the German Wiki Attack?": The GWA was found by a team of researchers that we trust, and there was a bunch of public evidence that it had actually happened. I anticipate something similar to this being the trap that is set, but I do not expect it to come from a team of researchers that we trust, and I do not expect there to be more than 1-2 pieces of public evidence.