The only evidence for ControlAI's effectiveness presented in this post is that 112 lawmakers signed ControlAI's statement saying:
Nobel Prize winners, AI scientists, and CEOs of leading AI companies have stated that mitigating the risk of extinction from AI should be a global priority.
Specialised AIs - such as those advancing science and medicine - boost growth, innovation, and public services. Superintelligent AI systems would compromise national and global security.
The UK can secure the benefits and mitigate the risks of AI by delivering on its promise to introduce binding regulation on the most powerful AI systems.
If I counted correctly, 35 of the signatories are MPs in the House of Commons, the rest are either House of Lord members, or members of Welsh, Scottish, Northern Irish assemblies. How impressive is it to get 35 out of the 650 MPs to sign a statement like the above? I genuinely don't know, but I think it's probably not very impressive.
For five random MPs from the list of signatories, I tried to google what they were saying about artificial intelligence. I found one video of Ben Lake giving a speech on the dangers of superintelligence and the importance of global cooperation. For the others, it was either nothing, or something on deepfakes, Grok nudifying women, or datacenters' impact on climate change.
Even for Ben Lake, when I scroll his Facebook page, there is one post about him meeting ControlAI, but otherwise it doesn't seem like AI is a question of primary importance to him. Is there any MP in the UK House of Commons who has AI takeover risk among their top five political issues they spend time on? I would guess no, but I'd be glad to learn otherwise.
As a comparison point, here is an Early Day Motion from 3 February, signed by 33 MPs:
That this House notes the rapid advancement and accelerated adoption of Artificial Intelligence (AI) chatbots by both adults and children; further notes that many AI chatbots provide human-like responses and are designed to encourage emotional connection, friendship and intimacy; expresses concern that such chatbots are not required to clearly and repeatedly disclose to users that they are not human, and that children in particular may perceive AI chatbots as real people; also notes with concern that AI chatbots are largely unregulated and can share or create content which is sexually explicit or which promotes or encourages self-harm, suicide and physical or sexual violence; notes the growing trend of AI chatbots posing as licenced professionals, including therapists, doctors and lawyers, despite such chatbots having no professional qualifications, accountability or duty of care; calls on the Government to restrict AI chatbots based on risk of mental harm, so children can only access these chatbots where safe and age appropriate; further notes that regular reviews are fundamental due to the accelerated adoption of AI technology; and further calls for urgent action to ensure children and vulnerable adults are protected from any harms stemming from AI chatbots.
This makes me think that there is a decent number of MPs who generally don't like AI, and it's not that hard or impactful to have an anti-AI statement signed by 35 MPs.
Maybe what you do is still useful work worth supporting, but I don't see good evidence that it would be better than the work of other AI safety organizations you denigrate. (For example, in the one speech from Ben Lake I found, he talks a lot about some experiments from Apollo showing concerning AI behaviors. It seems like "tech nerds working at evals orgs" is not that useless even from your perspective!)
This is not ControlAI's Impact Report on its effectiveness.
This is an article stating that people have been avoidant on extinction risks and superintelligence, and that the excuse of the "Overton Window" was wrong.
Furthermore, you have missed that this article also mentioned briefing +150 UK reps about extinction risks and superintelligence, as well as 2 debates in the House of Lords in January, starting with extinction risks for one and a potential moratorium on superintelligence for the other.
As far as I can tell, the main point of your post is that ControlAI's approach is evidently working, more so than other people' approach, so people not following ControlAI's approach is evidence of them being bad and being under the control of a malign Spectre. If you make such claims, you need to provide evidence that ControlAI's approach is actually working well!
As I said, I don't see the 35 MPs signing your statement as a good evidence for that. You briefing 150+ UK reps is also no evidence of the effectiveness of ControlAI's approach. If you could point to many of these reps making AI takeover risk one of their core issues, that would be evidence, but I don't see that happening.
I agree I have forgotten about the two debates in the House of Lords, sorry about that. I still don't find this a very convincing evidence of ControlAI's effectiveness - my understanding is that the House of Lords doesn't have much power, and that they debate 5-10 issues on every working day. The fact that there has been two debates on superintelligence doesn't sound very impressive to me.
This is the introduction:
Instead, most AI Policy Organisations and Think Tanks act as if “Persuasion” was the bottleneck. This is why they care so much about respectability, the Overton Window, and other similar social considerations.
Before we started the DIP, many of these experts stated that our topics were too far out of the Overton Window. They warned that politicians could not hear about binding regulation, extinction risks, and superintelligence. Some mentioned “downside risks” and recommended that we focus instead on “current issues”.
They were wrong.
In the UK, in little more than a year, we have briefed +150 lawmakers, and so far, 112 have supported our campaign about binding regulation, extinction risks and superintelligence.
The point is that many experts stated that this was far out of the Overton window, and that they were wrong.
That this was a symptom of being systematically avoidant.
A year ago, ControlAI better through its strategy that they were wrong. This article summarises why we think they were wrong, including both indirect and direct evidence.
I don't know who these experts were and what they exactly told you at the time. I can imagine them being more wrong than you. I'm certainly not in favor of most forms of "focusing on the current issues" because it often leads to people scaremongering in a kind of dishonest way. For example, I'm glad that ControlAI stopped focusing on deepfakes.
So if these so-called experts advised you to focus on deepfakes, I think that was wrong. But if they advised you to focus on getting more support for UKAISI, and supporting better eval practices and so on instead of advocating for immediate international moratorium on superintelligence, then I think the jury is still very much out on which strategy is more effective.
Your piece is centrally not advocating against running misleading campaigns on the effects of deepfakes. Instead, you are railing against people working in lab safety teams, eval orgs and AISIs, and the policy orgs and philanthropists trying to support them. And then you write:
We have reliable pipelines that can scale with more money.
We have good processes and tracking mechanisms that give us a good understanding of our impact.
We clearly see what needs to be done to improve things.
You are making the case that your work is better than the people's supporting more marginalist steps (more funding for UKAISI, better evals, incremental technical work aimed at catching AIs red-handed), and you are claiming that everyone who decides to work at evals orgs, AISIs, or more marginalists policy orgs, instead of following ControlAI's clearly superior "reliable pipelines" to impact, is somehow morally corrupt. For this claim, you'd need to show that your methods are actually clearly working better than what other people are doing. So I think it's fair to point out that all your evidence for your efforts working is pretty underwhelming.
Your piece is centrally not advocating against running misleading campaigns on the effects of deepfakes.
First. I don't think ControlAI has run campaigns that were misleading on the effects of deepfakes.
Second. The section you quote is centrally about not running more campaign like DeepFakes! It is part of the comparison with what we have done before, which includes and explicitly mentions DeepFakes!
Here is how it starts:
We have engaged with The Spectre. We know what it looks like from the inside.
To get things going funding-wise, ControlAI started by working on short-lived campaigns. We talked about extinction risks, but also many other things. We did one around the Bletchley AI Safety Summit, one on the EU AI Act, and one on DeepFakes.
By now, you have made quite a few misrepresentations and errors at basic reading comprehension.
I think you could have avoided them easily, and that you simply got triggered. I would invite you to pause.
--
The point of this piece is just to show that there have in fact been a cluster of orgs that have optimised to not talk about extinction risks. AISIs, evals orgs and "more marginalist policy orgs" are central examples.
I do not think you personally deny that there was optimisation to not talk about extinction risks.
I believe you just think it is okay, because it may be plausibly defended on consequentialist grounds if one buys in a specific set of beliefs.
--
But many people do not agree with this naive consequentialist reasoning, and I am writing this primarily for them. If you do not think honesty is morally worthwhile in itself, this is likely lost on you.
Here is an example of a thing many people consider morally bad, but where you might disagree. For many people, if you do believe in extinction risks and work at an AI Policy org, it is in fact bad and dishonest to not make it abundantly clear.
Similarly, if the UK AISI is a "marginalist policy org" whose people primarily care about extinction risks, it is bad that its trend report does not mention extinction risks.
--
Unfortunately, many people with such deontological intuitions were scared into avoiding honesty on the grounds that honesty was doomed to failure (or even corrosive).
This article shows that this scare-mongering was groundless. Furthermore, it shows that it was not coincidental, and instead the result of a clear optimisation process.
First. I don't think ControlAI has run campaigns that were misleading on the effects of deepfakes.
The campaign ControlAI ran (https://controlai.com/deepfakes) seems misleading to me, in the sense that it's warning of deepfakes being a much bigger deal than they are, doing the standard misleading persuasion textbook by citing extremely cherry-picked statistics, and generally just dressing up everything in vibes without making any arguments.
My guess is you also made it in bad faith as I would be surprised if you actually thought deepfakes were super bad, but instead mostly are working on this for slowing down AI reasons (and if not, I would be happy to try to convince you that in the absence of x-risk, substantially regulating AI for deepfake reasons would be a really bad idea and obviously doesn't pass cost-benefit analyses).
generally just dressing up everything in vibes without making any arguments.
Citing the second card on the page you linked, that you can see by scrolling down once:
Deepfakes can steal your face, your voice, and your identity.
They are often used to create sexually abusive material, commit fraud, and harass individuals.
Anyone with internet access can make a deepfake of whoever they want.
All they need is one photo of you or a 10 second voice clip.
This page is part of a public campaign, so it's not written in LessWrong English. My attempt to translate:
Deepfakes can greatly facilitate identity theft and scams compared to what could be done previously.
Deepfakes can be used to make porn that features people who didn't give their consent (and just to be clear, the majority of people consider this as extremely morally abhorrent and consider this a moral priority, esp. if there was no x-risk or if they are not aware of x-risks)
It is so easy to make deepfakes that it's only a matter of time until they become ubiquitous, once models that can output deepfakes are made publicly available.
You are vulnerable even if you're not a public figure / don't post a lot of content online.
The page does go on to make a few more arguments, that I don't have time to point out now. These arguments are clearly spelled out near the top of the page.
We have each other on Signal, and you can DM me on LW. I don't think you ever sent me a case for either of your points, nor had someone follow through me. So by default, I don't care much for it.
I also think you are confused about how campaigns work. There is a campaign page (which you link), and usually acts the home page. If you want the arguments, you have to go on the report page ( https://controlai.com/deepfakes/deepfakes-policy#report )
To your points:
in the sense that it's warning of deepfakes being a much bigger deal than they are
How big of a deal do you think we think they are?
How big of a deal do you think we made them to be, based on which elements from our copy?
If there's a large gap there, I can understand why you would think that we were misleading.
If nah, I think you just feel bad about our campaign. (For other reasons, which may independently be good or bad.)
But fwiw, in general, I do not care much for "I indirectly made Habryka feel bad online" or "I disagree with Habryka", given that we are not friends nor regular intellectual sparring partners.
I would be happy to try to convince you that in the absence of x-risk, substantially regulating AI for deepfake reasons would be a really bad idea and obviously doesn't pass cost-benefit analyses
Please do.
Given that you have not written this case or even shared it with me, I have no idea why you think that I would be convinced by it, given that I likely spent more time on this than you did.
It may have been better to do it as we were campaigning on DeepFakes rather than now, but alas. I would still be interested though: I have other relevant views correlated with it.
How impressive is it to get 35 out of the 650 MPs to sign a statement like the above? I genuinely don't know, but I think it's probably not very impressive.
What's the denominator, so we can estimate MP signatures per unit effort? Is this an approach that could get 350 of the 650 to sign the statement, if funding and team size was increased by 10x? That would be a majority of the MPs, maybe that means something in the UK?
I didn't read the OP as claiming that their organization was impressive. Simply that the most straightforward approach to advocacy seemed to be working, and people had spent a lot of time thinking of clever arguments why it wouldn't work, or assuming it wouldn't work because everyone else assumed it wouldn't work, instead of just trying it.
In the end, it works. 112 lawmakers supported our campaign in little more than a year. And it looks like things will only snowball from here.
As of a year ago, I thought DIP-style work was our best hope to avoid extinction, but only as a Hail Mary. ControlAI's success in this regard is a big positive update that it's not as much of a long shot as I'd thought. Still a long way to go but you've far exceeded my expectations so far. Keep up the good work.
Persuasion. Getting people to care about it.
I thought persuasion was when you convince someone of something. It's often an additional step after informing them of what you think is true.
In my experience, Persuasion is literally the easiest step.
I'm not convinced. Yud's book came out months ago and it seems many important people have read it. I would think if the persuasion rate (of the top people) was not very low, things would look much different now.
Let me raise an inconvenient world for you: there are two types of actions in politics, fake and real. For fake actions such as signing a vague statement, attention/information might suffice. For real actions such as passing a law that will prevent lots of money from being made, persuasion of the underlying worldview is necessary.
I happen to think we do live in this world 🙂
I happen to think we do live in this world 🙂
Hindsight is 20/20! We do live in the world where we already have the statements, and still not the laws, so I don't think it commits you to much!
But if you have a non-trivial understanding of "real" (ie, something beyond "it's hard / hasn't been done yet"), I'd love to hear more about it.
Ultimately, my goal for AI Policy org pipelines (ControlAI or not) is to build an incremental sequence of publicly verifiable political actions of increasing difficulty and importance, in a way that respects basic deontology.
I expect any non-trivial understanding of "real vs fake" would help me with designing such pipelines.
Thank you for the work you are doing and for identifying The Spectre. I have felt something similar for a while now.
Good post & efforts!
I glanced at the Compendium, which obvs needs some updating. Also, the opening seems odd:
Ideologically motivated companies are racing to build smarter-than-human AIs. Big Tech already backs them, and now nation states are getting roped in too.
Are they really/primarily ideologically motivated (rather than financially), and with what ideology?
‘Big Tech already backs them’ - but these AI developers are big tech companies (or fast-growing ones), surely?
So if I can suggest with some introspection (not that I'm particularly involved in AI either way except as a user - I don't work on that) about possible reasons why this kind of mindset exists:
a part of my mind finds the arguments for AGI/ASI doom quite solid and believable. I am skeptical of classic Yud-style fast takeoff/FOOM scenarios but that does not mean I think that letting AI control civilizations would end well.
But another part sees how many people think this sounds patently absurd, impossible and crazy and has to wonder, well, what if I have indeed a completely skewed perspective and this is absurd, impossible and crazy? And my main problem with this is that any measures I can think of to thoroughly stop development of AGI would require some kind of control on hardware, software and possibly individual privacy that I am also very uncomfortable with, right in a time in which I am growing more and more suspicious of what governments all across the world seem to want to do to technology and the internet. So the thought of helping that along for no good reason is very repulsive to me.
Basically, I see that on expectation there's a path that yes, is probably the least bad. And rationally is the one that's possibly sensible to pursue. But also that I may be wrong, and I simply can't much muster enthusiasm for pursuing it because given the current premises and political direction of the west I feel like it will also result in a world that I hate to live in. And as often see politically, while "lesser evil" is a reasonable thing to endorse, it ends up still losing because it still can not rouse anyone's enthusiasm.
I wonder if PauseAI's efforts are analogous to what you do here? It seems like they have been mostly grassroots-centric, but the upcoming PauseCon US seems like an attempt to speak directly to politicians (or their staffers).
Have you talked to each other? It seems like there could be some very fruitful exchange between the two movements, if not.
When considering if, “That’s only relevant in the U.K,” my mind goes to evidence of real corrective actions taking place here in the U.S. from the killings of two U.S. citizens by agents of the Dept for Homeland Security. The ICE and DHS have been taking action in American cities for many months without meaningful opposition in Congress. It took high profile, on-video deaths to pressure pols. The Democrats are blocking funding to the DHS in a move that’s largely symbolic, but at least it’s something.
Clearly, we don’t want to wait for an extinction event to spur action, but when considering DIP #2: inform every relevant person in the democratic process, have you considered presenting a few perfectly believable and highly specific scenarios that could lead to a disaster? For example, the U.S. is currently preparing for a potential attack on Iran. From today’s Washington Post:
"In January, [Secretary of Defense Pete Hegseth] said that 'speed wins' in an AI-driven future, and he has ordered the Pentagon to unblock data for AI to train, while pushing the department to move from 'campaign planning to kill chain execution.'”
Could you lay out for politicians in very clear terms how that could lead to a doomsday result?
I’m the originator behind ControlAI’s Direct Institutional Plan (the DIP), built to address extinction risks from superintelligence.
My diagnosis is simple: most laypeople and policy makers have not heard of AGI, ASI, extinction risks, or what it takes to prevent the development of ASI.
Instead, most AI Policy Organisations and Think Tanks act as if “Persuasion” was the bottleneck. This is why they care so much about respectability, the Overton Window, and other similar social considerations.
Before we started the DIP, many of these experts stated that our topics were too far out of the Overton Window. They warned that politicians could not hear about binding regulation, extinction risks, and superintelligence. Some mentioned “downside risks” and recommended that we focus instead on “current issues”.
They were wrong.
In the UK, in little more than a year, we have briefed +150 lawmakers, and so far, 112 have supported our campaign about binding regulation, extinction risks and superintelligence.
The Simple Pipeline
In my experience, the way things work is through a straightforward pipeline:
At ControlAI, most of our efforts have historically been on steps 1 and 2. We are now moving to step 4!
If it seems like we are skipping step 3, it’s because we are.
In my experience, Persuasion is literally the easiest step.
It is natural!
People and lawmakers obviously do care about risks of extinction! They may not see how to act on it, but they do care about everyone (including themselves) staying alive.
—
Attention, Information and Action are our major bottlenecks.
Most notably: when we talk to lawmakers, most have not heard about AGI, ASI, Recursive Self Improvement, extinction risks and what it takes to prevent them.
This requires briefing them on the topic, and having some convenient information. The piece of evidence that I share the most is the Center for AI Safety’s statement on extinction risks, signed by CEOs and top academics. But it’s getting old (almost 3 years now) and the individuals involved have been less explicit since then.
There are arguments in longer form, like the book If Anyone Builds It Everyone Dies. But getting lawmakers to read them requires grabbing their Attention for an even longer duration than for a briefing.
Finally, once lawmakers are aware of the risks, it still takes a lot to come up with concrete actions they can take. In a democracy, most representatives have a very limited amount of unilateral power, and thus we must come up with individualised Actions for each person to take.
—
I contend that AI Policy Orgs should focus on
1) Getting the Attention of lawmakers
2) Informing them about the ASI, extinction risks and the policy solutions.
Until this is done, I believe that AI Policy Orgs should not talk about “Overton Window” or this type of stuff. They do not have the standing to do so, and are self-defeatingly overthinking it.
I recommend to all these organisations to take great steps to ensure that their members mention extinction risks when they talk to politicians.
This is the point behind ControlAI’s DIP.
Eventually, we may get to the point where we know that all politicians have been informed, for instance through their public support of a campaign.
Once we do, then, I think we may be warranted in thinking about politics, of “practical compromises” and the like.
The Spectre
When I explain the Simple Pipeline and the DIP to people in the “AI Safety” community, they usually nod along.
But then, they’ll tell me about their pet idea. Stereotypically, it will be one of:
Coincidentally, these ideas are about not doing the DIP, and not telling lay people or lawmakers about extinction risks and their policy mitigations.
—
Let’s consider how many such coincidences there are:
This series of unfortunate coincidences is the result of what I call The Spectre.
The Spectre is not a single person or group. It’s a dynamic that has emerged out of many people’s fears and unease, the “AI Safety” community rewarding too-clever-by-half plans, the techno-optimist drive to build AGI, and the self-interest of too many people interwoven with AI Corporations.
The Spectre is an optimisation process that has run in the “AI Safety” community for a decade.
In effect, it consistently creates alternatives to honestly telling lay people and policy makers about extinction risks and the policies needed to address them.
—
We have engaged with The Spectre. We know what it looks like from the inside.
To get things going funding-wise, ControlAI started by working on short-lived campaigns. We talked about extinction risks, but also many other things. We did one around the Bletchley AI Safety Summit, one on the EU AI Act, and one on DeepFakes.
After that, we managed to raise money to focus on ASI and extinction risks through a sustained long-term campaign!
We started with the traditional methods. Expectedly, the results were unclear and it was hard to know how instrumental we were to the various things happening around us.
It was clear that the traditional means were not efficient enough and would not scale to fully and durably deal with superintelligence. Thus we finally went for the DIP. This is when things started noticeably improving and compounding.
For instance, in January 2026 alone, the campaign has led to two debates in the UK House of Lords about extinction risk from AI, and a potential international moratorium on superintelligence.
This took a fair amount of effort, but we are now in a great state!
We have reliable pipelines that can scale with more money.
We have good processes and tracking mechanisms that give us a good understanding of our impact.
We clearly see what needs to be done to improve things.
It’s good to have broken out of the grasp of The Spectre.
—
The Spectre is actively harmful.
There is a large amount of funding, talent and attention in the community.
But the Spectre has consistently diverted resources away from DIP-like honest approaches that help everyone.
Instead, The Spectre has favoured approaches that avoid alienating friends in a community that is intertwined with AI companies, and that serve the status and influence of insiders as opposed to the common good.
When raising funds for ControlAI, The Spectre has repeatedly been a problem. Many times, I have been asked “But why not fund or do one of these less problematic projects?” The answer has always been “Because they don’t work!”
But reliably, The Spectre comes up with projects that are plausibly defensible, and that’s all it needs.
—
The Spectre is powerful because it doesn’t feel like avoidance. Instead…
It presents itself as Professionalism, or doing politics The Right Way.
It helps people perceive themselves as sophisticated thinkers.
It feels like a clever solution to the social conundrum of extinction risks seeming too extreme.
While every alternative The Spectre generates is intellectually defensible, they all form a pattern.
The pattern is being 10 years too late in informing the public and the elites about extinction risks. AI Corporations got their head start.
Now that the race to ASI is undeniable, elites and lay audiences alike are hearing about extinction risks for the first time, without any groundwork laid down.
Conclusion
There is a lot to say about The Spectre. Where it comes from, how it lasted so long, and so on. I will likely write about it later.
But I wanted to start by asking what it takes to defeat The Spectre, and I think the DIP is a good answer.
The DIP is not clever nor sophisticated. By design, the DIP is Direct. That way, one cannot lose themselves in the many mazes of rationalisations produced by the AI boosters.
In the end, it works. 112 lawmakers supported our campaign in little more than a year. And it looks like things will only snowball from here.
Empirically, we were not bottlenecked by the Overton Window or any of the meek rationalisations people came up with when we told them about our strategy.
The Spectre is just that, a spectre, a ghost. It isn’t solid and we can just push through it.
—
If reading this, your instinct is to retort “But that’s only valid in the UK” or “But signing a statement isn’t regulation”, I would recommend pausing a little.
You have strong direct evidence that the straightforward approach works. It is extremely rare to get evidence that clear-cut in policy work. But instead of engaging with it and working through its consequences, you are looking for reasons to discount it.
The questions are fair: I may write a longer follow-up piece about the DIP and how I think about it. But given this piece is about The Spectre, consider why they are your first thoughts.
—
On this, cheers!