I think you may or may not get any safety in the sense you mean it, but you will definitely get massive concentration of power, which is dangerous.
Do you think that this proposal would somehow make the problem worse? We already have concentration of power today.
If you give substantially less restricted access to people who have shown themselves "trustworthy", you have to look at how "trustworthiness" is going to be assessed. In any realistic case, the answer will be that the people considered "trustworthy" will be those associated with, or at the very least vouched for by, already powerful "Establishment(TM)" organizations. That's exactly how Glasswing did it, and it's pretty much the essence of every proposal I've heard from Anthropic or anybody else.
You go from a bad situation where "The Labs" control the frontier models, but at least grant a reasonable degree of access to more or less anybody, to a worse situation where access fundamentally depends on existing power, and of course makes it easier to get more power. Power increases for the favored, who are already powerful, and decreases for everybody else, even if some of those excluded are also somewhat powerful at the moment.
How could it not make concentration worse?
As for whether you get enough safety to justify the concentration of power, well...
Say you're worried about bioweapons uplift. The people with the most motivation, and arguably the most demonstrated propensity [1] , to use bioweapons are governments, yet they and their contractors will have no trouble passing the vetting, at least if they're the "right" governments.
Same for computer security [2] . Both public and private entities are in the business of breaking security at scale. Many of both will also pass... whereas many of their targets will not, and therefore won't get any help in defending themselves.
Not to mention that even though the criteria tend to get cast in terms of organizations, it's still individuals who formulate the actual queries, and organizations often don't have as much control over their affiliates as they think they do. So you actually have two layers of failure: the organizations you trust may betray you, or the individuals inside them may betray both them and you.
You can probably exclude a huge number of obvious kooks using the kinds of filter they've been deploying, or indeed many kinds of filters. But even with AI uplift, obvious kooks have a relatively low chance of executing anything successfully. Actors capable enough to make effective use of what you offer are going to be a lot harder to catch. I'm not saying you'll catch none at all, but you'll do it at the cost of deterring a lot of legitimate defensive work, and that gets worse the tighter you try to make the net.
The basic problem is that you have no really reliable signal about who the "good guys" are. The people with the best chance of gaming any system you do set up are (a) the powerful (so you get further concentration), and (b) competent and motivated attackers. Defense is actively disadvantaged. If you're an attacker, attack is what you do; you specialize in exactly the things being restricted. You're motivated to either legitimately pass the checks, or find a way around them. On the other hand, defenders aren't necessarily specialized that way, and often have other things to think about. They're less likely to jump through hoops to get approved, and less likely still to put in the work to get around restrictions.
If you get a net safety benefit, I think most of it will actually just come from reducing the absolute amount of use your model gets. Which you can of course do better by just witdrawing it entirely.
Footnote added on edit: I feel like I should mention that nobody, government or otherwise, has actually show that much propensity to use bioweapons. That's probably because they're crummy weapons that you can't target or control. But at least if you're a government, you can maybe hope the plague will stay in the other government's territory, which is geographically separate from yours. If you're a non-government, you're more likely to be physically colocated with your enemies. Even if you have some kind of obsession about race or whatever, your targeting gets much harder. I don't think anybody, including LLMs, actually knows how to target a given race with a bioweapon, and even if you did, you'd be one mutation away from off-target effects. So you basically have to be a total lunatic to even try. Which, again, means your chances of succeeding are going to be bad, because lunacy tends to have far-reaching effects on competence. ↩︎
Sorry, I'm old enough that "cyber" still reeks to me of clueless Beltway arrivisme, and I won't use it. ↩︎
Quite bluntly, have you actually read the proposal and compared it against our current situation? Are you seriously worried Anthropic is going to continue their nefarious consolidation of power by... looping in trusted partners to double-check their safety results?
I am suggesting a short period of early release access. We have already seen from the Mythos release that this is actually sufficient to patch critical software and ensure the release is less publicly disruptive. I have no clue how a few months of early access results in a serious "imbalance of control" - at worst, you can trivially fix this by rotating what groups get early access, neh?
Equally, if you're worried about betrayal, surely that makes it better to have a trusted group of a few hundred people, rather than "the entire population of Earth"? It is quite obviously easier to monitor a few dozen organizations for malicious usage and bad actors, versus the entire public population.
We have somehow survived the problem of vetting who the "good guys" are despite the attackers incentives for centuries. It's the basic concept of security clearances.
If you get a net safety benefit, I think most of it will actually just come from reducing the absolute amount of use your model gets.
That is part of what I am suggesting, yes. Conversely, "withdrawing it entirely" gets you absolutely zero new information, so I'm not sure how that's possibly a fair comparison.
Perhaps I failed to communicate something clearly, but this all reads like a ridiculous strawman of my actual proposal, which is simply "we should have independent organizations involved in mundane auditing, so that more people are aware of capabilities before they become public, and we can involve the relevant security teams"
P.S. Cyber has been a common professional term for decades so maybe save the linguistic snobbery for your own personal blog? I don't see what purpose is served by spelling out your personal distaste with one particular word I used, and I struggle to read it as anything other than an insult when you say it "reeks of clueless". You could have been polite, or just cut the footnote entirely.
Edited to Add: Okay, I reviewed my post and comment and I don't mention "bioterrorism" anyway so I'm now extremely convinced you've confused my post for someone else's - why are you writing an entire footnote about something I never brought up?
Thinking about "Plan A" makes me want to make concrete proposals towards those goals.
I think Anthropic's "Project Glasswing" provides a clear and easily implemented first-step policy towards AI safety. With a few small tweaks, I think we can build a release process that is robust against today's mundane threats, while also building transparency and track records to guide future policy decisions. I think this policy can also strike a favorable balance of interests between the labs, government, and public safety.
This program has already paid off - we knew in advance that access to Mythos led to a concrete, graphable spike in cyber-security capabilities. That helped build broader awareness of the mundane dangers presented by current frontier models, without anything hitting the fan. This sort of "early access" program provides solid value for a relatively small amount of effort, since most of the pieces are already in place.
Phase 1: Transparency
I think the best place to start is with transparency, not control.
Get the major labs and the government to agree on a standardized group that has early access to new models. Project Glasswing already provides a decent starting point, if perhaps biased towards Anthropic's interests. Have each of these organizations issue a regular report, indicating whether they think the model is safe yet. Prediction markets can have fun using these reports to try and extrapolate a release date, as more and more organizations begin to sign off on the model.
Each organization has its own discretion on where it focuses its efforts, and what constitutes "safe". Early access users are purely advisory - they have neither authority nor liability. But each report builds up a public track record. That helps us identify each organization's unique forecasting strengths and weaknesses, and then labs (and future regulators) can weigh those reports as they see fit.
Phase 2: Formalization
Initially, labs might retain full veto control, as governments build their own formal approval processes. We can flexibly tighten or loosen oversight based on how capabilities evolve. Organizations like the CIA and NSA might do their own private evaluations. We could require a super-majority of early-access users to approve releases. Alternately, we could formalize how well systems resist "red team" efforts using the same models. Eventually bug reports drop back down to a trickle. Eventually we stop finding dangerous capabilities. And if we don't hit that "eventually", it's lucky for us that the public never got access.
I think it matters a lot what incentives go behind a veto, which is why I emphasize advisors and transparency at this stage. Initially, we want stewards who have a vested interest in both the economic and security side of the equation. Otherwise, a lot of powerful stakeholders have an incentive to oppose all of this. The Government naturally wants to see the economy flourish, and has practice balancing that against security concerns. The labs want to preserve their reputation and not invite stricter scrutiny, but also directly profit from releasing new models.
As capabilities grow more dangerous, it makes sense to invoke more security-minded stewards. That's outside the scope of this proposal, but probably an important follow-up. We should work to establish credible thresholds on when to tighten our safeguards. As evidence for dangerous capabilities accumulates, the political cost of opposing safety rises. Hopefully, by the time more extreme actions are necessary, there's also a clearer public consensus on the necessity of such actions. "Raising the sanity waterline" on the threat of ASI will also be an important part of this effort.
Right now, I'm treating it as a positive that this doesn't slow down frontier development. The major labs merely face a small speed-bump to release, and that speed-bump provides an extra incentive to ensure models have robust safeguards in place. They have very little reason to oppose either phase of this, and I think public opposition to such mild measures would cost them valuable political capital. Anthropic is already voluntarily doing most of this, and I hope other labs would have done similar in the same position.
This is just Step 1
I want to be clear: The risk factors explicitly addressed here are all mundane harms from a dangerous LLM, not stopping ASI.
But, as risks evolve, it lets us leverage a well-established information ecosystem, and starts to build political momentum towards stronger regulations.
In short, this is the foundation, in case we need "Plan A", while also trying to build a coalition with those whose concerns are more mundane, or who are worried about the economic impacts of more extreme actions.
If you accept that action is urgent, then I think it follows that it's important to have a concrete proposal that is tailored to the current political and mainstream levels of awareness. "ASI" still sounds like exotic science fiction, which forces us to start small and focus more on "if" branches than "when".
Right now we desperately need action that can meet skeptics halfway. I'm expecting the singularity, but most people aren't. That's why I emphasize how this addresses a lot of *other* concerns, and de-emphasize how this is a foundation for everything else.
Another Useful Foundation
I'd also like to throw my support behind Plan A's policy of tracking and verifying compute. Having a proven technology in place would also be an advantage in negotiating international verification protocols. Plus, even while we're just tracking and verifying internally, that helps keep frontier labs honest.
That said, I don't think I'm the ideal person to write that proposal - I'd just be editing down the concepts in "Plan A". I would love to see someone more familiar with the technical details work up a complementary proposal, with the same focus on mainstream appeal and building political momentum.