I think you are correct to point out that open-weight models and general proliferation of increasingly sophisticated ML engineering knowledge is a latent risk of a type which governments have a known playbook for even at the current level of proliferated capabilities.
Governments are known to ban or monitor & intervene in the dissemination of various instructive materials. An example which would be easy to demonstrate would be instructions for synthesis methods of various types of controlled substances.
It was once relatively harder to find a meth recipe online compared to, say, a niche recipe for a cuisine with a much smaller market than stovetop meth - and it is relatively harder than that to find a pipe-bomb recipe. Especially if you accounted for those recipes by quality.
LLM weights are containers of structured information which require varying amounts of work for people to elicit, depending on the specific model, what has been done to it, and the way those weights are being queried.
You can now find, very easily, artefacts which contain unprecedentedly good information on how to make meth and bombs. The anarchist cookbook does not contain a troubleshooting interface which can be searched in a variety of ways in order to determine that you are losing 5% of your yield and contaminating your final product because you have a burner nozzle turned all the way up during a step that needs something more like a 1 inch flame applied to the mixture, given your crucible material and design.
I once had a fun conversation with someone about which Science YouTuber would be the biggest disaster if Utility Function Flipped. After one honorable mention which didn't fit the premise in spirit, I chose The Thought Emporium - a DIY biohacker guy. He just happens to be The Guy Who Does Bio among the popscience YT pantheon like NileRed, styropyro, Explosions & Fire.
The conversation topic is a bit silly, these people are selected for making appealing content, not competence in a dangerous field, and I think my answer is plausibly wrong by my own estimation if I just went hard on identifying someone who's got what looks like the highest INT stat & is the most conscientious about helping reduce harms in the world. There's probably a science youtuber who does pretty good whitehat cyber-security, who when Utility Flipped would make the blood bank refrigerator thing look like a cute script-kiddy stunt - while The Thought Emporium gets Utility Flipped and ends up making a trip to a population center and giving a bunch of people really bad food poisoning.
Trying to argue why one or another would be worse starts being the kinda exercise that elicits dangerous outputs from creative & knowledgeable humans, given how the mess we have instead of utility functions sometimes causes us to behave inconsistently with how we'd want to trade off between 'Being Clever And Interesting In A Conversation' and 'Not Ideating & Spreading Exfohazards That Get People Killed.'
LLMs are a nightmare-come-to-life version of this fun & spicy conversation premise. Which open-weight model will cause the most harm in the next year as people figure out how to do increasingly sophisticated things to modify them to various purposes which interact in only occasionally predictable ways with the dangerous capabilities they contain - especially when the predictable ways of steering their outputs will occasionally be used to the ultimate purpose of greater harm?
This all pales in comparison to the type of harm I expect to follow immediately after labs manage to produce one of these artefacts which possessed capabilities in a type of domain that's even more dangerous than bio - even closed & alignment trained & monitored to the best of their ability.
One possible doesn't-quite-qualify honorable mention from that earlier conversation about "who's the worst science youtuber to Utility Flip" is someone like Eliezer. He does (as far as I can tell) a good job of deliberately avoiding, even by accident, giving accidentally-useful hints towards things he's identified as exfohazardous. Even a Utility Flipped Eliezer who doesn't get much more productive via valuing the opposite of "not ruining his own life to save humanity" would still, I'd guess, manage to careen good chunks of the planet into a very short and intense nightmare before they got eaten by an ASI... Though maybe a Utility Flipped Eliezer wouldn't accelerate the advent of ASI, given he'd be trying really hard to get labs to a near-miss on alignment that lands us with an ASI which 'I Have No Mouth's the lightcone. Corrupted!Yud would still have the convergent instrumental goal of steering the fate of distant galaxies.
I think the thing where Eliezer is the guy who's good at thinking about mind design & also this guy who I'd be really scared of being dangerously-misaligned is correlated.
Which is why I think that making AIs good at AI research & alignment is way worse than open-weight LLMs as a risk. I think that's a domain where you run into things showing up in the models where you no longer need the extra step of someone taking the weights and prying out the dangerous stuff - the presence of some short list of capabilities in that domain is just a thing that kills your planet given even naive but well-intentioned attempts to interact with that artefact - including the interactions implicit in the process of making it when you can't halt your contact with it at the precise moment it acquires a sufficient number of those capabilities.
Which is to say, I am not especially worried about the risk of too much Government intervention here. Just Government intervention that's insufficiently well informed about ALL of the dangers of this technology.
I am worried about misuse of open-weight LLMs because it will pull attention from decision makers to a phenomenon which may implicitly give them a bad picture of what this technology is.
Similar to how The War On Drugs has seemed to create a regulatory environment which has counterproductive & silly effects on things like medical research, where Governance is being shaped by interactions with people who don't have degrees is drug development.
Like that example, I'm worried that a bunch of attention is going to be drawn to people with home GPU servers & various misanthropic neurotypes - and will develop blinders or get sidetracked from the problem of big labs making something that suddenly wrenches the future out of our hands, including whether or not the arrangement of carbon atoms in our bodies gets to persist for another year instead of getting nanomachine-recycled into some accessory to computronium.
I suspect our conclusion is the same if I phrase it this way? We should really try to avoid a bunch of harm being done by proliferation of increasingly dangerous open-weight models, and simultaneously make sure people have better awareness of exactly what kind of risks this technology poses.
I just have a risk-model where it only makes sense to engage with the risks discussed in this post in ways where it seems most helpful to the project of stopping the race to ASI & buying us time to try to find a way out of the imminent death we've been approaching.
To that end, doing our best with describing & predicting lesser risks accurately is probably helpful. "Local Validity As A Key To Sanity and Civilization" and all that.
In a nutshell
Point 1: Safety isn't backdated on earlier models, todays Fable 5.2 will be trivial for someone in 2035 to jailbreak, but some capabilities will be just as deadly in 2035 as they are today, i.e. anything biological.
Point 2: If nukes were available on every corner to every Jill and Joe, we'd be in ruins. Reactive, legal enforcement is not sufficient.
Point 3: The fuzzy paradox where you have access to immense capabilities but minimal proactive safety mechanisms might be a sign we are living in a transitional period on AI safety.
Point 4: Infufficient progress is being made in model safety, the clock runs as the probabilities a small group of idiots do something terrible increases.
Note: it doesn't need to be Fable 5.2, but an open source equivelent that appears next year.
Old Models can be dangerous too, (or they soon will be)
When I was playing about with old LLM models, it occured to me how easy they were to jail break.
That led me to an obvious conclusion, "In 5 years time, will Fable 5.2 and GPT 6.1 be equally as easy to jailbreak?"
Even if vendors do a good job, it wont be long until we have open source equivalents. So be it in 2 years or 7, they'll be cracked and 'on the streets'
So, 'How much damage could Fable 5.2 or an equivalent do?'.
Some wreckless guessing:
- One shotting a nations Air Traffic Control systems , causing hundreds of planes to crash on a single day. (Purely speculative based on fragility and age of ATC systems)
- Bio Weapons
- Dirty bombs
- Mass EV carjacking and car accidents
...etc.
Safety Mobilization is too slow and ignores proliferation
In terms of available budget, Anthropic invested 6% of it's AI R&D computeinto safety.
Which on the positive ledger, produces consistent compounding effects. Today's security and alignment are orders of magnitude more sophisticated than what we had for GPT4.
It's not exactly screaming caution, especially given heads of safety walking every other Tuesday.
Recent safety talks with the Trump Admin and the heads of tech, Dario, Musk, Zuckerberg yielded an underwhelming AI voluntary safety accord. The Center for Humane Technology have put out several shorts from experts lamenting lack of progress in other nations too on real tangible mechanisms to keep the genie in the bottle.
I've often heard things like, 'well we had nukes and we learned to control our better nature'. I think that's fair on the level of nation states, but even then every few years the world experiences collective anxiety during periods of sabre rattling.
For millions,billions of individuals, the unlikely becomes the innevitable.
So, AI alignment aside, the clock keeps ticking, the probability of malicious use is not static but increasing, yet our beurocratic nature continues to prevail.
No one's stopping the bus
In my experience, working in various big companies over the years - bringing up risks with leadership and managers, usually gets met with a form of ritualistic haggling.
It's not like the CTO is going to be like "OMG, you've found a potential critical vulnerability, STOP THE BUS!".
It's something more like, "ah yes, Frank already raised that conern, sync with the cloud team." Which turns out is nothing like that concern at all, but you've got a few pennies, and a bit of rope so you do the best you can with what you've got.
So at absolute best, yes, we get some kind of gentlemans handshake between the US, EU and China to allow inspectors in each others labs and reduce the pace of development. But at that level of granularity, it's focusing on say harmonising GPT 8 or whatever, not retrospectively locking down Hugging Face, Github etc.
So it just happens?
So far, in my mind yes. I'm not seeing any evidence of retrospective safeguarding, and if Anthropic's scaremongering is to be belived with Fable then those capabilities may already be here.
Anthropic identified a Yemen weapons cell using Claude for rocket/missile guidance assistance, caught it when the missile failed and they went back to it for trouble shooting... Open source models don't get monitored when they run locally.
Almost every other day you read about someone getting arrested, either before or after the deed and their interactions/questions to chatGPT being brought up.
It's not hard to imagine a small group of determined bad actors in 2035 taking an advanced 26 model, running them in parallel to perform something heinous.
Unlike devastation from AI-misalignment, the probability of deadly misuse by the public continually rises in the absense of prohibative restrictions.
The real question is - what could they do with today's best model if it's fully jail broken in a couple years time.
AI could be set back in ways it could never recover
If catastrophic misuse accurs, on a scale that we can recover from, that would be the wake up call for serious, rapid and possibly even harsh curtailing of AI usage in the public domain.
Much like in IT, you don't get the cyber budget you've been asking for until you make the local news.
I'd go as far to say, the counter reaction to an AI disaster could be immediate, and extremely brutal. It would be the ammo Anti-AI artists, developers, musicians have been looking for. It could be the vehicle for vast, comperhensive curtailing;
"well, whilst we're here, maybe we can protect people's livelihoods too?"
and before you know it, not only are all models locked down, but AI usage isn't just 'metered', it's restricted and even illegal for things that today we consider mundane.
By failing to regulate AI appropriately now, we might eventually provoke a crisis that results in a crackdown, limiting our AI usage that didn't need to happen.