After the odds had dropped quite a bit, they’re looking good again, with a 60% chance of restoration by July 1 and 88% by July 31, in the wake of groundwork looking like it is being laid in various places:
leo: BREAKING: Claude Code v2.1.190 introduces several string changes that hint at preparations for a Fable 5 return, with it being permanently included in subscriptions with weekly usage.
The string “You’ve used your Fable 5 usage for this week” has been added, and “purchased separately from your plan” has been removed
leo: UPDATE: Fable 5 has now reportedly also reappeared in Amazon Bedrock
If the update is based purely on the above info I would treat the new odds as overconfident. These moves seem reasonable to make even if you have no confidence in the restoration, in order to be ready if that moment arrives.
This also suggests a potential permanent quota for Fable for subscribers. Even a modest amount is a big game here, since even a modest allocation means you can use it for non-coding tasks or minor coding tasks within the subscription.
With that on the horizon, it seems time to update our understanding of what has happened and is otherwise likely to happen, and how to think about all this.
A Rather Terrible Policy
Even if access is restored in the coming week as expected, this was a fiasco.
Dean Ball and I both invite you to imagine this situation in reverse.
Dean W. Ball: I invite you to imagine what various commentators and industry actors would have said if president Kamala Harris had export controlled grok indefinitely and for no disclosed reason.
The chosen implementation was such a train wreck that not only did we piss off our allies and call into question the reliability of the entire ‘American AI stack,’ we also caused the NSA to lose access to Mythos.
Andrew Curran: The Trump administration is pressuring META to agree to submit its models to the government for voluntary review. META is now the only holdout, as OpenAI, Anthropic, Google, xAI and Microsoft have all agreed to these terms. Reporting by the NYT.
Meta says they are ‘working through the details and hope to sign an agreement soon’ as per Meta spokesperson Francis Brennan.
The People Have Spoken
So say we (almost) all.
The good news is, these decisions are not up to a public vote, either of Twitter or otherwise. The public does not understand these issues. The public has no idea what it is talking about, and its decisions will be mostly uncorrelated with the right answer.
Thank You, Next
The public does not get to use Fable 5 until further notice.
That does not mean the labs are going to stop developing more capable AI models.
Andrew Curran: A new, more capable version of Mythos has emerged from training. I don’t know whether it will be called Mythos 5.1 or Mythos 6, or if Anthropic will keep it internal to accelerate further development – but it has arrived.
Stopping models like Fable 5 or Mythos 5 from being served to the public does nothing to slow down development. In fact, it probably speeds it up slightly by freeing up resources. There are also no rules preventing the labs from continuing to advance capabilities while any current model is under embargo – or from keeping progress quiet until they choose to release it. None of them can afford to pause or slow down.
We need only look at how capable GLM-5.2 is as proof of this. To protect their business models, the frontier labs must continually train increasingly capable systems to stay ahead of open source, and each other. The current continues to rage beneath the ice, and we continue to race toward our destination.
Andrew Curran: Original Mythos finished training February 7th. They have had plenty of time.
There was a brief period where Anthropic will have been somewhat slowed down by inability to use Mythos 5, internally, to train the next Mythos, forcing it to fall back on worse models. That period may already be over. I don’t think Curran would make this up without a source, but even if he claimed it purely on priors, those priors are not going to be that far off.
And if trying to release their best model gets it potentially restricted, Anthropic or OpenAI might well decide to keep it internal. Which is where the biggest risks are.
Inability to deploy, or uncertainty and delays around deployment, do modestly reduce the business case for pushing development, but not enough that you wouldn’t push maximally hard. There is no alternative, and the valuations of the companies make the related costs highly affordable.
Be Very Very Quiet
We will see if this prediction holds up.
Peter Wildeford (June 21, 7:07pm): The fact that we’ve gone eight whole days without any major AI news is kinda weird
Dean W. Ball: I’d assume the whole AI industry in America is effectively frozen from new public releases until USG resolves the Fable situation they have stumbled into.
Kevin A. Bryan: Frozen from releases but not from development, of course!
Dean W. Ball: correct. if anything development can maybe be faster
What These Babies Can And Cannot Do
It’s not news, but to remind us of the stakes: The full version of Mythos can hack basically anything complex that it wants to hack, if given sufficient information, time and context.
What it cannot do is hack into all the classified networks in a matter of hours without being given various other affordances an outside attacker would not have.
What actually happened was that a professional authorized red team, given physical access to air gapped systems, identified issues in our classified systems within hours.
Dustin Volz and Julian E. Barnes: In reality, the tests involved “red teams” of N.S.A. analysts who were using Mythos in a highly tailored environment that would be extremely unlikely for an adversary to replicate, officials said. The red teams began their tests within classified N.S.A. systems designed to be accessible only from certain computers and completely cut off from the broader internet.
To be clear, this was still impressive and exceeded expectations.
Mythos is the real deal.
Dustin Volz and Julian E. Barnes: The controlled tests proved impressive even within the halls of the N.S.A., a secretive fortress outside Washington that specializes in developing digital espionage techniques against foreign adversaries and protecting U.S. networks from cyberattacks.
… Still, even though the N.S.A. did not experience the doomsday scenario some had feared, analysts at the spy agency were stunned by how capable Mythos appeared to be in controlled test settings, which exceeded already lofty expectations.
A series of misunderstandings created quite the pull quote that Mythos ‘broke into almost all NSA classified systems in hours.’
Mythos provided a lot of uplift, and that first result is still a big deal.
But let’s not get carried away, as some did, including inside the government.
Bull Theory: The NSA’s own director says Mythos broke into almost all of its classified systems in hours [during a read team exercise].
Per The Economist, Senator Mark Warner, vice chair of the Senate Intelligence Committee, said General Joshua Rudd, who runs the NSA and the Pentagon’s Cyber Command, told him this directly.
This came out on June 11, the same day Amazon reportedly found a separate jailbreak in Anthropic’s models. Within hours, Trump ordered [Fable taken offline.]
IRIS C2: Let me help everyone put this story into perspective: The Pentagon, CyberCommand and NSA all have red teams.
The full time job of these red teams is to attempt to hack into classified systems. The people who work on these red teams (many of whom I know), are cleared to see the classified information that they might bump into along the way.
In the case of almost every exercise, the red teams begin with some kind of initial access to the air-gapped classified network.
The quote from General Rudd does NOT imply that any random Joe with access to Mythos could, from the outside world, gain initial access to these networks.
What he is saying is this: USG red teams went from having Cobalt Strike, et al to having Cobalt Strike et al + the uncensored version of Mythos—and, low and behold, they were more successful, and able to operate more quickly, once they had access to Mythos.
Here’s a real world example: In January, an NSA Red Team might know that the Cisco router within classified network was the subject of a recent CVE. Does that mean that the red team was in a position to write a reliable n-day exploit to target it? No. They usually would not have the time/budget to mess around with that. Does it mean that they could actually craft a high end Lua implant for the router? Also no.
Now they can do both. And they can do so relatively quickly. Thanks to Mythos.
Shashank Joshi: Shashank Joshi: This now widely circulated claim is based on a line I wrote last week. I accurately quoted Mark Warner, vice chair of the Senate intelligence committee, saying that the NSA chief had told him that Mythos “broke into almost all of our classified systems, not in weeks, but in hours”. But it would be a mistake to read that literally, I think. It surely depends on using Mythos alongside other tools under very particular conditions. I quoted it to give a sense of Mythos’ potency. But it was a mistake not to have added caveats.
An update. A US official tells me that Sen. Warner misunderstood the NSA director Gen. Rudd in this case. Rudd did use the ‘hours, not weeks’ wording, but the use of Mythos in this context was—as widely assumed—part of a red-teaming effort, i.e. testing the security of internal networks. The official also told me that the agency’s red-teams no longer have access to Mythos, because their authority for accessing it was under Project Glasswing.
Ed Tarnowski: Even if true, it’s not reason for panic.
Several open-source models have already proven capable, for example, of accomplishing the same Mythos discovery re: OpenBSD.
Bugs will continue to persist. Giving defenders access to frontier models lets them find and patch them now.
It took several days for us to sort out what actually happened here. The pull quote will probably remain one of those zombie debunked stories that keeps being cited by politicians, and that plausibly mattered in taking and keeping Fable offline.
Bull Theory unfortunately also repeats several other misconceptions, including the idea that Anthropic took Fable offline ‘instead of’ cutting off access to foreigners, when we have explicit confirmation from Bessent that the point was to force the models offline.
Connor Leahy: The recent Mythos/NSA warning shot [as in the pull quote] is looking more and more like how I expected a “warning shot” to look like.
An absolutely insane thing happens, and then the FUD machine kicks into action and adds infinity gazillion “””context””” until no one knows what is true anymore and everyone believes what is most convenient.
This, once again, shows why the theory of change of “we just wait for a wArNiNg ShOt and then everyone will suddenly be reasonable!” is flawed.
We need to do a lot of work to make the situation legible, and people reasonable!
What’s The Worst That Could Happen?
Teortaxes: A very tasty update package from Zvi
and yes… they’re beginning to understand. This is the whole damn point of “superhuman hackers”. To close the whole topic of hacking for good.
I have never understood the theory that if everyone had the same sufficiently advanced superhuman AI hacker, that this would favor defense over offense, given the ability to concentrate firepower on a given target, even ignoring the ‘most code will still not be superhuman for a while’ issue.
Teortaxes and others pointed to formal verification. I realize this exists, but so do social engineering, fraud and impersonation, and so potentially do other outside-the-formal-box forms of hacking the system, and formal verification of what matters for the larger system seems very hard and does not result in efficient code.
Teortaxes: This is indeed a very serious Q
Yes, I believe that, because formal verification exists. This is a bit different from vulnerability search, but the proof of concept is a general purpose 10T LLM that tops MathArena anyway. Fable or Fable+ could write *unbreakable* software.
securing even all web-exposed software is a superhuman task, obviously. but the main issue is labor cost. The endgame is that the weakest link in the attacked system is post-quantum cryptography and you’d need to disprove contemporary mathematics to get anywhere.
Glasswing isn’t that btw. Glasswing exists to make sure the US isn’t brought low by GLM 5.4 putting ransomware in every MRI scanner and iPhone. The current ecosystem is flimsy trash.
For how we even begin getting to that, read [The Great Refactor from IFP in 2025].
entirelyuseless: Non-superhuman coders have already written software they were unable to break with 100x effort (widely used cryptographic systems.) So evidently possible in general.
singularvessel: This is clearly doable for simple code, right? The question is just how the effort required scales with the complexity of the code.
NondescriptTransfer: Yes definitely. It’s kind of like chess, a win is you taking advantage of the opponents detour from perfect play. Which is why you see more and more draws at higher levels.
Security research is the same way, except the defender wins draws. As capabilities improve we’ll approach an asymptote of secure code. Also if you look at industry trends, code in general is far more secure than it used to be. AI will only accelerate this.
But yeah it all comes down to every hack is traced back to someone doing something dumb at some point and the hacker being the first one to catch it. As AI gets smarter and does dumb stuff less often, and there are more eyes on the code (from defender running AI) you’ll see fewer
A fun aspect of chess is that people mostly do about as many ‘dumb things’ no matter what level of play, from the perspective of that level of play, because as you improve your play more things count as dumb. In theory perfect chess is possible, but we keep seeing the AIs get better beyond where I thought perfect play might be. My model of security is similar.
My understanding is that if your code wants to do very well-defined simple things, meaningful formal verification is within reach. But many things are not so simple.
You do distinctly have to avoid there being ransomware in every MRI scanner and iPhone. That sounds quite bad.
Arthur Tellis (QTing Teortaxes): Yes, but I think there are ways to soften these tradeoffs: for example, one could limit Fable’s use for vuln research/security engineering to only software that it generates for a given user or can verify as belonging to a given user. This probably isn’t a sustainable equilibrium, but it biases these systems’ use toward hardening rather than vulnerability research in the short run.
I do not think you could pull this off, but even if you did it would be a horrendous pain in the ass, and would rule out use of Fable on a wide variety of the most valuable software projects.
In general though, I think that we are overrating the risk of this particular feature-bug. If, within the Glasswing construct, IT majors have ungated models, higher token budgets, and more legible/tacit context about any given codebase, they should be able to identify vulnerabilities faster than vulnerability researchers with Fable access. Moreover, KYC and classifier-driven observability of Fable use for security engineering/vulnerability research should act as at least modest barriers to entry for offensive researchers.
The strongest argument here is data retention, some amount of KYC plus classifier-driven observability as the real defense.
As in, if Anthropic wants to figure out who is using a lot of Fable tokens for major cybersecurity related operations, that is something they can definitely do, and they can associate it with your reputation.
This is not foolproof, it does not fully ‘fix’ the ‘jailbreak,’ but it is part of a defense-in-depth series of obstacles that greatly raises the de facto difficulty, risks and costs of offensive operations.
Do enough of that, as I expect Anthropic did, and the balance of impact of Fable shifts so that you’re safer with it released than not released.
But the real point is that we’re overrating the importance of 0-day exploitation in computer network operations and the peril of computer network operations in general within the IT economy. The vast majority of cyber operations do not rely on 0-day exploitation, which can be mitigated against in general through effective monitoring, information domain separation, etc.
The name of the game is effective exploitation. That requires a chain of exploitation. Mythos was special, as I understand it, in large part because it could on its own find and complete such chains, often in unexpected ways.
The cost of cyber espionage/attack worldwide, meanwhile, is less than a single basis point of the total value produced by the IT economy and likely less than a single percent of the value produced by democratized machine intelligence. We are witnessing excessive securitization of AI model release on the basis of this omni-use technology’s immense capability in program analysis and security engineering-vulnerability research.
This is where Arthur loses me. The value of cybersecurity is not ‘less than a basis point of all IT value.’
The value of cybersecurity is all of the value of all IT, period, in the sense that if your cybersecurity fully fails then you have no IT, or might be even worse off than that.
Tail risk management does indeed have to take priority sometimes. That does not need to involve ‘AI might get everyone killed.’ This is the same as saying ‘pandemic defense is a small percentage of the national budget’ as a reason not to worry about bioweapons. No matter the right level of worry about bioweapons, that argument is a very bad reason to not worry about bioweapons.
The question is not how much typically goes wrong in cybersecurity. It is what goes wrong in a world where cybersecurity collapses. What happens if they breach our national security systems? If hackers penetrate almost any corporate network? If our hospitals and other critical infrastructure can’t stay online? If you can’t do a variety of things online anymore because they are no longer secure?
Policymakers need to recalibrate. If AI is the transformative technology of our age, it is because its value in scientific production, software development, and labor augmentation; this value absolutely swamps the risk presented by patch reverse engineering, which is itself a malady swamped by the benefits of more effective patching. AI is machine intelligence: literally general-purpose capability embodied in complex software. To deprive the world of intelligence because of the risk of patch reengineering is to surrender to extreme risk intolerance
I fear that we’ve built up a cybersecurity complex — including political, bureaucratic, think tanks, and media actors — that is extremely risk-obsessed and views the IT economy as subordinate to the cyber economy. This is an error, however: cybersecurity risk is a tax that we must pay to benefit from the collective intelligence generated via the IT economy. We cannot allow it to be an anchor, depriving us of progressively advanced intelligence.
The fire department considers basically everything subordinate to fire risk and to their own convenience. I think this is quite bad, because the fire risks being prevented are often absurdly small, a vestige of a time when a fire could burn down a city.
However, if things were different, and a repeat of the Chicago Fire or similar was a real possibility, then you can see why saying ‘oh the fire department is small expense why are they bossing us around on every little detail of our streets and houses’ does not make so much sense anymore.
When we say ‘cybersecurity risk is a tax that we must pay’ the correct question is to talk price. What is the size of this tax? Are you so sure it is worth paying?
You could also challenge that the alternative involves paying that tax soon anyway, as part of the calculation. That matters, too, and must be factored in.
The only valid justification that I can see for such a measure is if we had *specific* intelligence that *a specific* rival were trying to use Fable/Mythos as part of a *specific* vulnerability research program focused on the sort of devices whose exploitation would enable collection operations of sensitive operations at scale that we couldn’t easily mitigate: the AWS Nitro hypervisor or an Owl cross-domain solution, for example.
While it’s possible that the USIC possesses such intelligence, it seems like this policy was improvised and made on the generic appreciation of the risk of rivals’ attempted use of these systems rather than a specifically calibrated threat and risk model. If such intelligence existed, it would presumably inform a more time-bound and focused mitigation regime rather than the export control policy that was promulgated last week.
Yeah, that’s not how catastrophic risks and existential risks work. People need to accept this. You do not get to wait until you know exactly who is going to do what, where, how and when before you act. If you want for that, and the moment does arrive, you will be too late.
You don’t get to say ‘oh, now I see the one particular system they’re trying to hack that we can’t let them hack’ because the way you find out is that they already hacked it.
The Data Retention Policy Is About Defense In Depth
If you want to stop abuses of a powerful AI model like Fable, you have to do some amount of data retention, so you can figure out, including retroactively, when someone is up to no good, and respond. This is part of any sane defense-in-depth strategy, and rather costly given many enterprises cannot accept it.
If you make a rule limiting AI frontier model development, as we may have to at some point, you need an agreement with robust monitoring and enforcement mechanisms. There will be great temptation to cheat.
If you make a rule limiting AI frontier model deployment, against making it available to the public, you do not do a great job protecting against existential risk, but you do not need such aggressive monitoring, as it is very easy to see what is or is not available to the public, it is and relatively easy to enforce by taking down providers.
davidad: Since I have spoken about the infeasibility of an international agreement that would halt or slow the development of superintelligence (game-theoretically unstable now, at best), I should clarify there is no such obstacle to agreements restricting public access to dangerous AIs.
This is because making an AI which meets some criteria publicly accessible is:
(a) a trivially easy condition to monitor, and
(b) trivially easy to immediately renege on, if the counterparty reneges.
Together these make a “we won’t if you don’t” agreement potentially stable.
The obvious hole is that if someone decides to yolo an open model release, you cannot put the weights back in the bottle. You can absolutely prevent most use and punish them for the release, but you cannot undo the release. If that release was truly existentially risky, we’re all cooked. But all the top releases so far have come from entities that could easily be required to seek prior approval.
It is unfortunate that we have put Scott Bessent in charge of so much of our AI policy, given that he seems not to understand AI. I have nothing against him, outside of his recent AI policy decisions, but he is here to lead commerce. He is not to be a tech person. We need a tech person.
The same is true for China. We need them to send their tech people.
Lingling Wei (WSJ): The dialogue itself fell short of expectations. Beijing put the Ministry of Foreign Affairs in charge rather than a technical body such as the Ministry of Science and Technology, Doshi said, limiting the substance of the exchange.
“If the Chinese were serious about dialogue, I think they would have probably put people who know the technology best in leading roles in their delegation, as we did,” he said, adding that the U.S. side accepted the arrangement: “We didn’t really hold them to that as much as we should have.”
Well, yeah, and the same is true about Bessent. But the good news is, we’re talking.
Several times in that post, Thompson attributes rather maximally negative motivations to Anthropic over actions that have very clear reasons: Data retention for defense-in-depth implies later training on such data, not wanting distillation of their models means they think no one else should be allowed to train models at all, wanting to say what their models cannot be used for at all (which all company ToSs do) means they ‘want to have power over everything and everyone,’ etc.
I believe this is a moderately strong version of the negative form of Anthropic Derangement Syndrome (ADS), which like Trump Derangement Syndrome (TDS) exists in both directions and to varying degrees.
It does look good, actually.
After the odds had dropped quite a bit, they’re looking good again, with a 60% chance of restoration by July 1 and 88% by July 31, in the wake of groundwork looking like it is being laid in various places:
If the update is based purely on the above info I would treat the new odds as overconfident. These moves seem reasonable to make even if you have no confidence in the restoration, in order to be ready if that moment arrives.
This also suggests a potential permanent quota for Fable for subscribers. Even a modest amount is a big game here, since even a modest allocation means you can use it for non-coding tasks or minor coding tasks within the subscription.
With that on the horizon, it seems time to update our understanding of what has happened and is otherwise likely to happen, and how to think about all this.
A Rather Terrible Policy
Even if access is restored in the coming week as expected, this was a fiasco.
Dean Ball and I both invite you to imagine this situation in reverse.
The chosen implementation was such a train wreck that not only did we piss off our allies and call into question the reliability of the entire ‘American AI stack,’ we also caused the NSA to lose access to Mythos.
Hopefully no one reading this thinks the new AI executive order is meaningfully ‘voluntary.’ Jessica Tillipman illustrates that Federal procurement is one set of ways they will be imposing the new licensing mandate, with the rules flowing down to subcontractors. It’s actually a lot less voluntary than that, because the government will lean on you as needed, as it is currently leaning on lone holdout Meta.
Meta says they are ‘working through the details and hope to sign an agreement soon’ as per Meta spokesperson Francis Brennan.
The People Have Spoken
So say we (almost) all.
The good news is, these decisions are not up to a public vote, either of Twitter or otherwise. The public does not understand these issues. The public has no idea what it is talking about, and its decisions will be mostly uncorrelated with the right answer.
Thank You, Next
The public does not get to use Fable 5 until further notice.
That does not mean the labs are going to stop developing more capable AI models.
There was a brief period where Anthropic will have been somewhat slowed down by inability to use Mythos 5, internally, to train the next Mythos, forcing it to fall back on worse models. That period may already be over. I don’t think Curran would make this up without a source, but even if he claimed it purely on priors, those priors are not going to be that far off.
And if trying to release their best model gets it potentially restricted, Anthropic or OpenAI might well decide to keep it internal. Which is where the biggest risks are.
See this Law360 report, Anthropic Export Controls Stir Fear of Unforseen Sanctions. It is impossible to reliably predict how the US government will react going forward.
Inability to deploy, or uncertainty and delays around deployment, do modestly reduce the business case for pushing development, but not enough that you wouldn’t push maximally hard. There is no alternative, and the valuations of the companies make the related costs highly affordable.
Be Very Very Quiet
We will see if this prediction holds up.
What These Babies Can And Cannot Do
It’s not news, but to remind us of the stakes: The full version of Mythos can hack basically anything complex that it wants to hack, if given sufficient information, time and context.
What it cannot do is hack into all the classified networks in a matter of hours without being given various other affordances an outside attacker would not have.
What actually happened was that a professional authorized red team, given physical access to air gapped systems, identified issues in our classified systems within hours.
To be clear, this was still impressive and exceeded expectations.
Mythos is the real deal.
A series of misunderstandings created quite the pull quote that Mythos ‘broke into almost all NSA classified systems in hours.’
Mythos provided a lot of uplift, and that first result is still a big deal.
But let’s not get carried away, as some did, including inside the government.
It took several days for us to sort out what actually happened here. The pull quote will probably remain one of those zombie debunked stories that keeps being cited by politicians, and that plausibly mattered in taking and keeping Fable offline.
Bull Theory unfortunately also repeats several other misconceptions, including the idea that Anthropic took Fable offline ‘instead of’ cutting off access to foreigners, when we have explicit confirmation from Bessent that the point was to force the models offline.
What’s The Worst That Could Happen?
I have never understood the theory that if everyone had the same sufficiently advanced superhuman AI hacker, that this would favor defense over offense, given the ability to concentrate firepower on a given target, even ignoring the ‘most code will still not be superhuman for a while’ issue.
So I asked Twitter.
Teortaxes and others pointed to formal verification. I realize this exists, but so do social engineering, fraud and impersonation, and so potentially do other outside-the-formal-box forms of hacking the system, and formal verification of what matters for the larger system seems very hard and does not result in efficient code.
A fun aspect of chess is that people mostly do about as many ‘dumb things’ no matter what level of play, from the perspective of that level of play, because as you improve your play more things count as dumb. In theory perfect chess is possible, but we keep seeing the AIs get better beyond where I thought perfect play might be. My model of security is similar.
My understanding is that if your code wants to do very well-defined simple things, meaningful formal verification is within reach. But many things are not so simple.
You do distinctly have to avoid there being ransomware in every MRI scanner and iPhone. That sounds quite bad.
I do not think you could pull this off, but even if you did it would be a horrendous pain in the ass, and would rule out use of Fable on a wide variety of the most valuable software projects.
The strongest argument here is data retention, some amount of KYC plus classifier-driven observability as the real defense.
As in, if Anthropic wants to figure out who is using a lot of Fable tokens for major cybersecurity related operations, that is something they can definitely do, and they can associate it with your reputation.
This is not foolproof, it does not fully ‘fix’ the ‘jailbreak,’ but it is part of a defense-in-depth series of obstacles that greatly raises the de facto difficulty, risks and costs of offensive operations.
Do enough of that, as I expect Anthropic did, and the balance of impact of Fable shifts so that you’re safer with it released than not released.
The name of the game is effective exploitation. That requires a chain of exploitation. Mythos was special, as I understand it, in large part because it could on its own find and complete such chains, often in unexpected ways.
This is where Arthur loses me. The value of cybersecurity is not ‘less than a basis point of all IT value.’
The value of cybersecurity is all of the value of all IT, period, in the sense that if your cybersecurity fully fails then you have no IT, or might be even worse off than that.
Tail risk management does indeed have to take priority sometimes. That does not need to involve ‘AI might get everyone killed.’ This is the same as saying ‘pandemic defense is a small percentage of the national budget’ as a reason not to worry about bioweapons. No matter the right level of worry about bioweapons, that argument is a very bad reason to not worry about bioweapons.
The question is not how much typically goes wrong in cybersecurity. It is what goes wrong in a world where cybersecurity collapses. What happens if they breach our national security systems? If hackers penetrate almost any corporate network? If our hospitals and other critical infrastructure can’t stay online? If you can’t do a variety of things online anymore because they are no longer secure?
The fire department considers basically everything subordinate to fire risk and to their own convenience. I think this is quite bad, because the fire risks being prevented are often absurdly small, a vestige of a time when a fire could burn down a city.
However, if things were different, and a repeat of the Chicago Fire or similar was a real possibility, then you can see why saying ‘oh the fire department is small expense why are they bossing us around on every little detail of our streets and houses’ does not make so much sense anymore.
When we say ‘cybersecurity risk is a tax that we must pay’ the correct question is to talk price. What is the size of this tax? Are you so sure it is worth paying?
You could also challenge that the alternative involves paying that tax soon anyway, as part of the calculation. That matters, too, and must be factored in.
Yeah, that’s not how catastrophic risks and existential risks work. People need to accept this. You do not get to wait until you know exactly who is going to do what, where, how and when before you act. If you want for that, and the moment does arrive, you will be too late.
You don’t get to say ‘oh, now I see the one particular system they’re trying to hack that we can’t let them hack’ because the way you find out is that they already hacked it.
The Data Retention Policy Is About Defense In Depth
If you want to stop abuses of a powerful AI model like Fable, you have to do some amount of data retention, so you can figure out, including retroactively, when someone is up to no good, and respond. This is part of any sane defense-in-depth strategy, and rather costly given many enterprises cannot accept it.
To those who say versions of ‘they will train on the data’ or ‘sure they say they won’t train on the data but in the future maybe they will change their policy,’ please stop. No, Anthropic is not going to forcibly train on your data, nor are they going to lie about this, because their most important customers would flee in droves.
Pick Up The Phone
If you make a rule limiting AI frontier model development, as we may have to at some point, you need an agreement with robust monitoring and enforcement mechanisms. There will be great temptation to cheat.
If you make a rule limiting AI frontier model deployment, against making it available to the public, you do not do a great job protecting against existential risk, but you do not need such aggressive monitoring, as it is very easy to see what is or is not available to the public, it is and relatively easy to enforce by taking down providers.
The obvious hole is that if someone decides to yolo an open model release, you cannot put the weights back in the bottle. You can absolutely prevent most use and punish them for the release, but you cannot undo the release. If that release was truly existentially risky, we’re all cooked. But all the top releases so far have come from entities that could easily be required to seek prior approval.
Thus, Lingling Wei reports that the US and China are having talks about guardrails, including discussion of nonstate actors getting their hands on open-source models, with Treasury Secretary Scott Bessent leading the American delegation. A hotline has been suggested.
It is unfortunate that we have put Scott Bessent in charge of so much of our AI policy, given that he seems not to understand AI. I have nothing against him, outside of his recent AI policy decisions, but he is here to lead commerce. He is not to be a tech person. We need a tech person.
The same is true for China. We need them to send their tech people.
Well, yeah, and the same is true about Bessent. But the good news is, we’re talking.
People Just Say Things
Ben Thompson continues to basically take the position that the underlying facts don’t matter, if Anthropic keeps talking smack then conflict is inevitable and thus their fault. He also praises Fable as an excellent model.
Several times in that post, Thompson attributes rather maximally negative motivations to Anthropic over actions that have very clear reasons: Data retention for defense-in-depth implies later training on such data, not wanting distillation of their models means they think no one else should be allowed to train models at all, wanting to say what their models cannot be used for at all (which all company ToSs do) means they ‘want to have power over everything and everyone,’ etc.
I believe this is a moderately strong version of the negative form of Anthropic Derangement Syndrome (ADS), which like Trump Derangement Syndrome (TDS) exists in both directions and to varying degrees.