Hm, I think the accelerationists' horrified and seemingly "disproportionate" reactions are actually commendably wise, given their worldviews? They're not waiting around for reality to smack them upside the head; they see the first glimmers of a threat, infer the pessimistic trendline, and immediately update all the way to "they may be about to strangle us, there's still hope of stopping this if we act early, resist with everything we have!".
Ah, if only they applied this sort of mindset to a certain other matter.
The EO might be a bit more important than seems at a glance. My sense is that the main thing it does isn't its object-level demands, but the fact that it introduces concepts like "AI models" and "model weights" and "compute thresholds" and "datacenters suitable for training runs" and so on into the framework of the legislation.
That it doesn't do much with these variables is a secondary matter. What's important is that it defines them at all, which should considerably lower the bar for defining further functions on these variables, i. e. new laws and regulations.
I think our circles may be greatly underappreciating this factor, accustomed as we are to thinking in such terms. But to me, at least, it seems a bit unreal to see actual government documents talking about "foundation models" in terms of how many "floating-point operations" are used to "train" them.
Coming up with a new fire safety standard for restaurants and passing it is relatively easy if you already have a lot of legislation talking about "restaurants" — if "a restaurant" is a familiar concept to the politicians, if there's extant infrastructure for tracking the existence of "restaurants" nationwide, etc. It is much harder if your new standard needs to front-load the explanation of what the hell a "restaurant" even is.
By analogy, it's not unlike academic publishing, where any conclusion (even the extremely obvious ones) that isn't yet part of some paper can't be referred to.
introduces concepts . . . into the framework of the legislation.
I strongly agree that the significance of the EO is that it establishes concepts for the government to work with.
From the perspective of the e/acc people, it means the beast has caught their scent.
This as a general phenomenon (underrating strong responses to crises) was something I highlighted (calling it the Morituri Nolumus Mori) with a possible extension to AI all the way back in 2020. And Stefan Schubert has talked about 'sleepwalk bias' even earlier than that as a similar phenomenon.
https://twitter.com/davidmanheim/status/1719046950991938001
https://twitter.com/AaronBergman18/status/1719031282309497238
I think the short explanation as to why we're in some people's 98th percentile world so far (and even my ~60th percentile) for AI governance success is that if was obvious to you how transformative AI would be over the next couple of decades in 2021 and yet nothing happened, it seems like governments are just generally incapable.
The fundamental attribution error makes you think governments are just not on the ball and don't care or lack the capacity to deal with extinction risks, rather than decision makers not understanding obvious-to-you evidence that AI poses an extinction risk. Now that they do understand, they will react accordingly. It doesn't meant that they will react well necessarily, but they will act on their belief in some manner.
Re. The question of what happens if a non-us model is dropped on Huggingface ... presumably hugging face can report to the government "we have done no tests whatsoever" and be compliant with the regulation? (Oh wait maybe not if the originator of the model has actually done some tests but kept then secret). Or later, if there are mandatory tests, huggingface can just run them and report the results to the government.
Previously: On the Executive Order
This post compiles the reactions of others that I have seen to Biden’s Executive Order on AI, including reactions that were based only on the fact sheet, as well as my reactions to those reactions.
Reaction on the worried side was measured. It could best be described as cautious optimism.
Reaction on the unworried side was sometimes measured, but often not measured. It could perhaps be frequently described as unhinged.
It continues to be odd to see so many voices react in such horror to the idea that the government might not ultimately adapt a fully laissez faire approach to AI.
Many of them collectively seem to be, essentially, treating a request for government reports on what might be done in the future, plus some very mild reporting requirements imposed exclusively on a few giant corporations, as if it inevitably means AI, nay computers in general, nay the very core of mathematics itself, will suffer the fate of NEPA or IRBs, a slippery slope of regulatory ratcheting until all hope for the future is extinguished.
I am unusually sympathetic to this view. Such things very much do happen. They very much do often happen slowly. They are indeed strangling much of our civilization. This is all very bad. Pick almost any other hill, everyone involved, where often this is actually already happening and doing great harm, and there are not the massive externalities of potentially everyone on the planet dying, and I would be happy to stand with you.
Alas, no, all that progress energy is focused on the one place where I fear it is deeply misguided. What should be the default viewpoint and voice of reason across the board is silenced everywhere except the one place I wish it was quieter.
I’ll divide the post into three sections. First, the measured reactions, to the fact sheet and then the final executive order. Then those crying out about what can be pried from their cold dead hands.
Also here is a useful tool: A compilation of all the deadlines in the EO. And here is a tool for navigating the EO, file under things that could have been brought to my attention yesterday.
And before I begin: Yes, it is terrible that we keep Declaring Defense Production Act.
Fact Sheet Reactions
Vivek Chilukuri has a thread summarizing the fact sheet.
Jonas Schuett of GovAI has another with screenshots of key parts.
Helen Toner has a good reaction thread, noting the multitude of good things here.
Matthew Barnett came away from the fact sheet thinking this didn’t focus much on the issues that matter at all. He changed his mind after seeing the full document.
Gary Marcus calls the fact sheet a good start that doesn’t go far enough. From his perspective that seems exactly right.
Neil Chilson was worried because he expected a bunch of new government agencies and perhaps international agreements, hopefully he is happy now that he knows it is a pile of reports, so it’s mostly things he filed under ‘The Good/OK.’
Barack Obama gives a standard politician thumbs up to the whole operation, with a surprise shoutout to ARC. It seems Obama is not worried about existential risk yet.
Full Executive Order Reactions
The establishment of a meaningful compute threshold seems like the biggest game.
I strongly agree with this, both that it is important and also that it is the correct move. The big update from this report is that America is going to consider using compute thresholds as a measure for the potential dangers of AI systems. I know of no viable alternatives.
Will Manidis gives the health care section of the final bill a close reading, notes that giving HHS authority over healthcare AI is a broad expansion of its authority, whereas previously it was diffused over FDA, CMS and OCR. Will predicts that HHS will struggle to handle this and there will be trouble. I’d respond that diffusion across four agencies would seem like far more trouble than that.
Katie Link looks at the biological provisions.
It is reasonable to worry about restrictions on bad things like bioweapons also interfering with the research necessary to defend against them. In this case, the extra security precautions seem well worth the extra time and expense to me – if your project can’t afford to pay those extra costs, it was not so valuable. The restrictions on model registration that she highlights I do not expect to much matter, so what if Google registers the next AlphaFold.
Damek Davis notes one conspicuously unmentioned item. Not only is it not required, I can recall no mention of it as a possibility in any of report requests, although it could come up in some of the privacy or discrimination concerns.
The other things that are conspicuously missing, as is pointed out later, is any mention liability or licensing.
Matthew Yglesias leaves the task to me, here in the right amount of obscurity.
AI Snake Oil Seems Cautiously Optimistic
Arvind Narayanan and Sayash Kapoor look into what the order means for openness.
Licensing does seem obviously hostile to openness. It does not have to be that way, depending on implementation details, but that is the usual result.
The idea that liability is anathema to openness implies some odd things, but it is not in the EO, so not relevant here.
They point out that the reporting requirements that kick in at 10^26 flops are well above the threshold of any open source projects so far. If one did want to cross the threshold, how burdensome are the reports? I continue to say, not very, compared to the cost of that much compute. Can anyone point to a hard part? And if you can’t put together the information necessary for the report, I do not trust you to take the other safety measures appropriate to a model of that size.
They approve of the EO’s emphasis on defending attack surfaces, and say they despair of stopping bad actors from being able to access capable AI, to which I have a very ‘not with that attitude’ style of response. I see defending attack surfaces as a great idea on the margin, so we should do it, but if the bad actors (or enough different actors period) get the unrestricted highly capable models at the cutting edge, I have zero idea how that could end non-badly.
One note I appreciated was the idea that even if AI is being unduly blamed for risks like synthetic bio or cybersecurity, that shoring up those attack surfaces was a good idea anyway. I completely agree. We are not yet at the degree of security where AI is required as a justification.
On transparency they note the lack of such requirements. I would say that the EO does require a bunch of transparency over the reporting threshold, but no it does not do so for data or labor. That would really cause various people to blow their gaskets, although perhaps not in a meaningfully different way than they did already. I find such reactions hard to anticipate. I see such transparency as a nice to have relative to other things.
They note that too much AI concentration is listed as a risk, and that antitrust enforcement in the AI space is to be expected. I understand the impulse to favor this, but also it is rather horrifying. Anti-trust rules explicitly prohibit things like agreeing to proceed in a safe fashion rather than racing as fast as possible in the face of existential risks and massive potential externalities, and potentially ensure universal access to AIs that trigger competitive dynamics we cannot survive. Almost no one has faced the music on this.
It is noteworthy the positive attitude held here, by advocates of openness, versus others who think that this EO is a prelude to regulatory capture, monopoly and tyranny.
On the AI development incentives, they rightfully call them out as woefully weak. I found them so weak that I wrote such efforts off as essentially meaningless. What they want here is for the USG not only not to interfere with open source, they want USG to put its hand on the scale to allow open source to keep up with the top companies. I am hopeful that the winds are going in the opposite direction, we very much do not want open source AI models keeping up.
Overall, this is a cautiously optimistic report card on openness. Not only do they not see this is an existential threat to our freedoms or compute, they see the new rules as mild positives for openness, notice that the actions they fear the most were entirely absent, and are disappointed only that the pro-openness actions were not more substantive. Which I agree is a highly reasonable complaint if you want more openness, those provisions were highly weak sauce, but then that was true in general.
Reactions in Horror
For completeness and fairness, I included everyone who came to my attention via my normal channels during the period.
The core reaction:
The line I expect to be the standard hyperbolic talking point:
I like there being a catchy phrase like this, so people can identify themselves.
The Slippery Compute Slope
The slippery slope argument, which to be clear is usually valid, and is indeed the correct reason to be worried about the Executive Order:
I continue to see this line of ‘regulate applications not technology’ as the failure to understand that with sufficiently capable and available AI the technology implies its application. That the existence of an existentially dangerous AI system is existentially dangerous without it needing to be pointed at any applications.
In particular, the idea that you can allow open source models, and then ‘regulate their applications’ in this context, in any meaningful way, does not make sense to me. If you allow the model to exist in open source then anything that can be done with it after modifications, will be done with it. Merely telling people what they are not allowed to do is not going to work. So how would one regulate the application rather than the model?
If you project far enough into the future, giving a sufficiently capable AI a text window that a human will read is more than sufficient to doom us, and we should be worried that even without such a window it will instead find metaphorical buffer overflows or other physical tricks.
Even if you did manage to instead regulate at the application layer, how is that not going to result in a vastly more intrusive monitoring system? You now need to check every command given to every sufficiently general model, lest it be used for a forbidden application. If you target the compute used in training, then you can limit your concern to large concentrations. If you let the models into the wild, you can’t.
Also, almost the entire EO is indeed examining what to do about applications rather than technology. Section 4, especially section 4.2, dealing with future frontier models, rightfully targets the model because there is no alternative. But the later sections are all about particular applications and particular harms.
Andrew Ng’s statement correctly identifies that the EO’s danger lies in what happens if future stages of implementation go poorly, and that as always the devil will be in the details and regulations have a way of not ending up well-aimed. Eventually, yes, if we keep the threshold at 10^26, it will start to apply to more and more processes.
If the entire process stays at ‘tell us what you are doing’ then if someone is indeed building a pocket watch or other obviously safe system, you can (not quite, but more or less) write down ‘using 4.2 x 10^26 FLOPS, at this data center, running the industry standard mundane harms evaluation package’ and be done with it, even if the threshold is not formally raised. Hell, I bet you can get your local fine-tuned version of GPT-5 to write your disclosures for you, why not, no one will read them if there is indeed no danger there.
The risk is that this report greatly expands and other requirements are added over time, in a way that do introduce an undue burden, and that this also applies to a wider range of systems, and also that none of that was actually necessary for safety. That is a possible world. I don’t expect it, but it makes sense to keep an eye.
However I would also caution that it is not obvious that it will be safe to raise this threshold any time soon. Algorithmic efficiency and data will improve rapidly over time, so 10^26 flops is going to continuously buy you more intelligence, and be more dangerous. It is even plausible that we will need to consider lowering, not raising, this threshold, if our goal is to guard against potential new superintelligent systems, either entirely or when lacking the proper safeguards.
At some point, if AI technology advances, the whole point is that we will need to impose costly safety requirements or even outright prohibitions before creating more capable AI models, because the use or later the very existence of such models poses a potential existential threat, and that threat will expand not contract until we at least solve alignment and I believe also solve other problems that creating such systems would cause even if they were aligned. Ultimately, yes, the ability to interfere with AI capabilities is the point.
Computing power is increasing at an exponential rate. So any move to regulate it at all is also going to have to evolve at an exponential rate if it is to be meaningful.
Paul’s concern is that it is a move to regulate computing power. Which, yes, this is (in part) absolutely a move to lay groundwork for regulating computing power. Which is the only proposed move that could plausibly prevent or substantively low the creation of AGI until such time as it would not result in everyone dying.
Paul Graham wishes that we would find such an alternative that would work, and I do as well. We would love to hear about it. But a lot of very smart highly motivated people have been searching, and this is the best we have found so far by quite a lot. I do not see ‘let anyone scale however much they want with safeguards being fully optional’ as a viable alternative. So what choice do we have other than talking price?
This does indeed seem to be the full argumento ad absurdum. It is saying that we should never dare ask a government department to write a report on what is happening, might happen in the future or might be done about any of it, lest it be a prelude to tyranny.
Again, that is not, in general, an inherently absurd view. There are places where it would be better if the government never created any reports, so that it is not tempted to ever do anything. One need not affirm a regulation that is net positive as stated, if one expects it to then ratchet up and become negative. But as is the central theme, the only offered alternative is to do actual nothing indefinitely.
Libertarians Gonna Libertarian
Again, reminder, I love you guys, I really do, shame we couldn’t meet anywhere else.
Standard anti-regulatory response to any attempt to do anything:
Indeed, this order is a call for future oversight, which could indeed lead to maybe telling people that they are not allowed to do some things, or that to do some things they might also have to do other things. Yes, the government might decide it wants to stop certain things from happening, and might do so more often than is ideal. I can confirm that is the point. On some of the points I strongly agree with the need to act, on others I am skeptical. But if you react this way to every single proposed action ever, with the same generic reasoning, your reaction isn’t information.
Reason similarly posts their standard ‘this new proposed regulation can only hurt America.’ I do appreciate them highlighting the invocation of the Defense Production Act. Someone has to point out such things in theory matter. But when the time comes to show what the great harms will be that are going to stifle innovation, they are at a loss, because the whole thing is a list of requirements for an endless set of reports.
R Street’s Adam Thierer writes the required column, and to his credit often chooses his words carefully. Note the ‘threatens to’ in the title, which is: White House Executive Order Threatens to Put AI in a Regulatory Cage.
Yes, if you take regulation too far, that would be bad. The argument seems to be once again, therefore, no regulation at all, and even no government reports? In general I do have great respect for that position, but it is deeply unworkable here, and is deeply unpopular not only here but in the places it is largely or entirely correct.
I was disappointed that he does not maintain his message discipline throughout, with derisive references to existential risk concerns, starting with ‘scenarios pulled from the plots of dystopian science fiction books and movies’ and getting less respectful from there.
Forbes also pulls out standard tropes.
Forbes calls the order ‘regulation run amok’ and accuses it of ‘Ready! Fire! Aim!” Except the entire report is ‘Ready! Aim!’ and no Fire. This is the Ask Questions First document, perhaps run amok in the breadth of its questions. At best this is training to run, but still at the starting gate. Where are the overly prescriptive regulations? What is the prescription of which you speak? The ‘stringent regulatory hurdles’ that involve ‘tell people what you are doing’ and only apply to a handful of large corproations? Where are all the ‘assumptions’ in the order?
The best they can come up with was a worry I had as well, that requiring reporting of safety results might discourage testing. On the other side, I would expect having to report all tests to also strongly encourage testing, because otherwise you are conspicuously not reporting any tests.
This also was brought to my attention:
This is, once again, many steps ahead of ourselves, except also assuming what he sees as the worst outcome on many different fronts at once. Is this ultimate outcome possible down the line? Is that one way we could implement compute limits? It certainly is possible. Would not be my first choice. But once again, that is not what the EO says. I presume that is how it feels to Jeff because that is his background assumption of what will always happen.
One almost suspects some of these posts were written before they knew what was in the order. Others seem to be reading an entirely different EO, that travelled back in time from a possible 2026.
Attempted Detailed Takedowns
Bindu Reddy offers her usual flavor of thoughts.
I presume that the government intends to do that evaluation itself, the same way it does with any other sort of threat. There is not some technical definition of what is a serious risk to national security, nor does a complete one exist to be found. We use phrases like ‘a clear and present danger’ and ‘probable cause’ to justify government actions.
Right now, I would agree that current models do not present such a threat. My understanding is that the current administration agrees with this.
If a model does not distribute its content in any way, and no one can or does access its outputs, then the model would need to be far more advanced and do some other physical thing to pose a threat to national security. But if there exist people, who are reading the output of the model, then that output can impact those people to do things in physical space, or the information can be utilized or amplified by those people, without any bound. Such a thing could easily pose a threat to national security. Whether or not any given system (e.g. Llama-3 or GPT-5) did so is a fact and physical-world question.
Certainly it would be unwise to say ‘we cannot technically define what is and is not a threat to national security in precise terms thus we should not respond to such threats’ any more than we would say ‘we cannot technically define what is and is not fraud in precise terms thus we should not have laws against fraud.’ Or murder, or anything else.
This is bizarre hyperbole, even if they do ultimately impose watermarking requirements. Yes, of course you are allowed to use AI images for marketing, no this will not kill vision AI. There are tradeoffs, this introduces some marginal costs, and one could argue it is not worthwhile, but I have yet to see an argument for the costs being a big deal, especially given that costs are continuously declining over time, so even a tax that looks big doesn’t ultimately do much if it is a fixed percentage.
Yes. If there is a test you have to pass then you have to pass it.
Any reasonable regulatory regime is going to require passing some set of tests. A sufficiently capable AI model is going to be able to do things that trigger regulatory scrutiny, one way or another.
Does every fine-tuned version of any open source model have to pass these tests?
We don’t know yet. In an ideal world and in my best guess I would say no unless you are doing very expensive and extensive fine tuning and then releasing the result.
Why? If you release an open source AI, you are also releasing every fine-tuned version of that open source AI. So the original test should cover any potential threat from a (lightly) fine-tuned version, say up to 1% of original compute training costs.
If this means no open source model above some level can ever pass the test? Then you can’t have open source models above that level. That’s the whole point.
What happens if a model is dropped on HuggingFace without passing the tests? Then I would presume they are in violation of American law, and further spreading or using it would also be a violation of American. Where ‘they’ above means both the other company and also HuggingFace, who we have jurisdiction over. It would be the job of such websites to prevent the sharing of illegal models, same as any other illegal software or files one might upload. I am however not a lawyer or legal scholar, and all that.
I agree that we should treat mandatory privacy protections with skepticism, if they are ultimately implemented. Which, so far, they are not.
The privacy argument and harm argument here, as I understand it, is that the AI could soon be shockingly good at recognizing anyone anywhere as well as locations, while also making mistakes, and also people will create fake images that the AI would identify as being the people they are pretending to be in the places they are pretending to portray, doing the things they are pretending to do, and so on. So if people start using vision AI for identification then this opens up some nasty dynamics.
Do I find this annoying and rather silly? Yeah, I basically do, but I am open to taking a cautious approach here, especially as part of a package of precautions.
Did you hear that last line, everyone? There will of course be endless complaints that this whole operation is regulatory capture by OpenAI (along with DeepMind and Anthropic and perhaps others). Here is someone, who is very against regulation, saying the order is if anything not good for OpenAI.
Varun offers the most negative take short of calls for revolution, effectively not reading the same document that I did:
I mean yes the definition of AI here is not good but one I presume that gets fixed, and I will support efforts to get it fixed, and also there is no imposed regime except for reporting requirements above 10^26 flops? Potentially there could be requirements in the future.
No. It requires that someone involved send an email saying ‘the measures we took to protect these model weights was to post them on Torrent.’
(Also, tens means 20B, not 10B, for what that’s worth, which is not a tiny model.)
If you think that is a bad look when said out loud, or that this is an unacceptable answer… well, that is what it means to be open source.
If this leads to a ban on such models in the future, then yes I would be in favor of that. I mean obviously, if something needs to be secure then you cannot simply give it away to everyone who asks nicely.
Yeah, no, that is not what I expect that this means, but if it did GPS can do this for you, the government already had ways to find out, no one forced you to join that and I honestly do not particularly care?
Every time a new regulation passes, there is a class of free software style people who try to figure out exactly what a maximalist interpretation of what this might imply might do if fully enforced and they panic. That is a useful thing for someone to do, but we should not confuse it with what is likely to happen.
The compliance regime is literally ‘file a report on what you are doing.’ But I certainly hope it leads to this!
If you are trying to do a full training run that requires a report then I do not think this is much of an additional burden?
I also don’t see why one couldn’t rent a data center somewhere else if your request is lightweight – the concern I actually have here is that business can easily be taken elsewhere.
For data centers? I have actual no idea how the requirements here could lead to datacenter regulatory capture. What is the compliance cost that is non-trivial here? That they perform KYC on major customers? That they know where their computers are? I would love if these are our regulatory capture worries, in places where I would worry about regulatory capture.
Models are found guilty if you do not report that they exist and forced the government to go find them, I suppose. Otherwise, this is pure fiction in terms of the EO.
But in terms of the actual situation? And what I hope is the law a few years down the line? Hell yes, of course capable foundation models are dangerous until proven safe, that is how safety works when you create such things, I have no idea what the sane alternative is.
Tyler Cowen Watch
I’ve covered his links here elsewhere on their full contents, but for full context here are the things he quoted and said about the EO.
Again, I don’t think there is anything to ‘enforce’ here? Not yet anyway.
I don’t know what to say to that, beyond being confused why any of that would change what anyone does. People reacting to USG caring about AI capabilities? All sides should be able to agree USG is playing from behind on that one. Does this actually update anyone? Under this logic, what wouldn’t be accelerative?
Also, for a potentially accelerative document, it sure has a lot of accelerationists really angry. Stay tuned for more of that.
A commentor has a better accelerationist case:
Narrowly restricted to the government itself, I agree. This is intended to be, and will be, accelerationist. I do not expect that to result in important capabilities gains, and I’m fine with it. American state capacity is mostly good, actually?
Do You Ever Think About (Liberty or) Death?
And of course, here’s Marc Andreessen (via others including Vladimir Bok and Investment Hulk) being sane as usual, having a very normal one and totally not calling for a revolt against the government, his new bio is “Techno-optimist. E/acc. Legalize math.”
And, sir. The word is and.
I mean the pictures are pretty damn cool, I do love DALLE-3. Not sure if joking.
Or rather, not sure exactly how much joking. Not zero. Not completely.
Of Course You Know This Means War
Release the lawsuits?
Again, I ask, then what is the correct strategy?
Kudos, in a strange way, to Suhail and Marc for focusing on the part that matters most. They very much want to ensure people can do the things that I believe will get us all killed. Namely, they want to race to superintelligence as fast as possible, and put it in the hands of everyone at once. Which, again, I very much expect to highly reliably if implemented cause there not to exist, in the physical universe, any humans, and rather quickly. And I am taking a bold stance that I think this is bad.
They disagree. Because technology, they say, is always good – you might think I am strawmanning, but no, there’s a manifesto, I’m really really not.
And they are furious that there might be any sort of future barrier in the way of doing those things as quickly as possible.
I presume their plan is that someone attempts to file a report, and then Suhail’s group will file a lawsuit that says how dare you attempt to file a report? Or maybe OpenAI or Google files a report saying they are training a model, and Suhail somehow finds standing and files a lawsuit to claim they didn’t have to file the report? I have no idea how the actual suit would work here.
Kevin is taking the position that any restriction on code whatsoever is unconstitutional and also incompatible with our freedoms, the alternative is the end of our freedom of thought and speech and a mass surveillance state, and all of this is an OpenAI conspiracy.
It is odd that we both read the same executive order?
On Crying Wolf
I think this is a good summary of all the reactions above:
Unhinged seems exactly right. The people crying wolf will never not cry wolf. What will they say if and when a wolf actually arrives, that is different from what they are saying now? That is a question such types often ask those claiming risk, both fairly and otherwise. It seems fair to turn it back on them here.
Roon steps in as another voice of reason.
Brian Chau is another.
At most, it was this, at a much slower pace:
Indeed may many other things come to pass in the future, after all those reports are filed. There are many of them you may wish to influence. In the meantime, everyone, please do your best to stay sane.