(The author is not affiliated with the Department of War or any major AI company.)
There’s a lot of disagreement about the new surveillance language in the OpenAI–Department of War agreement. Some people think it's a significant improvement over the previous language.[1] Others think it patches some issues but still leaves enough loopholes to not make a material difference. Reasonable people disagree about how a court will interpret the language, if push comes to shove.
But here's something that should be much easier to agree on: the language as written is ambiguous, and OpenAI can do better.
I don’t think even OpenAI's leadership can be confident about how this language would be interpreted in court, given the wording used and the short amount of time they’ve had to draft it. People with less context and resources will find it even harder to know how all the ambiguities would be resolved.
Some of the ambiguities seem like they could have been easily clarified despite the small amount of time available, which makes it concerning that they weren't. But more importantly, it should certainly be possible and worthwhile to spend more time on clarifying the language now. Employees are well within their rights to ask for further improvements until their own legal counsel can tell them that the language clearly prohibits what they’re worried about.
What the new language says
Please note that, with only a few paragraphs rather than the full contract, it's impossible to conclude anything with confidence. As Nathan Calvin explains, contracts often contain clauses which allow the earlier clauses to be disregarded or interpreted in unintuitive ways. In private communication, Alan Rozenshtein supports this, saying "The only way to understand a contract is to read it from beginning to end and make sure there are no (proverbial) bodies buried anywhere."[2] Given how many unjustifiably rosy interpretations of this contract Sam Altman has painted, I will not place much weight in small snippets until the full contract has been shared with someone who can verify that it doesn't substantially modify the parts that are public.
But with that said, let's analyze what we have. The new amendment adds two clauses.
Consistent with applicable laws, including the Fourth Amendment to the United States Constitution, National Security Act of 1947, FISA Act of 1978, the AI system shall not be intentionally used for domestic surveillance of U.S. persons and nationals.
For the avoidance of doubt, the Department understands this limitation to prohibit deliberate tracking, surveillance, or monitoring of U.S. persons or nationals, including through the procurement or use of commercially acquired personal or identifiable information.
Sam's internal post frames these as putting the issue to rest. Reading them carefully, they don't.
Ambiguities
Here’s a non-comprehensive list of ambiguities that could allow mass surveillance, in the colloquial sense of the term.
"Intentionally" and "deliberate" — Both clauses restrict only intentional or deliberate surveillance. Tech reporter Mike Masnick notes, “OpenAI has effectively adopted the intelligence community’s dictionary—a dictionary in which common English words have been carefully redefined over decades to permit the very things they appear to prohibit…Under the legal framework OpenAI has explicitly agreed to operate within [by citing various statutes], the NSA can target a foreign person, scoop up vast quantities of Americans’ communications in the process, retain all of it, and search through it later—and none of that counts as ‘surveillance of U.S. persons’ by the government’s own definitions.”
Many commenters have said similar things.
Jessica Tillipman, Associate Dean for Government Procurement Law Studies at GW Law, says:
"I agree it’s better, but I think the govt can drive a truck through the ‘intentionally’ language."
(Tillipman is, according to another lawyer we asked, “probably the nation's leading expert on government procurement law”.)
here's the informal/unofficial/etc answer from our law firm CEO – tldr, this language doesn't seem to add much to the previously shared contract details: (...) I’m concerned/surprised that the bar doesn’t extend to negligence or at minimum recklessness. The hierarchy of mens rea is purposely > knowingly > recklessly > negligently and courts often read "intentionally" to be somewhere in between "purposely" and "knowingly." "intentionally" is a higher bar and more difficult to prove than recklessness or negligence.
Legal Advocates for Safe Science and Technology writes:
the words “intentional” and “deliberate” leave a lot of wiggle room, especially for incidental collection and analysis. If history is any guide, the government is likely to exploit that wiggle room to allow surveillance most people would assume the language would prohibit.
"Personal or identifiable information" — This phrase is not defined in the agreement. Is metadata included in this definition? What about anonymized or pseudonymized data that an AI system could trivially de-anonymize? What about data where U.S. person identifiers are initially redacted (as is standard practice in national security work) but could be unmasked later? The contract doesn't address any of this, but the most liberal interpretations would leave little protections against surveillance.
"Tracking, surveillance, or monitoring" — Brad Carson (former General Counsel to the Army, former Undersecretary of the Army, former Undersecretary of Defense) points out that “surveillance” could refer to the FISA definition of surveillance, which doesn’t include analysis of commercial data. He also says that “tracking” and “monitoring” could be argued to require persistence over time, so that it doesn’t apply to static queries like "Tell me who went to the mosque in Tulsa and booked a trip to New York". If so, the contract wouldn’t block the DoW from doing the kind of analysis we’re worried about.
"Consistent with applicable laws (...) the AI system shall not be intentionally used for domestic surveillance of U.S. persons and nationals" — The clause opens by framing the prohibition as "consistent with" the Fourth Amendment, the National Security Act of 1947, and FISA. But these laws do not categorically ban domestic surveillance of U.S. persons in the way most people use that phrase. (See here for more on this.) The problem is that the second half of the sentence ("the AI system shall not be intentionally used for domestic surveillance of U.S. persons and nationals") could be read as being operationalized by the laws. If so, any system that complies with the law would comply with this clause. This is a problem when we’re concerned about a type of lawful use of these systems.
"For the avoidance of doubt, the Department understands this limitation to…" — The second clause is framed as the Department's stated understanding of the first clause, rather than as an additional prohibition. This leaves open the question about whether the stated “understanding” is a plausible interpretation of the first line, or what happens if new information changes the department’s understanding. I don’t know the answer, but it does create unnecessary ambiguity. Brad Carson (former General Counsel to the Army, etc., as mentioned above) writes:
And, [a hypothetical evil General Counsel] says, I particularly like that part where we say, rather strangely but certainly meaningful in some occult way, "the Department understands" rather than simply "This limitation prohibits...." I can probably argue that the latter is stronger than the former, so it must be meaningful in a way that helps my evil ways.
Why this isn’t unreasonable nit-picking
Some of this may seem like unreasonable nit-picking, but I really think it isn’t. When experts focus on seemingly minor matters of phrasing, like in the quotes above, that’s because they know that precise phrasing often does have huge implications in national security law. See the collapsible section for several examples of legal language that might look robust, followed by what actually happened.
Examples of legal language where the nitpicks mattered
Naive interpretation: Minimize surveillance of Americans in the process of foreign surveillance.
The language: Surveillance for foreign intelligence purposes…
(1) may not intentionally target any person known at the time of acquisition to be located in the United States; (2) may not intentionally target a person reasonably believed to be located outside the United States if the purpose of such acquisition is to target a particular, known person reasonably believed to be in the United States; (3) may not intentionally target a United States person reasonably believed to be located outside the United States;
Also, the NSA must adopt “minimization procedures” that are “reasonably designed ... to minimize the acquisition and retention, and prohibit the dissemination, of nonpublicly available information concerning unconsenting United States persons.”
What happened: Broad amounts of data were bulk collected under the justification that the “intention” at time of collection was to get foreign intelligence information, even though a large amount of US persons’ data also got swept up as “incidental” or “inadvertent” collection. And the minimization procedures didn't stop the NSA from using this data. According to the Brennan Center:
Between “inadvertent” and “incidental” collection, it is likely that Americans’ communications comprise a significant portion of the 250 million Internet transactions (and undisclosed number of telephone conversations) intercepted each year without a warrant or showing of probable cause. [...]
In 2011, the NSA persuaded the Foreign Intelligence Surveillance Court to approve a new set of minimization procedures under which the government may use U.S. person identifiers—including telephone numbers or e-mail accounts known to belong to Americans—to search the section 702 database for, and read, communications of or about those individuals. (...) The government may intentionally search for this information even though it would have been illegal, under section 702’s “reverse targeting” prohibition, for the government to have such intent at the time of collection.
Naive interpretation: Allow the government to obtain specific business records and tangible things relevant to authorized foreign intelligence or terrorism investigations.
The language: The FBI may apply for an order to compel people to produce "tangible things" for investigation if they have "a statement of facts showing that there are reasonable grounds to believe that the tangible things sought are relevant to an authorized investigation (other than a threat assessment) conducted in accordance with subsection (a)(2) to obtain foreign intelligence information not concerning a United States person or to protect against international terrorism or clandestine intelligence activities".
What happened: The government collected phone records of “virtually every person in the United States”. The FISA Court secretly interpreted "relevant to" as permitting bulk collection of all Americans' call records even though only a tiny fraction were used in any investigation. A DOJ fact sheet had claimed the reauthorization clarified that orders "cannot be issued unless the information sought is relevant", yet months later, the DOJ convinced the FISC that "relevant” information included all their bulk data collection because the bulk databases might include relevant data.
The language:18 U.S.C. §§ 2340 says "'torture' means an act committed by a person acting under the color of law specifically intended to inflict severe physical or mental pain or suffering (other than pain or suffering incidental to lawful sanctions) upon another person within his custody or physical control;"
What happened: The OLC redefined "severe" to mean pain "equivalent in intensity to the pain accompanying serious physical injury, such as organ failure, impairment of bodily function, or even death." They interpreted "specific intent" so that an agent who knows his techniques will cause extreme suffering still isn't guilty as long as causing pain wasn't his "precise objective." Under this reading, waterboarding, 7-day sleep deprivation, and slamming detainees into walls were all deemed legal. When Congress banned "cruel, inhuman, or degrading treatment," the OLC wrote another secret memo concluding the same techniques didn't meet that threshold either.
These are the kinds of loopholes the government can find when it tries. And the concern may not be hypothetical. There’s reporting that the Department of War specifically wants permission to use AI for the kind of analysis of commercial data that this contract is attempting to block. The Atlantic writes:
Anthropic’s team was relieved to hear that the government would be willing to remove those words, but one big problem remained: On Friday afternoon, Anthropic learned that the Pentagon still wanted to use the company’s AI to analyze bulk data collected from Americans. That could include information such as the questions you ask your favorite chatbot, your Google search history, your GPS-tracked movements, and your credit-card transactions, all of which could be cross-referenced with other details about your life. Anthropic’s leadership told Hegseth’s team that was a bridge too far, and the deal fell apart.
There’s also reporting from the New York Times on this. We don’t know much about the sources, but I think this is still more than enough reason to ensure that the contract actually would block someone who wanted to analyze Americans’ bulk data.
And in general: In order for a contract to be effective, it needs to constrain someone’s actions even when they’re trying their best to escape it. The point of a contract is that, if someone breaks it, then you expect to win a court fight against someone who argues against you as hard as they can. And as we've seen, when the DoW doesn't get what they want, they're capable of fighting pretty dirty.
Furthermore, I think it’s clear that OpenAI’s original contract was much too weak, and was only amended as a result of pressure from employees and the public.[3] This indicates that we can’t trust OpenAI’s default process to produce good language without outside pressure. Since pressure from employees and the public seems necessary here, some employees and members of the public must be evaluating these contracts as critically as if they were themselves going to sign on to them. And that’s a high bar.
Another question some people have raised is: Why didn’t Anthropic get this level of scrutiny when first signing on to work with the DoW?
Both outsiders and insiders are going to prioritize their efforts based on the amount of evidence they have that something bad is happening. At this point, we have more than enough evidence to justify the current level of scrutiny, including the reporting mentioned above and the fact that OpenAI’s first contract excerpt was clearly too weak to address the concerns here.
In addition, I think OpenAI has consistently claimed to have much stronger red lines than the evidence suggests they do.[4] I think it's important to hold companies to their word on this sort of thing. (Similarly, if Anthropic was to sign a contract with the DoW tomorrow, I would be very interested in whether they had compromised either of their stated principles.)
When Anthropic first signed a deal with the DoW, I do hope that internal employees did apply scrutiny to that. If employees had raised alarms, and Anthropic’s decision had in fact been unreasonable, then I expect that could have escalated far enough to receive public scrutiny as well.
Some of this would be easy to clarify
Not all of these problems are simple to resolve in full. But it seems like some easy improvements exist.
One improvement would be to rewrite the phrase "the Department understands this limitation" into a clear stipulated prohibition rather than a statement of understanding. It’s possible that the DoW would resist this, since it’s inconsistent with their narrative that companies shouldn’t impose any constraint beyond what’s lawful. But that’s the point. As long as one party to the contract insists that they haven’t given up anything beyond what’s already illegal, and their reading is (by a stretch) consistent with the language in the contract, there will be ambiguity about whether anything more is required.
Another improvement would be to add explicit definitions to the terms:
“surveillance”, to clarify whether “surveillance” is a term that means anything in the context of commercially acquired data.
“tracking” and “monitoring”, to clarify how much these terms require a systematic, repeated pattern over time, rather than just a large number of individual queries.
"personal or identifiable information" to clarify whether metadata is included, what kinds of anonymization or pseudonymization would be sufficient to exclude something from this category, and whether that’s consistent with easily de-anonymized data.[5]
“intentional” and “deliberate”, and define them in a way that rules out the sort of broad “incidental” collection that has historically been justified by these terms and led to scandals.
I don’t think it’s surprising that these questions would be raised and that people would want to see definitions here.[6]
Another easy clarification would be to share contractual language that we haven’t seen at all yet, when that language is important for verifying key claims. For example, what part of the contract prohibits intelligence elements in the DoW from using the provided services?[7] What language is supposed to give OpenAI full discretion over their safety stack?[8] I haven’t talked about that in this post because there’s not even any language to critique, but that just makes it even more important to get further information.
OpenAI can do much better
To reiterate, it’s genuinely hard to know how a court would interpret the stated language. I lean towards thinking that the critics are right, and that the DoW could expect to pursue many objectionable surveillance activities without worrying that OpenAI could stop them and win in court. But even if you don’t believe that, I think there’s a strong case that the language is far more ambiguous than it needs to be.
Furthermore, if the ambiguity never gets clarified, it will be disproportionately effective at preventing OpenAI from asserting its rights. In the announcement, OpenAI writes “As with any contract, we could terminate it if the counterparty violates the terms.” But will OpenAI be willing to do that if there’s a 50% chance that courts won’t side with them? What about 20%? If OpenAI terminates a contract and then loses in court, they could be forced to pay extremely high costs in damages.[9]
The contract needs more clear language. And OpenAI’s employees need to be able to vet it with their own external counsel.
For example, Charlie Bullock says it "seems like a significant improvement over the previous language with respect to surveillance". Though of course it's silent on AI-powered lethal autonomous weapons.
Also, when asked how common it is in contracts of this sort for later clauses to invalidate earlier clauses, he said "Oh all the time. Not usually a full invalidation but certainly a weakening through definitions, remedy provisions, etc." Rozenshtein is a law professor, research director and senior editor at Lawfare, and previously worked in the Office of Law and Policy in the National Security Division of the U.S. Department of Justice.
I think we can infer that the original contract was too weak without seeing the bulk of it, for two reasons. First, when choosing an excerpt, they’re incentivized to present the strongest and most reassuring language that they can. Second, when it got critiqued, they didn’t reveal further language that addressed concerns, but instead negotiated new language.
These kinds of definitions are considered important for small medical trials in a university hospital, much less agreements for how the Department of War should be able to use frontier and rapidly-improving AI capabilities.
It would also be helpful to clarify "U.S. persons." The Fourth Amendment, the National Security Act of 1947 (as amended), and FISA (as amended) do not use the same definition of U.S. person. Most notably, my understanding is that the Fourth Amendment protects everyone physically present in the U.S. who has developed a "sufficient connection" to the national community (United States v. Verdugo-Urquidez, 1990), which courts have generally understood to include undocumented immigrants and visa holders in addition to citizens and permanent residents. But the statutory definition of "U.S. person" in FISA and the National Security Act is narrower, covering only citizens and lawful permanent residents while excluding undocumented immigrants and nonimmigrant visa holders (such as someone working in the U.S. on an H-1B visa).
Former General Counsel to the Army Brad Carson is worried that this language doesn’t even exist. If it does, there’s also a question about whether it covers intelligence elements outside of intelligence agencies.
Jessica Tillipman (Associate Dean of Government Procurement Law Studies) writes: “The contract permits use ‘for all lawful purposes,’ subject to ‘operational requirements’ and ‘well-established safety and oversight protocols.’ OpenAI says it retains full discretion over the safety stack it runs in a cloud-only deployment. If the safety stack blocks a lawful use, which provision controls? The answer depends on the specific contract language governing the relationship between the permissive use standard and the deployment framework.”
There’s a further question about whether a court would support OpenAI’s decision to terminate even if they did agree that the terms were broken. Jessica Tillipman writes “I’m also curious about OpenAI’s recourse if the govt crosses a red line. In govt contracts, a contractor can’t just terminate for govt breach (w/ limited exception). If this is an OT [Other Transactions, a particular type of procurement] agreement, they may have negotiated broader termination rights, but we don’t know that.”
(The author is not affiliated with the Department of War or any major AI company.)
There’s a lot of disagreement about the new surveillance language in the OpenAI–Department of War agreement. Some people think it's a significant improvement over the previous language.[1] Others think it patches some issues but still leaves enough loopholes to not make a material difference. Reasonable people disagree about how a court will interpret the language, if push comes to shove.
But here's something that should be much easier to agree on: the language as written is ambiguous, and OpenAI can do better.
I don’t think even OpenAI's leadership can be confident about how this language would be interpreted in court, given the wording used and the short amount of time they’ve had to draft it. People with less context and resources will find it even harder to know how all the ambiguities would be resolved.
Some of the ambiguities seem like they could have been easily clarified despite the small amount of time available, which makes it concerning that they weren't. But more importantly, it should certainly be possible and worthwhile to spend more time on clarifying the language now. Employees are well within their rights to ask for further improvements until their own legal counsel can tell them that the language clearly prohibits what they’re worried about.
What the new language says
Please note that, with only a few paragraphs rather than the full contract, it's impossible to conclude anything with confidence. As Nathan Calvin explains, contracts often contain clauses which allow the earlier clauses to be disregarded or interpreted in unintuitive ways. In private communication, Alan Rozenshtein supports this, saying "The only way to understand a contract is to read it from beginning to end and make sure there are no (proverbial) bodies buried anywhere."[2] Given how many unjustifiably rosy interpretations of this contract Sam Altman has painted, I will not place much weight in small snippets until the full contract has been shared with someone who can verify that it doesn't substantially modify the parts that are public.
But with that said, let's analyze what we have. The new amendment adds two clauses.
Consistent with applicable laws, including the Fourth Amendment to the United States Constitution, National Security Act of 1947, FISA Act of 1978, the AI system shall not be intentionally used for domestic surveillance of U.S. persons and nationals.
For the avoidance of doubt, the Department understands this limitation to prohibit deliberate tracking, surveillance, or monitoring of U.S. persons or nationals, including through the procurement or use of commercially acquired personal or identifiable information.
Sam's internal post frames these as putting the issue to rest. Reading them carefully, they don't.
Ambiguities
Here’s a non-comprehensive list of ambiguities that could allow mass surveillance, in the colloquial sense of the term.
"Intentionally" and "deliberate" — Both clauses restrict only intentional or deliberate surveillance. Tech reporter Mike Masnick notes, “OpenAI has effectively adopted the intelligence community’s dictionary—a dictionary in which common English words have been carefully redefined over decades to permit the very things they appear to prohibit…Under the legal framework OpenAI has explicitly agreed to operate within [by citing various statutes], the NSA can target a foreign person, scoop up vast quantities of Americans’ communications in the process, retain all of it, and search through it later—and none of that counts as ‘surveillance of U.S. persons’ by the government’s own definitions.”
Many commenters have said similar things.
Jessica Tillipman, Associate Dean for Government Procurement Law Studies at GW Law, says:
(Tillipman is, according to another lawyer we asked, “probably the nation's leading expert on government procurement law”.)
Jeremy Howard writes:
Legal Advocates for Safe Science and Technology writes:
"Personal or identifiable information" — This phrase is not defined in the agreement. Is metadata included in this definition? What about anonymized or pseudonymized data that an AI system could trivially de-anonymize? What about data where U.S. person identifiers are initially redacted (as is standard practice in national security work) but could be unmasked later? The contract doesn't address any of this, but the most liberal interpretations would leave little protections against surveillance.
"Tracking, surveillance, or monitoring" — Brad Carson (former General Counsel to the Army, former Undersecretary of the Army, former Undersecretary of Defense) points out that “surveillance” could refer to the FISA definition of surveillance, which doesn’t include analysis of commercial data. He also says that “tracking” and “monitoring” could be argued to require persistence over time, so that it doesn’t apply to static queries like "Tell me who went to the mosque in Tulsa and booked a trip to New York". If so, the contract wouldn’t block the DoW from doing the kind of analysis we’re worried about.
"Consistent with applicable laws (...) the AI system shall not be intentionally used for domestic surveillance of U.S. persons and nationals" — The clause opens by framing the prohibition as "consistent with" the Fourth Amendment, the National Security Act of 1947, and FISA. But these laws do not categorically ban domestic surveillance of U.S. persons in the way most people use that phrase. (See here for more on this.) The problem is that the second half of the sentence ("the AI system shall not be intentionally used for domestic surveillance of U.S. persons and nationals") could be read as being operationalized by the laws. If so, any system that complies with the law would comply with this clause. This is a problem when we’re concerned about a type of lawful use of these systems.
"For the avoidance of doubt, the Department understands this limitation to…" — The second clause is framed as the Department's stated understanding of the first clause, rather than as an additional prohibition. This leaves open the question about whether the stated “understanding” is a plausible interpretation of the first line, or what happens if new information changes the department’s understanding. I don’t know the answer, but it does create unnecessary ambiguity. Brad Carson (former General Counsel to the Army, etc., as mentioned above) writes:
Why this isn’t unreasonable nit-picking
Some of this may seem like unreasonable nit-picking, but I really think it isn’t. When experts focus on seemingly minor matters of phrasing, like in the quotes above, that’s because they know that precise phrasing often does have huge implications in national security law. See the collapsible section for several examples of legal language that might look robust, followed by what actually happened.
Examples of legal language where the nitpicks mattered
1. FISA Section 702: "intentionally" and "minimize"
Naive interpretation: Minimize surveillance of Americans in the process of foreign surveillance.
The language: Surveillance for foreign intelligence purposes…
Also, the NSA must adopt “minimization procedures” that are “reasonably designed ... to minimize the acquisition and retention, and prohibit the dissemination, of nonpublicly available information concerning unconsenting United States persons.”
What happened: Broad amounts of data were bulk collected under the justification that the “intention” at time of collection was to get foreign intelligence information, even though a large amount of US persons’ data also got swept up as “incidental” or “inadvertent” collection. And the minimization procedures didn't stop the NSA from using this data. According to the Brennan Center:
2. Patriot Act Section 215 (as amended by the USA PATRIOT Improvement and Reauthorization Act of 2005): "relevant to an authorized investigation",
Naive interpretation: Allow the government to obtain specific business records and tangible things relevant to authorized foreign intelligence or terrorism investigations.
The language: The FBI may apply for an order to compel people to produce "tangible things" for investigation if they have "a statement of facts showing that there are reasonable grounds to believe that the tangible things sought are relevant to an authorized investigation (other than a threat assessment) conducted in accordance with subsection (a)(2) to obtain foreign intelligence information not concerning a United States person or to protect against international terrorism or clandestine intelligence activities".
What happened: The government collected phone records of “virtually every person in the United States”. The FISA Court secretly interpreted "relevant to" as permitting bulk collection of all Americans' call records even though only a tiny fraction were used in any investigation. A DOJ fact sheet had claimed the reauthorization clarified that orders "cannot be issued unless the information sought is relevant", yet months later, the DOJ convinced the FISC that "relevant” information included all their bulk data collection because the bulk databases might include relevant data.
3. Torture Memos: "severe" and "specific intent"
Naive interpretation: Make torture illegal.
The language: 18 U.S.C. §§ 2340 says "'torture' means an act committed by a person acting under the color of law specifically intended to inflict severe physical or mental pain or suffering (other than pain or suffering incidental to lawful sanctions) upon another person within his custody or physical control;"
What happened: The OLC redefined "severe" to mean pain "equivalent in intensity to the pain accompanying serious physical injury, such as organ failure, impairment of bodily function, or even death." They interpreted "specific intent" so that an agent who knows his techniques will cause extreme suffering still isn't guilty as long as causing pain wasn't his "precise objective." Under this reading, waterboarding, 7-day sleep deprivation, and slamming detainees into walls were all deemed legal. When Congress banned "cruel, inhuman, or degrading treatment," the OLC wrote another secret memo concluding the same techniques didn't meet that threshold either.
These are the kinds of loopholes the government can find when it tries. And the concern may not be hypothetical. There’s reporting that the Department of War specifically wants permission to use AI for the kind of analysis of commercial data that this contract is attempting to block. The Atlantic writes:
There’s also reporting from the New York Times on this. We don’t know much about the sources, but I think this is still more than enough reason to ensure that the contract actually would block someone who wanted to analyze Americans’ bulk data.
And in general: In order for a contract to be effective, it needs to constrain someone’s actions even when they’re trying their best to escape it. The point of a contract is that, if someone breaks it, then you expect to win a court fight against someone who argues against you as hard as they can. And as we've seen, when the DoW doesn't get what they want, they're capable of fighting pretty dirty.
Furthermore, I think it’s clear that OpenAI’s original contract was much too weak, and was only amended as a result of pressure from employees and the public.[3] This indicates that we can’t trust OpenAI’s default process to produce good language without outside pressure. Since pressure from employees and the public seems necessary here, some employees and members of the public must be evaluating these contracts as critically as if they were themselves going to sign on to them. And that’s a high bar.
Another question some people have raised is: Why didn’t Anthropic get this level of scrutiny when first signing on to work with the DoW?
Both outsiders and insiders are going to prioritize their efforts based on the amount of evidence they have that something bad is happening. At this point, we have more than enough evidence to justify the current level of scrutiny, including the reporting mentioned above and the fact that OpenAI’s first contract excerpt was clearly too weak to address the concerns here.
In addition, I think OpenAI has consistently claimed to have much stronger red lines than the evidence suggests they do.[4] I think it's important to hold companies to their word on this sort of thing. (Similarly, if Anthropic was to sign a contract with the DoW tomorrow, I would be very interested in whether they had compromised either of their stated principles.)
When Anthropic first signed a deal with the DoW, I do hope that internal employees did apply scrutiny to that. If employees had raised alarms, and Anthropic’s decision had in fact been unreasonable, then I expect that could have escalated far enough to receive public scrutiny as well.
Some of this would be easy to clarify
Not all of these problems are simple to resolve in full. But it seems like some easy improvements exist.
One improvement would be to rewrite the phrase "the Department understands this limitation" into a clear stipulated prohibition rather than a statement of understanding. It’s possible that the DoW would resist this, since it’s inconsistent with their narrative that companies shouldn’t impose any constraint beyond what’s lawful. But that’s the point. As long as one party to the contract insists that they haven’t given up anything beyond what’s already illegal, and their reading is (by a stretch) consistent with the language in the contract, there will be ambiguity about whether anything more is required.
Another improvement would be to add explicit definitions to the terms:
I don’t think it’s surprising that these questions would be raised and that people would want to see definitions here.[6]
Another easy clarification would be to share contractual language that we haven’t seen at all yet, when that language is important for verifying key claims. For example, what part of the contract prohibits intelligence elements in the DoW from using the provided services?[7] What language is supposed to give OpenAI full discretion over their safety stack?[8] I haven’t talked about that in this post because there’s not even any language to critique, but that just makes it even more important to get further information.
OpenAI can do much better
To reiterate, it’s genuinely hard to know how a court would interpret the stated language. I lean towards thinking that the critics are right, and that the DoW could expect to pursue many objectionable surveillance activities without worrying that OpenAI could stop them and win in court. But even if you don’t believe that, I think there’s a strong case that the language is far more ambiguous than it needs to be.
Furthermore, if the ambiguity never gets clarified, it will be disproportionately effective at preventing OpenAI from asserting its rights. In the announcement, OpenAI writes “As with any contract, we could terminate it if the counterparty violates the terms.” But will OpenAI be willing to do that if there’s a 50% chance that courts won’t side with them? What about 20%? If OpenAI terminates a contract and then loses in court, they could be forced to pay extremely high costs in damages.[9]
The contract needs more clear language. And OpenAI’s employees need to be able to vet it with their own external counsel.
For example, Charlie Bullock says it "seems like a significant improvement over the previous language with respect to surveillance". Though of course it's silent on AI-powered lethal autonomous weapons.
Also, when asked how common it is in contracts of this sort for later clauses to invalidate earlier clauses, he said "Oh all the time. Not usually a full invalidation but certainly a weakening through definitions, remedy provisions, etc." Rozenshtein is a law professor, research director and senior editor at Lawfare, and previously worked in the Office of Law and Policy in the National Security Division of the U.S. Department of Justice.
I think we can infer that the original contract was too weak without seeing the bulk of it, for two reasons. First, when choosing an excerpt, they’re incentivized to present the strongest and most reassuring language that they can. Second, when it got critiqued, they didn’t reveal further language that addressed concerns, but instead negotiated new language.
For one account, see this post, and especially the commentary on the FAQ.
These kinds of definitions are considered important for small medical trials in a university hospital, much less agreements for how the Department of War should be able to use frontier and rapidly-improving AI capabilities.
It would also be helpful to clarify "U.S. persons." The Fourth Amendment, the National Security Act of 1947 (as amended), and FISA (as amended) do not use the same definition of U.S. person. Most notably, my understanding is that the Fourth Amendment protects everyone physically present in the U.S. who has developed a "sufficient connection" to the national community (United States v. Verdugo-Urquidez, 1990), which courts have generally understood to include undocumented immigrants and visa holders in addition to citizens and permanent residents. But the statutory definition of "U.S. person" in FISA and the National Security Act is narrower, covering only citizens and lawful permanent residents while excluding undocumented immigrants and nonimmigrant visa holders (such as someone working in the U.S. on an H-1B visa).
Former General Counsel to the Army Brad Carson is worried that this language doesn’t even exist. If it does, there’s also a question about whether it covers intelligence elements outside of intelligence agencies.
Jessica Tillipman (Associate Dean of Government Procurement Law Studies) writes: “The contract permits use ‘for all lawful purposes,’ subject to ‘operational requirements’ and ‘well-established safety and oversight protocols.’ OpenAI says it retains full discretion over the safety stack it runs in a cloud-only deployment. If the safety stack blocks a lawful use, which provision controls? The answer depends on the specific contract language governing the relationship between the permissive use standard and the deployment framework.”
There’s a further question about whether a court would support OpenAI’s decision to terminate even if they did agree that the terms were broken. Jessica Tillipman writes “I’m also curious about OpenAI’s recourse if the govt crosses a red line. In govt contracts, a contractor can’t just terminate for govt breach (w/ limited exception). If this is an OT [Other Transactions, a particular type of procurement] agreement, they may have negotiated broader termination rights, but we don’t know that.”