Companies are building confidential computing architectures (“private AI clouds”) to run inference in a way that is private and inaccessible to the companies hosting the infrastructure. Apple, Google, and Meta all have versions of this in production today, and I think OpenAI and Anthropic are likely building this. Private AI clouds have real privacy benefits for users and security benefits for model weights, but they don’t provide true guarantees in the same way as end-to-end encryption. Users still need to place trust in the hardware manufacturer, third-party network operators, and abuse monitoring systems, among others.
A lawsuit between the New York Times and OpenAI has compelled OpenAI to produce 20 million user chats as part of the copyright claim. In OpenAI’s current architecture, they store user content, and it’s subject to subpoenas or emergency requests.
OpenAI wants to prevent similar overreaches in the future. In a blog post outlining their actions fighting the order, they wrote:
Our long-term roadmap includes advanced security features designed to keep your data private, including client-side encryption for your messages with ChatGPT. We believe these features will help keep your private conversations private and inaccessible to anyone else, even OpenAI.
OpenAI is describing a private AI cloud — where user data is processed inside a trusted execution environment (TEE) and is inaccessible to the provider, even if that provider has root access to the servers. This is a very different threat model than encryption at rest or in transit. A TEE aims to protect and encrypt data while it’s being processed. Users can send their data to the cloud without the cloud providers being able to see it.
OpenAI isn’t the first company to build a private cloud. Apple released Private Cloud Compute last June, where intensive Apple Intelligence tasks are offloaded to their data centers. Meta shipped Private Processing for WhatsApp earlier this year, which they describe as enabling “you to use Meta AI for processing messages off-device in a confidential and secure environment where no one, not even Meta or WhatsApp, can read or access your personal message”. Google announced Private AI Compute last month, which is built on TPUs.
I think it’s likely that Anthropic is also building its own version of this, and I predict that both OpenAI and Anthropic will launch private AI clouds by mid-2027 (~70% confidence). This architecture will eventually become the default for how users interact with AI systems online.
Private clouds have clear privacy benefits for users, but they also improve security for powerful AI models. User queries can’t be accessed by the provider or compelled by a subpoena. Model weights can be encrypted so that they’re only decrypted inside the private cloud’s trust boundary, and exfiltrating them then requires breaking the same protections that secure user queries. For frontier labs trying to hit SL5, this is an important step. Anthropic talks about this in their Confidential Inference via Trusted Virtual Machines report with Pattern Labs.
Private AI clouds use three main primitives to meet their security and privacy goals:
No private AI cloud is fully verifiable to users today. Private clouds also operate under a much weaker threat model than end-to-end encryption. Users still need to place significant trust in the model provider, hardware manufacturer, and operators of other parts of the system.
TEEs don’t have a particularly strong security track record. They aren’t a magic wand to eliminate trust; they just shift who needs to be trusted. Instead of trusting a cloud provider, you’re trusting that Intel, AMD, Apple, Google, or NVIDIA have correctly implemented their confidential computing design in hardware. This isn’t a trivial assumption — we’re still finding new speculative execution vulnerabilities in CPUs eight years after Spectre and Meltdown. It also isn’t a solved problem for TEEs— the largest section of Intel SGX’s Wikipedia page is the list of vulnerabilities in it!
Existing confidential computing architectures have been added on to the design of CPU chips. And while there is some penalty, manufacturers have made trade-offs to ensure acceptable performance. SEV-SNP and SGX both use deterministic encryption to protect CVM memory from the hypervisor, which has led to vulnerabilities.
Verification of these designs by independent security researchers is difficult. There’s not much public information that I could find on the details of how Apple brought confidential compute to Apple Silicon, or how Google extended it to TPUs. I spent a lot of time earlier this year reverse-engineering NVIDIA’s Confidential Compute, and I was limited by the lack of public documentation.
Oblivious HTTP’s privacy goals depend on the relay and gateway not colluding, which is why separate entities operate them. Apple uses two third parties, Fastly and Cloudflare, to operate the relay. If they colluded with Apple, they could de-anonymize a user’s request and route the targeted user to a compromised node.
I recommend reading Trail of Bits’ full report on their security review of Meta Private Processing. They found an issue with Meta’s implementation of the OHTTP protocol that could lead to de-anonymizing users (listed in the report as TOB-WAPI-3) through a malicious server returning client-specific OHTTP keys. A mitigation to address this attack vector is required by the RFC. Meta failing to implement a MUST requirement of the RFC doesn’t inspire a ton of confidence in them getting the other, more difficult parts of the design right. Apple addressed this in their design by publishing advertised keys to their transparency log.
Remote attestation lets clients verify the TEE’s environment through an attestation report, which includes cryptographic measurements of the firmware, boot chain, and loaded software. But this only helps if you can verify that measurement corresponds to source code you’ve reviewed. This requires reproducible builds — any developer with the source code can build a binary with the same hash (i.e., the build is deterministic).
Neither Apple nor Meta currently have fully reproducible builds for their private cloud components. Google hasn’t published any source code, although they have released a summary of NCC’s audit report and is built on top of Project Oak. Trail of Bits rated the lack of reproducible builds as a high severity finding for Meta’s Private Processing (TOB-WAPI-18) because of the potential for backdoors to be added:
Meta does not currently have reproducible builds or a verifiable software bill of materials (SBOM). Lack of source-level transparency makes it infeasible for independent reviewers to detect malicious behavior. A single-instruction change in a single binary executable could be enough to add an intentional backdoor to the CVM.
Meta has since made partial progress, but builds are still not fully reproducible. Apple has made portions of source code for security-critical PCC components available as part of their Virtual Research Environment. Without reproducible builds, researchers cannot verify that the published source code corresponds to the binaries actually running in production.
TEEs protect the confidential virtual machine (CVM) from software attacks by the hypervisor, but generally don’t protect against physical attacks. AMD SEV-SNP’s threat model excludes online, rowhammer-style DRAM integrity attacks. NVIDIA Confidential Compute also excludes “sophisticated physical attacks”.
Physical attacks are cheaper than most people assume, and should receive more attention. I’m not aware of a general-purpose TEE that can stand up to $10,000 in physical attacks —research earlier this year used less than $1,000 of commercial interposer hardware to break NVIDIA Confidential Compute, AMD SEV-SNP, and Intel SGX.
Trail of Bits outlined an attack scenario in their report (TOB-WAPI-7), that’s since been fixed, where an attacker could use physical attacks to extract an attested TLS private key and use that key for a person-in-the-middle attack within Meta’s network. This was possible because attestation reports didn’t have freshness guarantees. Without including a client-provided nonce, the attestation could be re-used indefinitely. This lets an attacker perform the physical extraction offline, and then re-use that extracted key for a person-in-the-middle.
During the hardware manufacturing for Apple’s Private Cloud Compute (PCC), x-ray and optical scans are taken of the printed circuit boards (PCBs). These scans are permanently recorded and run through anomaly detection to compare the PCB to known reference images. After their deployment, PCC nodes are also sampled and imaged using the same process.
Non-targetability provides defense-in-depth against physical attacks. An attacker who compromises one node can only intercept traffic that happens to route there, but they can’t steer a targeted user’s requests to that node. This limits the impact of physical attacks, but doesn’t eliminate risk to users who happen to be randomly routed to the compromised node.
Model queries today are far from anonymous. OpenAI requires KYC through Persona to use GPT-5 and Anthropic requires phone numbers to sign up for an account.
Apple’s Private Cloud includes a fraud detection protocol that passes 8 bits of unspecified information during token issuance. Apple doesn’t disclose what information this contains, although this is only processed within the TEE.
Abuse detection and response are tricky problems for private clouds. There are ways to do this, for example, by running abuse classifiers inside the TEE. This is the approach Meta takes. They run an “Output Guard TEE” which is a model that checks whether the message summaries comply with Meta’s standards.
OpenAI’s blog talks about using similar automated classifiers, but also building escape hatches into a future private AI cloud:
We will build fully automated systems to detect safety issues in our products. Only serious misuse and critical risks—such as threats to someone’s life, plans to harm others, or cybersecurity threats—may ever be escalated to a small, highly vetted team of human reviewers.
I’m skeptical that a balance can be struck. If the provider can see messages matching certain criteria, what stops them from expanding that criteria? Ideally, private clouds should report the frequency with which these escape hatches are being used in a verifiable way. Security researchers should also push back on privacy claims if those guarantees are routinely broken through abuse detection.
What would it take for these systems to be as trustworthy as end-to-end encryption? I think this is a very high bar, but is something companies should strive for.
The Reproducible Builds project has done a lot of work advocating for and enabling reproducible builds for open-source projects. Wolfi (and Chainguard) are also doing great work in this space. Private cloud providers would need to publish deterministic build configurations and let researchers verify hashes against attestation reports. This is a hard engineering problem, but is tractable.
The trusted computing base (TCB; the parts of the system that are assumed to be trusted rather than explicitly verified) is still pretty large. Limiting the TCB was an explicit design goal for Google’s architecture. Parts of commercial cloud stacks, where private clouds are likely to run, are not open source. Limiting the TCB expands what can be verified and proven to be secure.
Current architectures exclude most physical attacks from their threat model. It might not be possible to fully prevent advanced physical attacks, but chips can be hardened, so the cost of attacks is significantly raised. Apple’s manufacturing controls are another approach. Even if the hardware is vulnerable, tamper-evident architectures make tampering by attackers detectable.
Chip designs are proprietary. There were very few low-level details on how NVIDIA’s confidential compute implementation worked until “NVIDIA GPU Confidential Computing Demystified” was published earlier this year, although it’s still not particularly well documented. Open-source hardware like OpenTitan would let researchers verify no backdoors exist in the root of trust, but I’m not optimistic this happens soon. I would like to see companies make it easier for researchers to investigate the security of confidential compute hardware.
Companies and governments have been trying to kill end-to-end encryption for years. If private clouds need to have escape hatches for safety, users should demand transparency on how broad those hatches are. Hand-wavy privacy claims that collapse under abuse monitoring aren’t actually guarantees. I would like to see transparency reports, similar to OpenAI or Anthropic’s government request report, baked into the TEE protocol and published verifiably.
I think the confidential computing AI space is under-resourced relative to its importance. Tinfoil, Opaque, Edgeless Systems, Phala, Lucid Computing, and Confident Security are all startups working in this space, but we still need more smart people thinking about these problems.
Private AI clouds make accessing a user’s data significantly harder. They also strengthen security protections for powerful AI models. But private clouds are not an absolute guarantee that your data is only accessible to you. Users need to trust several parties, and whether this is acceptable depends on their threat model.
Of the current private cloud implementations, I think Apple’s is the furthest along for security and privacy. They have strong physical security controls and there’s a usable research environment accessible to researchers. Google’s implementation is very early on. There’s not much public visibility into the specifics yet, but Google has built confidential computing infrastructure at scale for a long time. Meta has two in-depth audit reports. This stuff is tricky to get right, and they get points for transparency even without a perfect report, but the findings showed gaps in the fundamentals.
No private cloud today is truly trustless or verifiable, but there are clear privacy wins for users and security benefits for model providers. I think that private AI clouds will eventually become synonymous with AI and will become the default way users interact with AI systems within a few years.
Building one is an ambitious goal. It’s tough to design, secure, and operate reliable and performant systems that stand up to sophisticated insider threats. I’m glad that companies are attempting to tackle the challenge.
I’ll be writing more about this space and AI security more broadly— subscribe to follow along. If you’re working in this space or just have thoughts, I’d love to hear from you.
Acknowledgements
Thanks to Dave Banerjee for feedback on a draft of this post.