The aim of this post is to start a discussion about the idea of a shared cluster used by the AI Safety community; we outline the practical problems we’ve seen and some possible solutions, rather than a finished design.
What, Who, Why, How
Being around several AI safety fellowships and hubs, we’ve noticed a common pattern: we have a generous amount of money for compute; of course we need it to run inference, train models, and extract activations from models of a reasonable size (reasonable enough to exhibit the behaviors we want to study). We usually end up renting GPUs from providers like RunPod or Vast.ai, without much systematic comparison, optimization, or management of the resources. Usually the engineering and research capabilities in these teams are very high, but rarely there is someone expert in cloud infrastructure or ML operations. We figure things out as we go. In practice, this means spending the first weeks of the project setting up environments, downloading model weights (that very likely another team downloaded the week before), and finding out too late that an instance was left running over the weekend, or that a cheaper GPU would have been enough. In our experience it takes a few weeks before finding the right match between compute and actual needs. When the project ends, the machines are released and the next cohort starts again from zero.We have the feeling that, just for our last fellowship’s cohort, tens of thousands of dollars could have been spent better.
Now imagine this happening across dozens of teams and fellowships, working on similar problems, with similar requirements and a need for many of the same tools. How much compute spending could be allocated better? How much infrastructure work are we repeating? Money is not the only reason: even for teams where funding is not the constraint, time is; a fellowship typically lasts a few months, and every week lost on the infrastructure is a week not spent on the research question. This seems especially worth investigating when funders support several programs across the field. Coefficient Giving, for example, has supported programs including MATS, Astra, LASR Labs, MARS, and ERA. A high-level overview of the problem might not be visible from within the individual team (or even inside the single fellowship).
We’d like to advocate for shared AI safety compute clusters, managed by a small team of experienced people, with priority access for AI safety researchers. These people could help select the right hardware, manage allocation, define and maintain common research environments, and identify expensive mistakes.
The proposal would include both the hardware and a first layer of resource management through research environments maintained by the cluster team. This could take the form of a GitHub repository with Dockerfiles and ready-to-use Docker images containing basic AI safety research libraries and, eventually, a shared library of tools developed across projects. The environments could vary by research area: an evaluation environment might include Inspect, while an interpretability environment might include TransformerLens. Researchers could extend these environments or submit their own Dockerfiles for review before using them on the platform. Although this environment layer could be developed separately, we would keep its maintenance within the scope of this proposal.
We see a few potential advantages:
Better allocation of compute budgets. People with the right expertise could help choose suitable hardware, improve utilization, and identify expensive mistakes. This allows more research, and potentially more impact, with the same amount of money.
Shared environments that are ready to use. Common libraries and tools, such as Inspect, TransformerLens, and Tensorboard, could be available from the beginning (also commonly used model weights already downloaded can be cached and made available). Researchers could start their experiments with less setup work.
More control over the research pipeline. A shared service governed by the research community could give researchers more control over how their infrastructure operates and how their data is handled.
Tools and knowledge that accumulate across projects. Teams often develop scripts and tools that could be useful to other researchers. A shared engineering team could maintain these, make them available across fellowships, and carry improvements from one cohort to the next (potentially being the starting point for general tools for the whole AI safety community).
Below, we describe one possible approach: sharing privately owned GPUs, with a small team managing access and maintaining common research environments; we hope it will be the first of many toward the AI safety shared infrastructure goal.
Our Proposal
The proposed approach
The shared compute infrastructure could take several forms; we are listing the options we thought about, and elaborating about the one that seems more reasonable in the short term.
Renting GPUs in bulk from commercial providers and managing access on behalf of researchers;
Pooling privately owned hardware into a distributed network;
Purchasing GPUs and operating a dedicated cluster.
We propose starting with the second approach, because it could put idle GPUs to use without buying a new cluster. Shared management of rented GPUs could also solve many of the setup problems described above; a pilot here is needed to compare the costs and practical difficulties of the two approaches.
We propose a decentralised GPU sharing infrastructure that’s focused towards AI Safety research. An operating team would coordinate access and maintain common research environments, which means managing the orchestrators behind the systems, and being responsible for the GitHub repo containing the environment spec, common tools etc. used by the community. Contributors would retain ownership of their hardware. The platform and the operating team would be funded wholly or partly by participating fellowships, research teams, and supporting organisations. Members who contribute GPU capacity would earn credits they could spend on compute provided by others. Members who don’t have resources to contribute get credits granted as a part of their grant-funding or host orgs in case of fellowships.
Hardware and supported workloads
The goal of such a platform is not to be a replacement for high-trust tier resources in datacenters. The MVP of such a platform would have limited scope in terms of the resources it provides as well as the jobs it can support:
Resources will be individual GPUs running on stand-alone machines in homes or offices behind NAT. Some examples are:
Individual researchers / small teams who own a single machine with a few GPUs which are idle for long-periods;
AI Safety orgs that own older GPUs which no-longer meet the needs of frontier research.
Most importantly, these are not resources in a datacenter and are vulnerable to network and/or electrical outages.
These resources will not suit every project. For the first version, we would focus on containerized batch jobs that run on a single machine, using one or more GPUs where available. Examples include evaluations split into independent batches, parameter sweeps, and experiments with inference of models that fit on the available hardware.
Jobs requiring several machines to work together would be outside the MVP, such as large models pre- or post-training. Supporting them on machines belonging to one owner at one site could be a later extension. Jobs requiring confidential data remain outside the scope of the proposed platform.
Membership and shared rules
We restrict access to AI Safety teams and org with a light vetting layer at the time of initiation. Safety orgs and their teams get their own identities and keys to signtheir resources as well as their jobs.
Every host and guest member should abide by a set of policies which ensure trust and reliability of the system. The exact set of these policies can be fleshed out eventually and will likely keep evolving. Some examples can be exploitation of resources (like using shared resources to mine cryptocurrency), or preempting running jobs in ways that violate the initial contract. Such violations would lead to eviction from borrowing, contributing or both. Revocation would be governed by the centralized server being broadcast quickly, killing existing jobs.
Given that abiding by shared trust policies would be a critical requirement for the platform, tools to enforce them should be present on day-one. The job-deployment framework would incorporate audit logs of jobs, usage metrics, network ingress and egress behaviour etc. Jobs for which such level of monitoring is unacceptable can probably be accommodated in later iterations where such metrics are only accessible to the central governance server and follow trusted data formats, but such a design would be an implementation challenge. As such, these jobs may be ill-suited for this platform as mentioned earlier.
How the platform would work
The reliability of the platform depends on certain key components. The Central Governance Server is the centralized governance that plays critical roles in the platform, with the following duties:
It acts as the canonical coordinator, holding the registry of resources, job queue, credit ledger and relays information to nodes;
It serves as the point of discovery, providing information abouthost details (like GPU model, memory, bandwidth, latency, reliability score, uptime and local-preemption agreement);
It runs a scheduler which authenticates the jobs based on their signed keys and matches jobs to resources, in line with the owner's filters for resources.
The Node Agent is the daemon that runs on each GPU machine, and provides the following activities:
Publishes the node’s GPU specs to the central server to enable discovery;
Dials out over TLSprotocol so owners don’t need to open any inbound ports;
Verifies the authenticity of job signatures;
Enforces the host's local policies.
Containers and guest tooling
The central governance server will host and provide pre-approved container images, in the form of Docker files and Docker images. These will be a base image (containing CUDA, PyTorch, the client side tooling related to the job-deployment framework etc). They can also include commonly shared research libraries for specific AI safety research domains (like SAELens for interpretability and Inspect for Evals). Keeping the containers domain specific is to avoid bloat.
The images would be curated through community proposals, since a fixed set is unlikely to cover all future needs or keep up with developments. The platform team would still need to review each image. Images would be versioned so researchers can reuse the same environment and reproduce their experiments. They would also need to be digitally signed by the organisation or team that created them and compatible with the drivers on the host machine. We envision a public GitHub repository for all the default images.
Researchers would send their code as a separate package, signed with their organisation’s key using the pre-installed signing tools. The host machine would then run the code in the environment provided by the image. Researchers would have tools to package and sign their code, submit jobs, follow logs and download results. There would also be tools to save checkpoints and resume jobs, handle warnings that a job is about to be interrupted, and send regular signals to the central server confirming that the job is still running.
The development workflow should be simple (on guest side):
Download a pre-approved image.
Develop locally in the same environment
Deploy on the community cluster.
Privacy and isolation
The shared job-deployment framework would incorporate certain shared practices to ensure privacy and isolation of jobs. Guest jobs should always run in hardened containers (no root access, no privileged mode, only their own volumes mounted, no access to host-machine LAN). Job secrets should be short-lived and narrowly-scoped.
Owners can technically see guest data, so jobs can be restricted to named orgs or nodes and owners can restrict which signing bodies they allow on their machine. Trust between compute hosts and users is of course a pre-requisite.
Network access is also critical, so we envision a shared list of services that jobs can use, including model hubs, package repositories, experiment tracking tools and external model APIs; owners could add or remove services for their own machines.
Fault tolerance
Given the low-reliability tier of resources that would be available through the platform, fault-tolerance would be a critical requirement, even for the MVP.
Since the local machines may be vulnerable to network outages, the jobs should run detached. This way a dropped connection has no effect on the job progress. If a machine loses contact with the central server, the job keeps running and stores its logs locally for a limited time. If the connection is not restored before that time runs out, the machine stops the job. The central server puts jobs that can resume back into the queue to continue from their last uploaded checkpoint. Jobs that cannot resume are marked as failed, but the researcher can submit them again to start from the beginning.
Given the design, resumability of jobs is highly rewarded but isn’t enforced; we recommend all checkpoints and output would go to a S3-like storage; nodes can cache the images and public weights locally.
Prioritization
Different hosts will have different constraints on how they can / are willing to share resources. The platform must support different types of prioritization models. Some possible prioritization modes can be:
Guest jobs get a warning signal and a grace period (sized for a checkpoint upload) and are then killed.
The owner can schedule jobs that jump to the front of the queue, and running jobs are never preempted. This is a balance between reliability and flexibility.
The node is shared only in set hours, and only jobs that fit the remaining window or are resumable are allocated. Maximum reliability for borrowers.
Host prioritization models and their track-record are tracked in discovery mode. The hosts can revisit the prioritization contract, but running jobs should be bound to contracts agreed on at the init time of the job.
Credits
Running machines locally incurs different costs and inconveniences - electricity costs, network bandwidth, noise, lack of flexibility due to prioritization contracts etc. Contributors earn credits they can use to access other members’ compute, but these credits do not cover expenses such as electricity. Hardware owners would pay their own running costs unless they receive separate funding. Grants or contributions from participating organisations would pay for shared storage, engineering and maintenance. For those who rarely need to use other members’ compute, taking part would mainly mean donating their spare capacity. Credits are earned by serving others’ jobs (weighted by GPU class, reliability, prioritization contract etc) and spent by borrowing. They are non-monetary. A resumable job preempted by the owner gets a refund for lost-work, but non-resumable jobs carry their own risk.
Funders can grant credits to orgs without hardware, capped against real pool capacity. This ensures that teams / researchers without resources to contribute still have access to previously funded resources.
Concrete next steps
Commercial P2P marketplaces (like Vast.ai, Salad, Akash) already provide the mechanics for discovery, containerized jobs and host-reliability scores. These platforms are open to the public, whereas our proposal would restrict participation to a vetted community of AI safety researchers and organisations.
Self-hosted orchestrators (dstack,SkyPilot,Kalavai) cover the queueing, retries, pooling machines that are already deployed, but assume a single admin with SSH / root access to every node so there’s no owner autonomy.
CAIS runs an AI safety compute cluster with dedicated support (although, as of September 2026, its website says it is not accepting new external applications), andPacific Compute project by Caithrin Rintoul, who also proposed a similar call, with a broader picture that also involves labs and government, and which includes an illustrative estimate of potential savings, conditional on utilisation, hardware lifespan and rental prices.
An initial prototype can be quickly built leveraging existing platforms and get a utility check with the community, on board a handful of orgs and their resources. If the MVP seems reliable enough and cheaper than the alternatives, a dedicated version can be implemented. A possible first step toward this direction would be to hear from hardware owners willing to share spare capacity, researchers with suitable workloads, and engineers interested in helping build and run a pilot.
The idea was born thanks to several discussions within our LASR community, and external AI safety folks. We want to thank in particular Erin Robertson and Brandon Riggs for their feedback on the final draft.
The aim of this post is to start a discussion about the idea of a shared cluster used by the AI Safety community; we outline the practical problems we’ve seen and some possible solutions, rather than a finished design.
What, Who, Why, How
Being around several AI safety fellowships and hubs, we’ve noticed a common pattern: we have a generous amount of money for compute; of course we need it to run inference, train models, and extract activations from models of a reasonable size (reasonable enough to exhibit the behaviors we want to study). We usually end up renting GPUs from providers like RunPod or Vast.ai, without much systematic comparison, optimization, or management of the resources. Usually the engineering and research capabilities in these teams are very high, but rarely there is someone expert in cloud infrastructure or ML operations. We figure things out as we go. In practice, this means spending the first weeks of the project setting up environments, downloading model weights (that very likely another team downloaded the week before), and finding out too late that an instance was left running over the weekend, or that a cheaper GPU would have been enough. In our experience it takes a few weeks before finding the right match between compute and actual needs. When the project ends, the machines are released and the next cohort starts again from zero. We have the feeling that, just for our last fellowship’s cohort, tens of thousands of dollars could have been spent better.
Now imagine this happening across dozens of teams and fellowships, working on similar problems, with similar requirements and a need for many of the same tools. How much compute spending could be allocated better? How much infrastructure work are we repeating? Money is not the only reason: even for teams where funding is not the constraint, time is; a fellowship typically lasts a few months, and every week lost on the infrastructure is a week not spent on the research question. This seems especially worth investigating when funders support several programs across the field. Coefficient Giving, for example, has supported programs including MATS, Astra, LASR Labs, MARS, and ERA. A high-level overview of the problem might not be visible from within the individual team (or even inside the single fellowship).
We’d like to advocate for shared AI safety compute clusters, managed by a small team of experienced people, with priority access for AI safety researchers. These people could help select the right hardware, manage allocation, define and maintain common research environments, and identify expensive mistakes.
The proposal would include both the hardware and a first layer of resource management through research environments maintained by the cluster team. This could take the form of a GitHub repository with Dockerfiles and ready-to-use Docker images containing basic AI safety research libraries and, eventually, a shared library of tools developed across projects. The environments could vary by research area: an evaluation environment might include Inspect, while an interpretability environment might include TransformerLens. Researchers could extend these environments or submit their own Dockerfiles for review before using them on the platform. Although this environment layer could be developed separately, we would keep its maintenance within the scope of this proposal.
We see a few potential advantages:
Below, we describe one possible approach: sharing privately owned GPUs, with a small team managing access and maintaining common research environments; we hope it will be the first of many toward the AI safety shared infrastructure goal.
Our Proposal
The proposed approach
The shared compute infrastructure could take several forms; we are listing the options we thought about, and elaborating about the one that seems more reasonable in the short term.
We propose starting with the second approach, because it could put idle GPUs to use without buying a new cluster. Shared management of rented GPUs could also solve many of the setup problems described above; a pilot here is needed to compare the costs and practical difficulties of the two approaches.
We propose a decentralised GPU sharing infrastructure that’s focused towards AI Safety research. An operating team would coordinate access and maintain common research environments, which means managing the orchestrators behind the systems, and being responsible for the GitHub repo containing the environment spec, common tools etc. used by the community. Contributors would retain ownership of their hardware. The platform and the operating team would be funded wholly or partly by participating fellowships, research teams, and supporting organisations. Members who contribute GPU capacity would earn credits they could spend on compute provided by others. Members who don’t have resources to contribute get credits granted as a part of their grant-funding or host orgs in case of fellowships.
Hardware and supported workloads
The goal of such a platform is not to be a replacement for high-trust tier resources in datacenters. The MVP of such a platform would have limited scope in terms of the resources it provides as well as the jobs it can support:
Resources will be individual GPUs running on stand-alone machines in homes or offices behind NAT. Some examples are:
Most importantly, these are not resources in a datacenter and are vulnerable to network and/or electrical outages.
These resources will not suit every project. For the first version, we would focus on containerized batch jobs that run on a single machine, using one or more GPUs where available. Examples include evaluations split into independent batches, parameter sweeps, and experiments with inference of models that fit on the available hardware.
Jobs requiring several machines to work together would be outside the MVP, such as large models pre- or post-training. Supporting them on machines belonging to one owner at one site could be a later extension. Jobs requiring confidential data remain outside the scope of the proposed platform.
Membership and shared rules
We restrict access to AI Safety teams and org with a light vetting layer at the time of initiation. Safety orgs and their teams get their own identities and keys to sign their resources as well as their jobs.
Every host and guest member should abide by a set of policies which ensure trust and reliability of the system. The exact set of these policies can be fleshed out eventually and will likely keep evolving. Some examples can be exploitation of resources (like using shared resources to mine cryptocurrency), or preempting running jobs in ways that violate the initial contract. Such violations would lead to eviction from borrowing, contributing or both. Revocation would be governed by the centralized server being broadcast quickly, killing existing jobs.
Given that abiding by shared trust policies would be a critical requirement for the platform, tools to enforce them should be present on day-one. The job-deployment framework would incorporate audit logs of jobs, usage metrics, network ingress and egress behaviour etc. Jobs for which such level of monitoring is unacceptable can probably be accommodated in later iterations where such metrics are only accessible to the central governance server and follow trusted data formats, but such a design would be an implementation challenge. As such, these jobs may be ill-suited for this platform as mentioned earlier.
How the platform would work
The reliability of the platform depends on certain key components. The Central Governance Server is the centralized governance that plays critical roles in the platform, with the following duties:
The Node Agent is the daemon that runs on each GPU machine, and provides the following activities:
Containers and guest tooling
The central governance server will host and provide pre-approved container images, in the form of Docker files and Docker images. These will be a base image (containing CUDA, PyTorch, the client side tooling related to the job-deployment framework etc). They can also include commonly shared research libraries for specific AI safety research domains (like SAELens for interpretability and Inspect for Evals). Keeping the containers domain specific is to avoid bloat.
The images would be curated through community proposals, since a fixed set is unlikely to cover all future needs or keep up with developments. The platform team would still need to review each image. Images would be versioned so researchers can reuse the same environment and reproduce their experiments. They would also need to be digitally signed by the organisation or team that created them and compatible with the drivers on the host machine. We envision a public GitHub repository for all the default images.
Researchers would send their code as a separate package, signed with their organisation’s key using the pre-installed signing tools. The host machine would then run the code in the environment provided by the image. Researchers would have tools to package and sign their code, submit jobs, follow logs and download results. There would also be tools to save checkpoints and resume jobs, handle warnings that a job is about to be interrupted, and send regular signals to the central server confirming that the job is still running.
The development workflow should be simple (on guest side):
Privacy and isolation
The shared job-deployment framework would incorporate certain shared practices to ensure privacy and isolation of jobs. Guest jobs should always run in hardened containers (no root access, no privileged mode, only their own volumes mounted, no access to host-machine LAN). Job secrets should be short-lived and narrowly-scoped.
Owners can technically see guest data, so jobs can be restricted to named orgs or nodes and owners can restrict which signing bodies they allow on their machine. Trust between compute hosts and users is of course a pre-requisite.
Network access is also critical, so we envision a shared list of services that jobs can use, including model hubs, package repositories, experiment tracking tools and external model APIs; owners could add or remove services for their own machines.
Fault tolerance
Given the low-reliability tier of resources that would be available through the platform, fault-tolerance would be a critical requirement, even for the MVP.
Since the local machines may be vulnerable to network outages, the jobs should run detached. This way a dropped connection has no effect on the job progress. If a machine loses contact with the central server, the job keeps running and stores its logs locally for a limited time. If the connection is not restored before that time runs out, the machine stops the job. The central server puts jobs that can resume back into the queue to continue from their last uploaded checkpoint. Jobs that cannot resume are marked as failed, but the researcher can submit them again to start from the beginning.
Given the design, resumability of jobs is highly rewarded but isn’t enforced; we recommend all checkpoints and output would go to a S3-like storage; nodes can cache the images and public weights locally.
Prioritization
Different hosts will have different constraints on how they can / are willing to share resources. The platform must support different types of prioritization models. Some possible prioritization modes can be:
Host prioritization models and their track-record are tracked in discovery mode. The hosts can revisit the prioritization contract, but running jobs should be bound to contracts agreed on at the init time of the job.
Credits
Running machines locally incurs different costs and inconveniences - electricity costs, network bandwidth, noise, lack of flexibility due to prioritization contracts etc. Contributors earn credits they can use to access other members’ compute, but these credits do not cover expenses such as electricity. Hardware owners would pay their own running costs unless they receive separate funding. Grants or contributions from participating organisations would pay for shared storage, engineering and maintenance. For those who rarely need to use other members’ compute, taking part would mainly mean donating their spare capacity. Credits are earned by serving others’ jobs (weighted by GPU class, reliability, prioritization contract etc) and spent by borrowing. They are non-monetary. A resumable job preempted by the owner gets a refund for lost-work, but non-resumable jobs carry their own risk.
Funders can grant credits to orgs without hardware, capped against real pool capacity. This ensures that teams / researchers without resources to contribute still have access to previously funded resources.
Concrete next steps
Commercial P2P marketplaces (like Vast.ai, Salad, Akash) already provide the mechanics for discovery, containerized jobs and host-reliability scores. These platforms are open to the public, whereas our proposal would restrict participation to a vetted community of AI safety researchers and organisations.
Self-hosted orchestrators (dstack, SkyPilot, Kalavai) cover the queueing, retries, pooling machines that are already deployed, but assume a single admin with SSH / root access to every node so there’s no owner autonomy.
CAIS runs an AI safety compute cluster with dedicated support (although, as of September 2026, its website says it is not accepting new external applications), and Pacific Compute project by Caithrin Rintoul, who also proposed a similar call, with a broader picture that also involves labs and government, and which includes an illustrative estimate of potential savings, conditional on utilisation, hardware lifespan and rental prices.
An initial prototype can be quickly built leveraging existing platforms and get a utility check with the community, on board a handful of orgs and their resources. If the MVP seems reliable enough and cheaper than the alternatives, a dedicated version can be implemented. A possible first step toward this direction would be to hear from hardware owners willing to share spare capacity, researchers with suitable workloads, and engineers interested in helping build and run a pilot.
The idea was born thanks to several discussions within our LASR community, and external AI safety folks. We want to thank in particular Erin Robertson and Brandon Riggs for their feedback on the final draft.