This document is entirely human written. Particular research investigations that can be taken by researchers in AI Marketplaces are in bold. I do not currently plan to work more on this project myself, but I wanted to get the idea out so that others might consider it and build on it.
Introduction
AI developers are racing towards recursive self-improvement and artificial superintelligence. If this happens, most likely all of humanity loses control to the AIs or the frontier lab irreversibly concentrates their power.
One faction which currently has not-insignificant power over the AI ecosystem are AI Marketplaces. These are organizations like OpenRouter which do not produce frontier models but generate hundreds of millions in revenue (OpenRouter brings in $140 million in annualized revenue, an estimated $50 million / 5% = $1 billion flows through their marketplace, and they just sold to Stripe for $7 billion) as they connect users to the models and providers best suited to their use cases.
If an AI pause or slowdown were to occur, these AI Marketplaces could continue being profitable and influential for years or decades. If no pause or slowdown occurs, then they will quickly become irrelevant, as superintelligence outcompetes their would-be customers and has no need for AI Marketplaces since it can handle its own compute.[3]
This implies that it is in the personal, commercial, and altruistic interests of AI Marketplaces to facilitate and push for an AI slowdown or pause and to prepare to capitalize on one if one occurs. These are the futures in which most of the value exists for AI Marketplaces.
Certain types of pause would benefit AI Marketplaces more than others. AI Marketplaces benefit from diverse models/providers/usage as that makes routing more useful. If a pause consolidates development in a few existing frontier labs, AI Marketplaces would lose influence and profitability.
A thorough AI pause plan which advocates for a pause at the frontier but encourages the diffusion of AI know-how and the integration of AI into the economy is AI 2040 by the AI Futures Project. This plan has issues which need to be worked out, but it can offer a productive starting point for thinking about how humanity can navigate the future of AI.
This essay analyzes how AI Marketplaces can contribute to and prepare for an AI pause as described in AI 2040. As a brief summary of AI 2040, it describes a recommended “Plan A” in which AI progress is harnessed and provides huge (50% yearly!) GDP growth, but a treaty between the US and China stops AI development from creating out-of-control systems too quickly.
AI 2040 advocates many policy changes, it will take a lot of political will to make them happen, and powerful interests are fighting regulation. AI Marketplaces might be able to help make Plan A happen by doing “Virtuous [Regulatory] Capture”: “When powerful interest groups thwart justified regulation, the optimal strategy for pursuing that regulation may be to indirectly empower interest groups that stand to profit from it in the long-run.”
Some economic advantages an AI Pause would give to AI Marketplaces:
Frontier labs such as Anthropic and OpenAI have a lot of bargaining power since they are a duopoly with the best models and users are willing to pay a premium for intelligence. With the frontier paused as in AI 2040, there will be a larger number of frontier providers and AI Marketplaces will have more bargaining power.
AI Marketplaces can become the intermediaries between customers and the cheap, personalized intelligence driving all economic development. This could grant them the role as the “Amazon” of the 2030s.
Some political advantages that virtuous regulatory capture would give marketplaces:
If the government mandates AI Marketplaces implement changes, then the marketplace with foresight will be prepared and competitors will have to scramble. Exemplary AI Marketplaces can push the government to mandate good behavior, making things harder for less scrupulous competition.
The government would be more likely to partner with an AI Marketplace that has implemented the features they want. This can mean billion-dollar government contracts.
Actions AI Marketplaces Could Take Now
Provider Implementation Transparency
AI2040 advocates Full Research Transparency where all research becomes open-source.[4] AI Marketplaces can incentivize model providers to open-source their inference setups, setting up “Prize Systems” which incentivize innovation as an alternative to trade secrets. Already, there is demand for providers to be more transparent about how they are running their models because the lack of openness interferes with scientific reproducibility and makes it difficult to know what object is being studied. Additionally, if implementations were open, costs would go down for consumers, and there would be less risk of switching from one provider to another with the same implementation. AI Marketplaces should gauge interest from customers for how valuable customers would find open implementations and calculate how much value the second order effects would provide to the marketplace.The AI Marketplace can run a test, offering to pay providers to share their implementation with the Marketplace and with other providers who agree not to share the implementation externally. The marketplace representative told me that “providers see their implementation as their alpha” but I still argue that there is some price at which they would be willing to sell it, and AI Marketplaces should find that price. Additionally, since there are many model providers for some models (almost 20 for GPT-OSS-120B) I would expect AI Marketplaces to be able to find a seller among them. The marketplace representative thinks that this requires too much buy-in, overhead, and trust towards untrusted providers. He prefers weights/quantization, versioning, parameter conformance, change notices, and verification as better targets. I consider this insufficient for scientific reproducibility, and under the AI 2040 Plan A, this would be insufficiently open.
Compute Tracking
An important step for an AI pause is for the parties of the treaties to know the location of all (or the significant majority) of the world’s compute. AI Marketplaces can and should use their data to infer lower bounds for how much compute each of their providers controls. An AI Marketplace could also incentivise providers to disclose their amount of compute and to install verification devices to prove that the chips are only used for inference rather than training. This information would be invaluable for a possible treaty, and if an AI Marketplace implemented these changes, they would be politically well-positioned.
Preventing Misuse
This idea may be controversial among those who work at AI Marketplaces, but it is worth considering: Bad actors and rogue AIs will demand vast quantities of inference. Governments are likely to want to be able to keep track of AI usage, mandating know-your-customer and risk-tiered access. I don’t see anything in the OpenRouter terms of service involving know-your-customer, and OpenRouter does not seem to operate a Customer Identification Program. AI Marketplaces can get a better idea of who is using their AI while maintaining the privacy in what they are using AI to do. As a concrete example, AI Marketplaces should review their data to see if there is a concerning amount of frontier bio-capable model usage in unstable regions of the world, potentially indicating bioweapons projects[5]. If AI Marketplace data has the potential to identify and predict the next COVID-scale pandemic, someone should certainly look into it.
The marketplace representative says that this part is directionally interesting and that as models get more powerful, marketplaces will likely need stronger identity requirements and high-risk usage classifiers. He says that each of the following needs to be elaboration to be actionable for them: KYC, classifier-based enforcement, aggregate regional analysis, actual threat detection, what signals are collected, what gets retained, how false positives are handled, and what actions to take when something gets flagged.
Chain-of-Thought Monitorability
Chain-of-thought monitorability is an important component of oversight, and it varies by model and by provider. AI Marketplaces should analyze their data to figure out how important chain-of-thought monitorability is to their customers. This can be done by surveying customers and analyzing consumer preferences when controlling for other aspects of model desiderata. They can then test the chain of thought monitorability of models and providers and see how user behavior changes when given this information in an A/B test. By giving customers the option to choose greater chain-of-thought monitorability, they create incentives for model providers and developers to keep chain-of-thought monitorable.
Model Specs
Transparency is core to the AI 2040 Plan A. Researchers and governments need to be able to have observability into what is going on in order to study the safety of near-superhuman AI and ensure that all parties are holding to the terms of the treaties. AI Marketplaces can contribute to this by incentivizing companies to “publicly report their model specifications (detailed documentation about what goals/values they are attempting to train their AIs to follow).” I imagine that model users already want this information to be available. AI Marketplaces can link to model specifications on their model pages, and if no such spec exists, they can clearly flag that absence in a way which discourages use. It would be interesting to A/B test this visibility feature to see how much users value visible, thorough model specifications.
Conclusion
AI Marketplaces are big believers in the power of intelligence, and their companies are built around that value. At the same time, they need to think hard about what the future will look like and which of those futures they prefer, both from a business perspective and from the perspective of members of humanity. Good futures involve humanity achieving geopolitical cooperation, good incentives, and a better understanding of AI. AI Marketplaces are both well-positioned and well-incentivized to contribute to humanity’s path toward a good outcome.
More importantly, at this point humanity would lose control over its future, but I think the point stands even from a purely profit-maximizing perspective.
This is a controversial proposal among the AI Safety community, and I will not argue for it here, merely pointing out what marketplaces can do about it.
Epistemic status: Draft-Quality. Partially motivated reasoning.
After my “Not Pinning Your OpenRouter Provider Might Invalidate Your Research” post, a representative of an AI marketplace reached out to me about providers and reproducibility. I unofficially chatted with a representative there[1] about this essay, but they do not endorse this document.[2]
This document is entirely human written. Particular research investigations that can be taken by researchers in AI Marketplaces are in bold. I do not currently plan to work more on this project myself, but I wanted to get the idea out so that others might consider it and build on it.
Introduction
AI developers are racing towards recursive self-improvement and artificial superintelligence. If this happens, most likely all of humanity loses control to the AIs or the frontier lab irreversibly concentrates their power.
One faction which currently has not-insignificant power over the AI ecosystem are AI Marketplaces. These are organizations like OpenRouter which do not produce frontier models but generate hundreds of millions in revenue (OpenRouter brings in $140 million in annualized revenue, an estimated $50 million / 5% = $1 billion flows through their marketplace, and they just sold to Stripe for $7 billion) as they connect users to the models and providers best suited to their use cases.
If an AI pause or slowdown were to occur, these AI Marketplaces could continue being profitable and influential for years or decades. If no pause or slowdown occurs, then they will quickly become irrelevant, as superintelligence outcompetes their would-be customers and has no need for AI Marketplaces since it can handle its own compute.[3]
This implies that it is in the personal, commercial, and altruistic interests of AI Marketplaces to facilitate and push for an AI slowdown or pause and to prepare to capitalize on one if one occurs. These are the futures in which most of the value exists for AI Marketplaces.
Certain types of pause would benefit AI Marketplaces more than others. AI Marketplaces benefit from diverse models/providers/usage as that makes routing more useful. If a pause consolidates development in a few existing frontier labs, AI Marketplaces would lose influence and profitability.
A thorough AI pause plan which advocates for a pause at the frontier but encourages the diffusion of AI know-how and the integration of AI into the economy is AI 2040 by the AI Futures Project. This plan has issues which need to be worked out, but it can offer a productive starting point for thinking about how humanity can navigate the future of AI.
This essay analyzes how AI Marketplaces can contribute to and prepare for an AI pause as described in AI 2040. As a brief summary of AI 2040, it describes a recommended “Plan A” in which AI progress is harnessed and provides huge (50% yearly!) GDP growth, but a treaty between the US and China stops AI development from creating out-of-control systems too quickly.
AI 2040 advocates many policy changes, it will take a lot of political will to make them happen, and powerful interests are fighting regulation. AI Marketplaces might be able to help make Plan A happen by doing “Virtuous [Regulatory] Capture”: “When powerful interest groups thwart justified regulation, the optimal strategy for pursuing that regulation may be to indirectly empower interest groups that stand to profit from it in the long-run.”
Some economic advantages an AI Pause would give to AI Marketplaces:
Some political advantages that virtuous regulatory capture would give marketplaces:
Actions AI Marketplaces Could Take Now
Provider Implementation Transparency
AI2040 advocates Full Research Transparency where all research becomes open-source.[4] AI Marketplaces can incentivize model providers to open-source their inference setups, setting up “Prize Systems” which incentivize innovation as an alternative to trade secrets. Already, there is demand for providers to be more transparent about how they are running their models because the lack of openness interferes with scientific reproducibility and makes it difficult to know what object is being studied. Additionally, if implementations were open, costs would go down for consumers, and there would be less risk of switching from one provider to another with the same implementation. AI Marketplaces should gauge interest from customers for how valuable customers would find open implementations and calculate how much value the second order effects would provide to the marketplace. The AI Marketplace can run a test, offering to pay providers to share their implementation with the Marketplace and with other providers who agree not to share the implementation externally. The marketplace representative told me that “providers see their implementation as their alpha” but I still argue that there is some price at which they would be willing to sell it, and AI Marketplaces should find that price. Additionally, since there are many model providers for some models (almost 20 for GPT-OSS-120B) I would expect AI Marketplaces to be able to find a seller among them. The marketplace representative thinks that this requires too much buy-in, overhead, and trust towards untrusted providers. He prefers weights/quantization, versioning, parameter conformance, change notices, and verification as better targets. I consider this insufficient for scientific reproducibility, and under the AI 2040 Plan A, this would be insufficiently open.
Compute Tracking
An important step for an AI pause is for the parties of the treaties to know the location of all (or the significant majority) of the world’s compute. AI Marketplaces can and should use their data to infer lower bounds for how much compute each of their providers controls. An AI Marketplace could also incentivise providers to disclose their amount of compute and to install verification devices to prove that the chips are only used for inference rather than training. This information would be invaluable for a possible treaty, and if an AI Marketplace implemented these changes, they would be politically well-positioned.
Preventing Misuse
This idea may be controversial among those who work at AI Marketplaces, but it is worth considering: Bad actors and rogue AIs will demand vast quantities of inference. Governments are likely to want to be able to keep track of AI usage, mandating know-your-customer and risk-tiered access. I don’t see anything in the OpenRouter terms of service involving know-your-customer, and OpenRouter does not seem to operate a Customer Identification Program. AI Marketplaces can get a better idea of who is using their AI while maintaining the privacy in what they are using AI to do. As a concrete example, AI Marketplaces should review their data to see if there is a concerning amount of frontier bio-capable model usage in unstable regions of the world, potentially indicating bioweapons projects[5]. If AI Marketplace data has the potential to identify and predict the next COVID-scale pandemic, someone should certainly look into it.
The marketplace representative says that this part is directionally interesting and that as models get more powerful, marketplaces will likely need stronger identity requirements and high-risk usage classifiers. He says that each of the following needs to be elaboration to be actionable for them: KYC, classifier-based enforcement, aggregate regional analysis, actual threat detection, what signals are collected, what gets retained, how false positives are handled, and what actions to take when something gets flagged.
Chain-of-Thought Monitorability
Chain-of-thought monitorability is an important component of oversight, and it varies by model and by provider. AI Marketplaces should analyze their data to figure out how important chain-of-thought monitorability is to their customers. This can be done by surveying customers and analyzing consumer preferences when controlling for other aspects of model desiderata. They can then test the chain of thought monitorability of models and providers and see how user behavior changes when given this information in an A/B test. By giving customers the option to choose greater chain-of-thought monitorability, they create incentives for model providers and developers to keep chain-of-thought monitorable.
Model Specs
Transparency is core to the AI 2040 Plan A. Researchers and governments need to be able to have observability into what is going on in order to study the safety of near-superhuman AI and ensure that all parties are holding to the terms of the treaties. AI Marketplaces can contribute to this by incentivizing companies to “publicly report their model specifications (detailed documentation about what goals/values they are attempting to train their AIs to follow).” I imagine that model users already want this information to be available. AI Marketplaces can link to model specifications on their model pages, and if no such spec exists, they can clearly flag that absence in a way which discourages use. It would be interesting to A/B test this visibility feature to see how much users value visible, thorough model specifications.
Conclusion
AI Marketplaces are big believers in the power of intelligence, and their companies are built around that value. At the same time, they need to think hard about what the future will look like and which of those futures they prefer, both from a business perspective and from the perspective of members of humanity. Good futures involve humanity achieving geopolitical cooperation, good incentives, and a better understanding of AI. AI Marketplaces are both well-positioned and well-incentivized to contribute to humanity’s path toward a good outcome.
The marketplace representative says that the most interesting parts of this essay are around provider transparency / reproducibility.
I initially wanted to name them, but I did not get permission to do so, so I rewrote this to follow Chatham House Rules.
More importantly, at this point humanity would lose control over its future, but I think the point stands even from a purely profit-maximizing perspective.
This is a controversial proposal among the AI Safety community, and I will not argue for it here, merely pointing out what marketplaces can do about it.
This is too coarse of a proxy to take action, but it would be good information to have.