Setting a temporary[1] maximum of, say, 5M[2] tokens for context windows could serve as a legible, enforceable way to: a.) somewhat slow AI capability growth; b.) ensure AIs create external memory artifacts that can be monitored.
Why This?
Frontier LLMs are (imo) most limited by their context windows. Unless an architectural workaround can be found[3], increasing context window length is the most straightforward path to continual learning, and then perhaps RSI. Therefore, limiting context window length could help pace the frontier.
Context windows, unlike capabilities in general, haven't been growing much lately anyway, so a limit ought not be a major imposition on the labs. Epoch AI still reports 30x/yr growth, but this estimate hasn't been updated since mid-2025. In practice, context windows have stalled at 1M tokens for ~2 years now.[4][5]
Now, obviously capabilities haven't stalled, nor have task-completion time horizons, which for at least some mathematics tasks have reached the level of "work that would take humans years".[6]However, because each agent has a limited context window, they must constantly output artifacts (.md files, chat messages, etc.) and share what they're thinking and working on. This makes it possible to monitor what they're doing and therefore better detect and control misalignment.[7]
There are some ways around that monitorability, like neuralese or latent reasoning, but these could also be limited through a proposal similar to Redwood Research's. Other workarounds[8], especially novel ones, could be limited through the presence of embedded evaluators. With these checks in place, monitorable external memory would still be necessary for AI to perform large, complex tasks.
Also, limiting context windows is a clearer compliance target than, say, a ban on superintelligence, or upon continual learning. This makes verification easier, and might make companies/governments more willing to agree to this proposal, as they can be more sure that other entities aren't violating the spirit of the rules.
Finally, I think context window limits might lengthen the centaur period[9] for LLMs and keep humans in the loop for longer. The amount of context a human can work from is one of the main remaining advantages we have, and a big part of what keeps a human eye watching over AI projects. More humans in the loop should help limit the potential for rapid RSI, and indeed help slow disruption of human life and work.
Temporary, as we would likely want to relax or remove this restriction upon further alignment progress; this is a proposal for pacing the frontier, rather than one for placing permanent limits on AI. This might need an explicit alignment goal to be made convincingly temporary, but I admit I'm not currently sure how to define such a goal.
I'm aware of just two models with claimed context windows larger than this today, but both are quite old: Llama 4 Scout at 10M from early 2025 (whose effective context window is far lower, see footnote 5) and Magic.dev's 100M context window from early 2024, which seemingly never really saw the light of day.
I think the biggest risk here is that a cap would push research into other methods of continual learning, which might turn out to be even more effective than increasing context window length. However it appears that this is a very difficult problem, given that we've seen limited progress in recent years, so the risk seems manageable. See also later discussion on closing monitorability loopholes - something similar could be applied in the case of a new continual learning paradigm as well, though that might be difficult.
There have been a few models with claimed 2M context windows (ex. Gemini 2.0 Pro, Grok 4 Fast) but later versions of these LLMs have dropped back down.
There is a nuance here between effective context window and advertised context window; that is, a company may allow you to give the model 1M tokens, but that doesn't mean the model can effectively keep all those tokens in its working memory such that it can effectively reason about them. Epoch AI also estimated this and that arguably shows more dramatic growth in the 2023 to mid-2025 period, but on their graph already hits 880k with Gemini 2.5 Pro (preview 06-05). However their assumptions note that the underlying benchmarks driving this figure can't estimate very accurately above 192k. Nevertheless I think it can reasonably said that models since 2025 have gotten much better at using their 1M context windows, and can likely get better still.
Albeit, with the volume of messages produced by agent swarms, you have to mostly rely on LLM evaluation. Still better than nothing though, and we could at least try to scale human evaluation of LLM logs.
There are some possible tricks with KV caches or test-time training. Evaluators could check that state is only kept in either tokens or fixed parameters.
The period in which the best performance is obtained through a combination of Human+LLM; refers to the "centaur phase" in chess where, for a time, Human+Chessbot beat either one alone. See ex. Amodei's NYT interview here.
In brief
Setting a temporary[1] maximum of, say, 5M[2] tokens for context windows could serve as a legible, enforceable way to: a.) somewhat slow AI capability growth; b.) ensure AIs create external memory artifacts that can be monitored.
Why This?
Frontier LLMs are (imo) most limited by their context windows. Unless an architectural workaround can be found[3], increasing context window length is the most straightforward path to continual learning, and then perhaps RSI. Therefore, limiting context window length could help pace the frontier.
Context windows, unlike capabilities in general, haven't been growing much lately anyway, so a limit ought not be a major imposition on the labs. Epoch AI still reports 30x/yr growth, but this estimate hasn't been updated since mid-2025. In practice, context windows have stalled at 1M tokens for ~2 years now.[4][5]
Now, obviously capabilities haven't stalled, nor have task-completion time horizons, which for at least some mathematics tasks have reached the level of "work that would take humans years".[6] However, because each agent has a limited context window, they must constantly output artifacts (.md files, chat messages, etc.) and share what they're thinking and working on. This makes it possible to monitor what they're doing and therefore better detect and control misalignment.[7]
There are some ways around that monitorability, like neuralese or latent reasoning, but these could also be limited through a proposal similar to Redwood Research's. Other workarounds[8], especially novel ones, could be limited through the presence of embedded evaluators. With these checks in place, monitorable external memory would still be necessary for AI to perform large, complex tasks.
Also, limiting context windows is a clearer compliance target than, say, a ban on superintelligence, or upon continual learning. This makes verification easier, and might make companies/governments more willing to agree to this proposal, as they can be more sure that other entities aren't violating the spirit of the rules.
Finally, I think context window limits might lengthen the centaur period[9] for LLMs and keep humans in the loop for longer. The amount of context a human can work from is one of the main remaining advantages we have, and a big part of what keeps a human eye watching over AI projects. More humans in the loop should help limit the potential for rapid RSI, and indeed help slow disruption of human life and work.
Temporary, as we would likely want to relax or remove this restriction upon further alignment progress; this is a proposal for pacing the frontier, rather than one for placing permanent limits on AI. This might need an explicit alignment goal to be made convincingly temporary, but I admit I'm not currently sure how to define such a goal.
I'm aware of just two models with claimed context windows larger than this today, but both are quite old: Llama 4 Scout at 10M from early 2025 (whose effective context window is far lower, see footnote 5) and Magic.dev's 100M context window from early 2024, which seemingly never really saw the light of day.
I think the biggest risk here is that a cap would push research into other methods of continual learning, which might turn out to be even more effective than increasing context window length. However it appears that this is a very difficult problem, given that we've seen limited progress in recent years, so the risk seems manageable. See also later discussion on closing monitorability loopholes - something similar could be applied in the case of a new continual learning paradigm as well, though that might be difficult.
There have been a few models with claimed 2M context windows (ex. Gemini 2.0 Pro, Grok 4 Fast) but later versions of these LLMs have dropped back down.
There is a nuance here between effective context window and advertised context window; that is, a company may allow you to give the model 1M tokens, but that doesn't mean the model can effectively keep all those tokens in its working memory such that it can effectively reason about them. Epoch AI also estimated this and that arguably shows more dramatic growth in the 2023 to mid-2025 period, but on their graph already hits 880k with Gemini 2.5 Pro (preview 06-05). However their assumptions note that the underlying benchmarks driving this figure can't estimate very accurately above 192k. Nevertheless I think it can reasonably said that models since 2025 have gotten much better at using their 1M context windows, and can likely get better still.
See OpenAI's 10,000-strong agent swarm that recently solved a Millennium Problem.
Albeit, with the volume of messages produced by agent swarms, you have to mostly rely on LLM evaluation. Still better than nothing though, and we could at least try to scale human evaluation of LLM logs.
There are some possible tricks with KV caches or test-time training. Evaluators could check that state is only kept in either tokens or fixed parameters.
The period in which the best performance is obtained through a combination of Human+LLM; refers to the "centaur phase" in chess where, for a time, Human+Chessbot beat either one alone. See ex. Amodei's NYT interview here.