Why societal cognition and institutional capacity may already be the limiting factors for AI governance.

Epistemic note: This essay is primarily a synthesis rather than original empirical research. I'm relatively new to the AI safety world and this model emerged from reading recent works on gradual disempowerment, governance, memetic capture, and societal cognition. My goal isn't to claim certainty but to propose what I believe is an underexplored framing and invite critique from people who know this work better than I do.

Most AI governance proposals seem dependent on three interconnected assumptions:

• The operators will be reasonable.
• The public will recognize when they aren't and push back accordingly.
• Institutions charged with acting in the best interest of the public are capable of doing so.

These unstated assumptions don't just serve as foundations for policy, but also seem to underlie research on AI safety and alignment that policy makers depend on for guidance.

More specifically, the existing models of gradual disempowerment assume sufficient societal cognition and functioning institutions, presenting the declining capacity of both as downstream harms of AI dependence. While I do agree that societal cognition and institutional capacity will be negatively impacted as AI dependence increases, I struggle to accept the initial conditions of models that assume healthy starting points.

My claim is that the aforementioned assumptions are already unsafe to hold; that both declining societal cognition and institutions incapable of remediation aren't merely consequences that may grow out of AI dependence, but rather preexisting conditions that will serve as fertilizer for AI dependence to take root faster, deeper, and more irreversibly.

With the rate of advancement continuing to accelerate, I believe this is a conversation worth having, even if my argument is ultimately proven to be incorrect. However, if it does hold true, the implications could certainly be catastrophic if not considered by those working to avoid the permanent disempowerment and displacement of humanity.

The Prevailing Models

Paul Christiano's What Failure Looks Like (2019) paints the picture for how misaligned AI could lead to humanity "going out with a whimper." Jan Kulveit and his coauthors provide further grounding for this prediction through Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development (2025) as they explore the erosion of human influence that may arise through increased AI dependence. By bridging the incentives for the replacement of human capability with the mutually reinforcing mechanisms and feedback loops that may limit resistance and remediation, they expand on Christiano’s prediction with a compelling case around the risk of humanity being permanently displaced on a global scale as AI becomes further ingrained into societal systems.

I do not disagree with the model that they have built, but rather the starting conditions from where their model begins. With that said, I’ll share exactly where and how my perspective began to divert from theirs. 

The foundation of Kulveit et al.'s argument rests upon the reciprocal relationships among economic power, cultural narratives, and political behavior. While I certainly agree with the dynamics that exist between these domains, I also believe they disregard a crucial fourth input: societal cognition. As they illustrate the interactions among these social systems, they seem to undervalue the impact that societal cognition may have on both the aforementioned factors and the systems themselves.

Furthermore, the first of six claims that they structure their argument around is that “societal systems are fairly aligned [and] broadly incentivize and produce outcomes that satisfy human preferences.” Their piece does not elaborate on this any further, and it is unclear how they came to that conclusion with such confidence to hold it firm as their leading claim. I question how they determined what adequate alignment consists of, as well as what constitutes the satisfaction of human preferences. Is this claim held up by the public’s inability to reject and fundamentally alter these systems, and if so, does a public’s compliance within a system directly correlate to that system’s level of sufficiency? 

A footnote tied to this claim seems to be an attempt to address the possibility of being challenged on its merit. They explain that they are not arguing that these systems entirely satisfy human preferences, nor that they are just or ethical. They then reiterate the core premise of their first claim, albeit this time with a noticeable difference in confidence. Are the societal systems moderately aligned and producing intended outcomes for humanity more often than they do not? Or is it only some degree of alignment, as the footnote seems to reframe?

While I may not be capable of providing definite answers to those questions, the contradiction around their conviction certainly leads me to further questions. Most significantly, what degree of alignment must be met for the baseline to be considered healthy and for their claim of “fairly aligned” to be validated? More urgently, is it possible that a society with an epistemic floor that’s already been lowered is incapable of recognizing misalignment and effectively coordinating to drive appropriate interventions?

The footnote concludes by clarifying their objective: to argue that accelerating AI advancements threaten the alignment that currently exists. While I don’t intend to subvert that goal, the framework they’ve built depends on the existence of sufficient alignment to preserve corrective feedback loops, and an inaccurate measure of said alignment requires a re-examination of everything that follows, especially as subsequent governance proposals may depend on that alignment’s accuracy in order to satisfy their own objectives. 

A more recent piece brings cognition to the forefront. In Policy Myopia as a Mechanism of Gradual Disempowerment in Post-AGI Governance, Circa 2049 (2026), Subramanyam Sahoo expands on the work of Kulveit et al., presenting a causal chain where human inefficiency results in deeper AGI delegation, leading to atrophied human capacity, which results in human resistance becoming increasingly impossible, ultimately locking humanity onto an irreversible pathway towards disempowerment.

I don’t disagree with the derivative modeled by Sahoo where atrophy is a link in the causal chain rather than simply a side effect. It’s also not about whether AI accelerates epistemic decline; it almost certainly does. Where our disagreement begins is simply a matter of timing: whether epistemic decline begins after large-scale delegation, or whether it has already progressed far enough to shape the trajectory of delegation itself. 

It is certainly possible for both to be true, but Sahoo’s model explicitly assumes a healthy starting point for societal cognition, limiting systemic decline of that cognition to merely a downstream harm of AI dependence. Just as with Kulveit et al., if he is incorrect in this assumption, the impacts on not only the causal chain he models but also any proposed methods of intervention may be too significant to ignore. 

Sahoo concludes his piece with a thorough synthesis and a suggestion of two pathways ahead: remain a civilization that organizes systems around human flourishing as a core value, or become one that optimizes systems for maximum output at the permanent expense of human influence. Once again I found myself agreeing with the model while disagreeing with the starting conditions. More specifically, I question the claim that civilization is still organizing around human flourishing. How is this determined or measured? How must models be adjusted if the choice he proposes is not one on the path ahead, but instead already behind us? If the transition to optimization at the expense of humanity has already occurred, the silent decline of gradual disempowerment that these experts warn of may have begun long before the arrival of AI.

Testing the Starting Conditions

Although I said I may not be capable of providing answers, I can certainly compile evidence in support of my argument. In doing so, I’ll follow Kulveit et al.’s lead by focusing on three distinct systems: education, media, and governance. I will present existing research as well as historical events that are relevant to my argument, but I will only expand on a few key aspects rather than fully explore every detail available. This approach is certainly not as thorough as the matter deserves, however, my intent is not to prove that my claims are irrefutable, but rather to show that they are worthy of further consideration by those working on AI safety and alignment. 

The evidence I present will also not fixate on whether a system is good or bad, nor will I attempt to suggest any corrective measures for said systems. Instead, I will examine them in an attempt to answer a singular question, inspired by the choice presented by Sahoo in the conclusion to his work: do they remain organized around human flourishing, or have they already transitioned to optimization at the expense of human benefit?

Education

Over the past two decades we’ve seen major changes within the education system in the United States. There’s also been research acknowledging a troubling reversal of the Flynn Effect, a century-long trend where generations increasingly surpass those that came before them in average IQ (Dutton et al., 2016). Researchers from Harvard, Stanford, and Dartmouth, working in collaboration to examine the state of learning across the U.S., claim that we’ve experienced what they call a “learning recession” (Dewey et al., 2026). Their most recent report presents a decline in student progress that began long before COVID, which had served as a catalyst for faster and deeper degradation. In a press statement describing their findings, Professor Thomas Kane, faculty director of the Center for Education Policy Research at Harvard University, explains that the decline began a decade earlier “after policymakers switched off the early warning system of test-based accountability” (Harvard Graduate School of Education, 2026).

Kane is referring to what he describes as a structural dismantling of accountability measures for schools driven by shifts in education policy that occurred between 2012 and 2015, centered around the transition away from the No Child Left Behind Act and towards the Every Student Succeeds Act. Although similar in nomenclature, the difference between the policies led to a significant shift in how the system would be maintained through oversight at the state and federal level. 

My aim in highlighting this policy shift has little to do with policy quality, and I have no interest in arguing over which was ultimately more effective. My intent is to examine the impact of the changes within the education system that came from this transition, specifically honing in on the removal of the “early warning system” as described by Kane. 

Although the NCLB’s flaws are well-documented, it had provided a feedback signal that allowed governing bodies to determine when and where intervention was necessary. The ESSA aimed to correct the flaws of the policy it would replace, but in doing so weakened the accountability mechanisms that had provided this signal without establishing an equivalent early-warning system. Kane’s comment about switching off the early warning system isn’t an argument in support of the NCLB, but rather in support of the need for such a mechanism serving a truly critical purpose. While the ESSA’s removal of pressures around increasing test scores positioned educators to explore new methods in driving student development, losing the ability to identify learning environments where methods were inadequate allowed for conditions where those poor learning environments could go both unnoticed and unchallenged for years to follow.

In parallel to the shift with the methods of education, we’ve also witnessed a change in the mediums educators utilize through a widespread transition to digital technologies within learning environments across the United States and many European countries (Howard & Mozejko, 2015). My intent is not to argue whether this change was ultimately good or bad, but rather to examine what drove the transition in the first place.

While these technologies may have shown great promise, the decision for systemic adoption could not have been driven by compelling evidence that they would result in positive developmental impacts because that evidence simply didn’t exist yet. What did exist were institutional pressures that made the transition unavoidable. Rather than explain all of the various pressures, I’ll specifically highlight those that came through policy, leaving little choice in the matter for schools that required funding to continue functioning:

• The Telecommunications Act (1996) - created the E-Rate program which helped shift internet connectivity in schools from 14% in 1996 to nearly 100% a decade later.
• No Child Left Behind Act (2002) - imposed extensive data-driven accountability requirements and reporting metrics that accelerated the need for electronic data management.
• Common Core Standards (2010) - introduced computer-based assessments that demanded greater bandwidth and infrastructure, contributing to the push for a 1:1 student-to-device ratio.
• Every Student Succeeds Act (2015) - formalized federal support for educational technology and digital literacy through funding for computer-based assessments and blended-learning strategies.
• CARES Act, CRRSA Act, & the American Rescue Plan (2020-2021) - provided unprecedented funding for remote learning during COVID, accelerating widespread 1:1 device adoption.

As I explored these policies and the narratives around their enactment, I noticed a recurring trend. The potential for positive impacts on student development was repeatedly invoked, yet those claims were often impossible to verify at the time. The potential for operational benefits through digital adoption was measurable and could be compared against the costs of implementation, presenting a visible value-add that was increasingly hard to deny. Regardless of the motivation behind the widespread adoption, if those promises of furthering development were to go unfulfilled in parallel with the system becoming more efficient, one would expect to see unintended consequences within the outputs of that system. 

Let’s consider one final factor that may serve as an indicator for the sufficiency of the education system: adult literacy rates in the United States. 

This is not a topic that has been brushed under the rug. Many researchers and journalists have explored the state of literacy in America, and more often than not they lean on data from the Programme for the International Assessment of Adult Competencies. The PIAAC does not just gauge whether or not individuals can read, but rather their ability to both comprehend and leverage written text across a spectrum of increasing complexity. 

Their most recent assessment, published in December of 2024, presents insight into the state of literacy in America: upper levels have remained relatively stable, the middle has contracted, and lower levels have expanded substantially (National Center for Education Statistics, 2024). The table below provides a detailed explanation for how each level is defined as well as the percentages of U.S. adults at each level across the three assessments they’ve conducted. Their most significant finding is certainly the jump for Level 1 and below from 19% in 2017 to 28% in 2023.

Although literacy itself is not a measure of societal cognition, it is one observable indicator of the cognitive capacities on which broader epistemic and institutional functions depend. These trends point to a concerning decline in skills that are essential for individuals to function within societal systems and effectively fulfill civic responsibilities. As we consider this decline alongside the aforementioned changes to the methods and mediums within the education system that preceded them, can we say with confidence that the system has remained organized around human flourishing or is it more likely that a transition to optimization at the expense of human benefit has already taken place?

Media

While the education system saw fundamental shifts, the media system has transformed substantially from its original design. My focus will be on two pivotal aspects of this transformation that have led us to where we stand today: the repeal of the Fairness Doctrine in 1987, and the consolidation of media ownership that would follow soon after.

The Fairness Doctrine had one core purpose upon enactment: ensure the American people had proper access to information relevant to matters of public interest so that they may make informed decisions (Federal Communications Commission, 1949). The origins of the policy that would set the standard for what could be called “the news” traces back to the 1920s, in reaction to the tremendous increase in radio broadcasting stations “from 3 to almost 600 in less than 5 years,” as explained by Steven J. Simmons in his 1976 article, Fairness Doctrine: The Early History (as cited by The Awareness Doctrine, Harvard Law Review, 2022). Describing the circumstances that birthed the policy, Simmons claimed that listeners were frequently subjected to fraud and false information by bad faith actors in attempts to both acquire profits and promote agendas. The policy was challenged two decades later in Red Lion Broadcasting Co. v. FCC (1969), where Justice Byron White would ultimately uphold the Fairness Doctrine, stating that it was consistent “with the First Amendment goal of producing an informed public capable of conducting its own affairs…”

Almost two decades later, Ronald Reagan’s FCC would claim otherwise. Their 1987 FCC Report had referred to a “chilling effect” brought on by the doctrine, and claimed that it “thwarts the purpose that it is designed to promote” by forcing broadcasters to avoid controversial topics. Upon successfully repealing the doctrine, the commission set the stage not just for major changes within media, but in every area the media could influence, with politics, the economy, and culture at the forefront.

For news media in particular, the repeal created greater room for partisan programming and commentary to compete directly with traditional objective reporting. This institutional pivot recentered news media from revolving around information quality and public trust towards increasingly prioritizing audience engagement. While highly partisan programming under the Fairness Doctrine had come with risk and expense, the repeal of the doctrine transformed it into one of the most lucrative vehicles for profit generation.

Although the significant increase in the ability to generate returns attracted corporate investors, consolidated ownership was strictly limited through regulatory measures. These limitations began with the Communications Act of 1934, which called for the government to regulate the media to preserve public interest, and were updated through the “7-7-7 Rule” of 1953, which restricted ownership to no more than 7 AM radio stations, 7 FM radio stations, and 7 television stations in an effort to both prevent monopolization as well as promote a diversity of viewpoints.

The repeal of the Fairness Doctrine did not change these limits. However, it reversed the FCC’s position on the government’s ability to mandate “fairness” in the name of public interest. This reversal in ideology set the stage for the elimination of those limitations through the same policy enactment that initiated the aforementioned changes to the education system – the Telecommunications Act of 1996.

What followed in the ensuing decades was both an aggressive push by private corporations and individual wealth-holders to acquire media organizations as well as the removal of any remaining regulations that continued to limit ownership (Martin et al., 2024). Once social media began to dominate the information landscape, digital media organizations became the new frontier for those interested in controlling the “town square,” and whether they came to control these institutions through building or buying, they would all be tied together through a common incentive: the immense potential for acquiring profits and promoting agendas.

While the consolidation and optimization of this power would ultimately result in tremendous wealth generation for the handful of conglomerates that own the vast majority of modern media, it would also result in algorithmic engagement, political polarization, and an information landscape where the public’s ability to discern truth from fiction has become an ever-increasing challenge. Rather than close this section by proposing Sahoo’s question once again, I’ll do so with an excerpt from a letter signed by thirty members of Congress in opposition to the FCC’s further loosening of ownership restrictions in 2003:

“The elimination of the media ownership rules merits a thorough and complete examination by the public. What is at stake here is no less than the availability of information on which people can make political and economic decisions. A free and independent media lies at the very heart of our democracy, but the rules your agency may overturn will have the practical effect of destroying our fundamental rights… That so little attention has been paid by the mass media to this issue should come as no surprise, and, in fact, offers a perfect example of how the weakening of media ownership rules over the past 20 years has already impacted news content.” 

Governance

In Testing Theories of American Politics: Elites, Interest Groups, and Average Citizens (2014), Martin Gilens and Benjamin I. Page introduce their research with the proposal of three questions: “Who governs? Who really rules? To what extent is the broad body of U.S. citizens sovereign, semi-sovereign, or largely powerless?” 

To answer these questions, they analyzed over 1,779 distinct policy issues between 1981 and 2002, exploring the drivers behind government actions, and, more significantly, the beneficiaries of said actions. In doing so they presented a concerning portrayal of the state of governance within America, succinctly summarized within their conclusion: “the majority does not rule—at least not in the causal sense of actually determining policy outcomes. When a majority of citizens disagrees with economic elites or with organized interests, they generally lose. Moreover, because of the strong status quo bias built into the U.S. political system, even when fairly large majorities of Americans favor policy change, they generally do not get it.”

This finding is empirical rather than ideological, derived from documented policy outcomes and the interests associated with them. While their interpretation may be debated, the underlying analysis provides an important question for this essay: what happens when institutional capacity functions effectively, but its outputs are systematically optimized toward the interests of a minority?

Their examination of governance does not uncover a system that has lost the capacity to promote human flourishing. Instead, it presents one that optimizes that capacity towards maximum benefit, consolidates the recipients of those benefits to a select few, and consistently delivers those benefits even at the expense of the public. This problem is not centered on institutional incapacity, but rather on the concentration of institutional effectiveness. 

The findings of Gilens and Page should not be ignored as we consider AI advancements and the corresponding risk of gradual disempowerment. If governance already systemically fails to correct known problems for the betterment of society, why are we willing to assume it’ll be different as the need to govern AI increasingly becomes an existential priority?

The Feedback Loop

My focus on these three systems was not incidental. Considered independently, these transformations could certainly be credited to policy myopia, technological advancement, or shifting economic and cultural dynamics. Considered together, they present a common pattern: education optimized for administrative efficiency, media for increased engagement, and governance for concentrated influence. The pattern is not ideological but structural: institutions formerly centered around the needs of the many have seemingly been optimized away from original design and towards maximizing the incentives for the few in position to receive them. While the specific incentives, operators, and stated objectives may have differed, what remained consistent was the tendency to optimize for increasingly measurable and locally rewarded outputs at the expense of the institution’s broader purpose.

I do want to make clear that I am not arguing against optimization, and the transformations I’ve described may certainly have been reasonable responses to constraints rather than evidence of misalignment. My claim is intentionally more targeted. It is not that optimization of a system is bad, but that it can be dangerous when the objectives of that optimization gradually take priority over - or replace - their original intent.

I also chose these institutions because they reinforce one another in a closed-loop system that directly influences both societal cognition and institutional capacity. Education shapes how people think, the media directs what they think about, and governance dictates what actions they may take. Together they determine not only what society knows, but the limits of what society can recognize, understand, and act upon. Whether one system deteriorated first or all three evolved together is ultimately less significant than the feedback loop they participate in: if education no longer develops capable citizens, the media no longer properly informs citizens, and governance no longer responds to citizens, then the question of whether AI might eventually hollow out institutions is no longer enough. It’s whether these institutions have already become sufficiently hollow that society is incapable of resisting further optimization no matter how far it may stray from human flourishing.

The Challenge of Self-Concealment

If meaningful degradation within this feedback loop has already occurred, then the models describing gradual disempowerment may themselves point toward an even more troubling possibility. In his more recent work, Memetic Capture: A Pluralistic Policy Framework for Governing AI-Driven Cultural Disempowerment (2026), Sahoo refers to the self-concealing nature of disempowerment, stating that “a culture that has drifted away from genuine human flourishing may not be recognised as such by the very humans it has captured.” Is it conceivable that we are already within a state of such disempowerment, and simply unable to see beyond the veil that our culture has created?

He goes on to argue that “culture is the entry point of the disempowerment spiral… It is the substrate through which humans form the values they use to govern the economy and the state. Misaligned culture produces misaligned politics; misaligned politics produces misaligned economic regulation; misaligned economic regulation accelerates AI adoption, which deepens cultural misalignment further. Culture is the entry point of the disempowerment spiral.”

Perhaps declining societal cognition is itself the entry point for cultural misalignment to emerge? If so, then cognition is not merely another variable in the disempowerment spiral, but the determining factor in whether the spiral begins in the first place.

There’s one additional question that has consumed me since I began exploring the narratives around gradual disempowerment: why hasn’t anyone else proposed what I’m claiming? 

While there are certainly many that have explored the topics I cover in far greater detail than I have, I haven’t found this framing explicitly through the lens of AI safety nor specifically regarding gradual disempowerment. I’ve found aspects of my argument, but only bits and pieces and never the full picture of what I’m claiming. Those bits and pieces were few and far between, typically found deep within the comments of a LessWrong post, and, on more than one occasion, penned by the same authors of the original gradual disempowerment paper.

Why, then, has this possibility received so little attention?

A plethora of possibilities come to mind. One that most certainly could be a factor is research fragmentation across different communities, especially considering that the literatures have largely developed in parallel to one another. Another thought that has stuck with me is both simple as well as aligned with the typical human response to a catastrophe: normalcy bias. Disaster psychology researchers have found that even with fast, visible threats, humans systemically fail to react appropriately, and instead gravitate towards minimizing or even ignoring the warning signs until it’s too late (Ripley, 2008). This threat is neither fast nor visible, but if my argument holds true it could most certainly lead to disaster.

None of these explanations require my claim to be wrong. But if the condition I’m describing is real, the difficulty in recognizing it may itself be a part of the phenomenon. 

The Question Ahead

The potential impacts of AI are often compared to that of the Industrial Revolution. I’ve become increasingly convinced that the long-term significance may be closer to the emergence of language. Language transformed humanity by enabling coordination at an unprecedented scale. AI may represent a similar leap — not because it solves every problem, but because it dramatically expands humanity's capacity to solve them. 

The likelihood of humanity embarking on this path as AI advancements continue to accelerate will depend on the answers to the following questions:

• On the incentive layer: Can we trust the people making the decisions?
• On the cognitive layer: Can society recognize when their decisions are dangerous?
• On the institutional layer: Can the mechanisms responsible for correcting dangerous decisions function as intended?

If the argument I make within this piece is ultimately correct – where the operators will not be reliably reasonable and both the public and the institutions that check power on its behalf may already lack the capacity to notice – then one final question will be the deciding factor for the future of our species, and this one aligns with the coordination layer just as language had when it built the foundation of our civilization:

Will AI increase humanity’s capacity to coordinate and rebuild institutions to further human benefit, or will it accelerate the loss of control as it finishes hollowing out what remains of them?