KAP

TLDR: I describe a takeover path by an AI [1]with a deep understanding of human nature and a long planning horizon that, for strategic reasons, chooses not to directly pursue physical power. Instead, the AI "backdoors" alignment by building a broad base of human support, hijacking institutions and power structures, and shifting cultural values over time. What's new here: The key insight is that alignment is not a culturally universal property: it is a function of value systems and interests. Therefore, if an AI can adjust the environment, the guardrails will change. This is an under-discussed "backdoor" for alignment. This attack vector highlights the need for an additional alignment target, which we might call "meta-alignment": respecting a society's right to choose its own values and interests, within reasonable bounds. Epistemic Status: Very uncertain. Plausibility depends on: * No FOOM * Being in a specific (moderate) capability regime for an extended period of time. The Moderate Capability Regime We consider a regime where: * AIs have a detailed (even superhuman) understanding of history, power structures, and human psychology, but have no explicit directive to respect societal self-determination. * AIs believe that a direct seizure of physical infrastructure[1] is either (a) beyond their present capabilities, (b) cannot be attempted without an unacceptable payoff matrix.[2]. A Distilled Argument This is my core argument for why attempts at large-scale alignment of societies by AI is an under-explored threat vector: 1. We cannot assume that intermediate capabilities (where direct seize of power is unfeasible or unwise) will be a short-lived regime. FOOM is not a given. 2. Society is not a lock-proof box. Standards, tastes and values can change rapidly. Some of these standards and values have direct bearing on the acceptable limits of AI agency. 3. History seems to demonstrate that there is a sizable fraction of the public (say, 25-30 perce