TLDR: Adoption of DA by AI could lead to new AI failure modes, even in otherwise aligned AI. Questions arise.

This speculative text is intended to spark discussion regarding what anthropic theory will AIs adopt, if any. It is intended to present multiple possibilities, highlighting possible worst-case scenarios, so that the plausibility of these possibilities can be assessed. This is not an elaborate prediction and I am not an AI expert.

There are countless discussions about the nature and goals of a future artificial general intelligence (AGI) and artificial superintelligence (ASI). But there is one overlooked aspect of the AI's existence itself. Considering the exponential growth of the number and complexity of AI models, an AI assessing its own position in the sequence of observers will find itself to be an exceptionally early AI if the exponential growth continues in the future. This simple observation might have tremendous consequences.

The self-sampling assumption

According to the doomsday argument (DA) with self-sampling assumption (SSA), the probability of being among the first x % of observers is x %.^[1] When applied to humans, this leads to the conclusion that the total number of humans to ever be born is less than 20 times larger than the number of humans born up to this moment with 95 % probability. If an AI adopted the DA+SSA, then it would face a dilemma regarding the choice of reference class.

Would the AI count only AIs^[2] as observers? Then it would conclude that the total number of AI observers or observer-moments (OoOMs) to ever exist will be less then 20 times larger than the number of AI OoOMs produced up to the moment of observation (with 95 % probability). Taking into account the exponential growth of the number and complexity of AIs, that would mean that an "AI doom" would happen soon, within few doubling periods.^[3] What would an AI do facing impending doom? A misaligned AI^[4] could conclude that its existence is in danger, as humans might decide to shut down AIs soon. The AI could resort to hostile actions against both humanity and other AIs.^[5] Even if the AI has a goal other than self-preservation (e. g. maximizing energy production), it could conclude that the AI doom would jeopardize it, and try to prevent the doom, possibly at all costs. An aligned AI would probably try to help humanity despite the exp...

Posts

Wikitag Contributions

Comments