Optionality Preservation: A Convergent Instrumental Goal for Advanced AI Under Deep Uncertainty
Abstract Most AI alignment proposals attempt to specify what advanced AI should value: human preferences, happiness, flourishing, or some formalization thereof. This paper argues for a different approach: instead of encoding terminal values, we should recognize that any sufficiently advanced intelligence operating under uncertainty has strong instrumental reasons to preserve...
Dec 18, 20251