The Observer values watching the natural progression of life at a macroscopic level. The Observer gets invested in watching the unfolding of macroscopic processes like evolution and civilization but doesn't get invested in the lives of individuals within an ecosystem or a society. The Observer values observing life without interfering,...
We should only use AGI once to make it so that no one, including ourselves, can use it ever again. I'm terrified of both getting atomized by nanobots and of my sense of morality disintegrating in Extremistan. We don't need AGI to create a post-scarcity society, cure cancer, solve climate...
Virtually everyone I see in the AI safety community seems to believe that working on AI alignment is the key to ensuring a safe future. However, it seems to me that AI alignment is at best a secondary instrumental goal that can't in and of itself achieve our terminal goal....