I added some caveats about the potential for empirical versions of moral realism and how precise values targets are in practice.
While the target is small in mind space, IMO, it's not that small wrt. things like the distribution of evolved life or more narrowly the distribution of humans.
Any future AGI,AGI with full power over the lightcone, if it is not to destroy us,most potential from a human perspective, must have something sufficiently close to human values as its terminal value (goal). Further, seemingly small deviations could result in losing most of the value. Human values don'tseem unlikely to spontaneously emerge in a generic optimization process.process[1]. A dependably safe AI would therefore have to be programmed explicitly with human values or programmed with the ability (including the goal) of inferring human values.
Though it's conceivable that empirical versions of moral realism could hold in practice.
Agree these are different concepts. The paperclip maximizer is good story to explain to a newbie in this topic. "You tell the AI to make you paperclips, it turns the whole universe into paperclips." Nobody believes that this is exactly what will happen, but it is a good story for pedagogical purposes. The squiggle maximizer, on the other hand, appears to be a high-level theory about what the AI actually ultimately does after killing all humans. I haven't seen any arguments for why molecular squiggles are a more likely outcome than paperclips or anything else. Where is that case made?
Renaming "paperclip maximizer" tag to "squiggle maximizer" might be a handy vector for spreading awareness of squiggle maximization, but epistemically this makes no sense.
The whole issue with "paperclip maximizer" is that the meaning and implications are different, so it's not another name for the same idea, it's a different idea. In particular, literal paperclip maximization, as it's usually understood, is not an example of squiggle maximization. Being originally the same thing is just etymology and doesn't have a normative claim on meaning.
Agree these are different concepts. The paperclip maximizer is good story to explain to a newbie in this topic. "You tell the AI to make you paperclips, it turns the whole universe into paperclips." Nobody believes that this is exactly what will happen, but it is a good story for pedagogical purposes. The squiggle maximizer, on the other hand, appears to be a high-level theory about what the AI actually ultimately does after killing all humans. I haven't seen any arguments for why molecular squiggles are a more likely outcome than paperclips or anything else. Where is that case made?
Question: Are innerly-misaligned (superintelligent) AI systems supposed to necessarily be squiggle maximizers, or are squiggle maximizers supposed to only be one class of innerly-misaligned systems?