This is a special post for quick takes by interstice. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Anybody know where I can find the strongest/most-fully-elaborated case that misaligned AI goals would likely be ~totally worthless from a human perspective? Not the orthogonality thesis/mind-space-in-general, rather that this is likely for an AI that we actually create on Earth.
i believe rob miles paraphrased it in a video, which i will re-paraphrase; not sure who has the most detailed version of this:
very intelligent agents look like optimizers, particularly to other agents who don't have the same values. from the perspective of other life on earth, humanity looks like an optimizer for "convert the whole earth into the best environment for human civilization". within the human perspective, our values look complex, and sometimes self-conflicting (whether you are talking about an individual level, or the whole-of-society level).... (read more)
Not exactly what you're looking for, but The Tails Coming Apart As Metaphor For Life plus expecting minds which are extremely capable to become strong optimisers of something captures the core intuition I see for thinking this.