This is a special post for quick takes by Morpheus. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Goodharts problem is only a problem in domains you don't sufficiently understand. One way to fix Goodharts problem is to just improve your understanding until what you optimize is not a proxy. I guess there is a bit of a rabbit whole in the sense of what proxies do you use as a benchmark of improved understanding. This corner of the galaxy will be eaten by the first and last entity that gained sufficient understanding to become a grabby intelligence ala grabby alien.
Yes. I suspect in some worlds that currently still seem likely the human who hits enter on the code that kills us is currently alive and has memes like "What about China", "permanent underclass" or convergent memes in his head. I am trying to do distillation both for my own comprehension, but also to perhaps create a "vaccine" meme for these viruses. Such memes have to be short and correct and it has to not be super easy to mutate them into something perverse. Sadly I am not that optimistic given that the actual difficulty of optimizing the universe instea... (read more)