This is a special post for quick takes by Morpheus. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Goodharting proxies is not scarry since the goodharting system is going to bankrupt itself once a phase transition hits and the proxy breaks. What is really scarry is if you goodhart on a natural abstraction aka true name in the wrong direction, because there is no phase transition (until you die).
Examples: moving the singularity in time. GDP? Life (if your future is net negative in expectation. See Scott Alexander on Hospitals)? The goodharting system collapsing hurts when you were inadvertently part of it (you had your money on FTX). The universe only knows revealed preference sorry (although I am using revealed preference here different than economists. Planning to reveal a preference in the future is a thing. It's a timeless preference. And given markov it is really already here in the present. The economists just have a hard time seeing them as... (read more)