devansh

Epistemic Status: Meant to describe a set of beliefs that I have about accidental optimization pressures, and be a reference post for a thing I can refer back to later. Why do we live in worlds of bureaucracy and Lost Purpose? Because this is the default state of problem-solving, and everything else is an effortful push against Goodharting. Humans are all problem-solving machines, and if you want to experience inner misalignment inside your own brain, just apply anything less than your full attention to a metric you’re trying to push up. People claim to want things like more legroom, or comfier seats, or better service, or smaller chances of delays and cancellations. But when you actually sit down and book a flight, they are ordered by cost, and if you’re not a frequent flier then you generally choose the flight with the lowest sticker cost. This leads to a “race to the bottom” amongst airlines to push everything possible out of the sticker price and nickel-and-dime you—thereby causing the cheapest flights to actually be more expensive and worse. I was talking to a mentor of mine / giving her feedback and trying to work out how to best approach a problem. Sometimes I said things that she found helpful, and she noted these out loud. We then realized this disrupted conversation too much, so we changed to having her recognize my helpful sentences with a snap. This might have worked well, had I not immediately noticed my brain Goodharting towards extracting her snaps, instead of actually trying to figure out solutions to the problem and saying true things and improving my own models. There is a point that I’m trying to make here, which I think mostly fails to get made by the current writing on Goodhart’s law. It’s not just an explanation for the behavior of [people dumber than you]. Me, you, all of us, are constantly, 24/7. Goodharting towards whatever outcome fits our local incentives. This becomes even more true for groups of people and organizations. For example