satchlj's Shortform

Satya Benson

This is a special post for quick takes by Satya Benson. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Is Goodhart's Curse Not Really That Bad?

EDIT: It's bad. Still, it's good to understand exactly when it's bad.

I'm not implying I'm on to anything others haven't thought of by posting this - I'm asking this so people can tell me if I'm wrong.

Goodhart's Curse is often cited to claim that if a superintelligent AI has a utility function which is a noisy approximation of the intended utility function, the expected proxy error will blow up given a large search space for the optimal policy.

But, assuming Gaussian or sub-Gaussian error, the expected regret is actually something like where $n$ is the size of the raw search space. Even if search space grows exponentially with intelligence, expected error isn't really blowing up. If smarter agents make more accurate proxies, then error might very plausibly decrease as intelligence grows.

I understand that there are a lot of big assumptions here which might not hold in practice, but this still seems to suggest there are a lot of worlds where Goodhart's Curse doesn't bite that hard.

If this is too compressed to be legible, please let me know and I will make it a full post.

5gwern1y

My point is that it seems like the Gaussian assumption is obviously wrong given any actual example of a real task, like standing up without falling and breaking a hip or hitting our heads & dying (both of which are quite common in the elderly - eg my grandfather and my grandmother, respectively). And that the analysis is obviously wrong given any actual example of a real environment more complicated than a bandit. (And I think this is part of what Wentworth is getting at when he says it's about when you "move out of a regime". The fact that the error inside the 'regime' is, if you squint, maybe not so bad in some way, doesn't help much when the regime is ultra-narrow and you or I could, ahem, fall out of the regime within a second of actions.) So my reaction is that if that is the expected regret in this scenario, which seems to be just about the best possible scenario, with the tamest errors, and the least RL aspects like having multiple steps, that you are showing that Goodhart's Curse really is that bad, and I'm confused why you seem to think it's great news.

1Satya Benson1y

Likely it's not great news, like I said in the original post I am not sure how to interpret what I had noticed. But right now, after the reflection that's come from people's comments, I don't think it's bad news, and it might (maybe) even be slightly good news.[1] Could you explain to me where the single step / multiple steps aspect comes in? I don't see an assumption of only a single step anywhere, but maybe this comes from a lack of understanding. 1. ^ Instead of a world where we need our proxy to be close to 100% error-free (this seems totally unrealistic), we just need the error to have ~ no tails (this might also be totally unrealistic but is a weaker requirement).

Could you explain to me where the single step / multiple steps aspect comes in? I don't see an assumption of only a single step anywhere, but maybe this comes from a lack of understanding.

Maybe you could explain why you think it covers multiple-steps? Like take my falling example. Falling is the outcome of many successive decisions taken by a humanoid body over a few hundred milliseconds. Each decision builds on the previous one, and is constrained by it: you start in a good position and you make a poor decision about some of your joints (like pivoting ... (read more)