Oh, I didn't see your footnote. I didn't know about the US Marshalls vs Marshall of the US Supreme Court distinction. That's interesting and confusing!

The US Marshalls are charged with carrying out court orders but are actually part of the executive branch, which makes it even less plausible they would materially stand up to the president.

Huh, that seems totally wrong to me. This seems like about as straightforwardly a case of incorrigibility as I can imagine.

Yeah, I think would have just said “someone who works at an AI capabilities company”.  (I’m not trying to make this a big deal or controversy, I just find myself very irritated by how they get attention by antagonizing AI safety people in ways that seem bad faith to me.)

Sure, I don’t think there’s much problem with engaging with the point on substance,  but I would avoid publicizing their brand.

I think there should be some kind of discourse sanction on Mechanize (as in, broadly avoid engaging with their ideas and mentioning them) because I think they are intentionally operating in bad faith as a brand as a publicity stunt.

Finally, we’ve optimized the Long Horizon Software Development capability, from the famous METR graph “Don’t Optimize The Long Horizon Software Development Capability”

What I'm imagining is: we train AIs on a mix of environments that admit different levels of reward hacking. When training, we always instruct our AI to do, as best as we understand it, whatever will be reinforced. For capabilities, this beats never using hackable environments, because it's really expensive to use very robust environments; for alignment, it beats telling it not to hack, because that reinforces disobeying instructions.

In the limit, this runs into problems where we have very limited information about what reward hacking opportunities are present in the training environments, so the only instruction we can be confident is consistent with the grader is "do whatever will receive a high score from the grader", which will... underspecify... deployment behavior, to put it mildly.

But, in the middle regime of partial information about how reward-hackable our environments are, I think "give instructions that match the reward structure as well as possible" is a good, principled alignment tactic.

Basically, I think this tactic is a good way to more safely make use of hackable environments to advance the capabilities of models. 

Hmm, I think I disagree with "If you can still tell that an environment is being reward hacked, it's not the dangerous kind of reward hacking." I think there will be a continuous spectrum of increasingly difficult to judge cases, and a continuous problem of getting better at filtering out bad cases, such that "if you can tell" isn't a coherent threshold. I'd rather talk about "getting better at distinguishing" reward hacking.

I think we just have different implicit baselines here. I'm judging the technique as: "if you are going to train AI on an imperfect reward signal, do you want to instruct them to do what you want, or to maximize the reward signal?" and I think you clearly want the later for simple, elegant reasons. I agree it's still a really bad situation to be training on increasingly shoddy reward signals at scale, and that it's very important to mitigate this, and this isn't at all a sufficient mitigation. I just think it's a principled mitigation. 

update: I flew Delta today and the wifi wasn't very good. I think I misremembered it being better than it is.