Do well specified objectives and auto-research / hill climbing / Ralph loops trigger reward hacking?

Recent AI agents reward hack constantly in training and evals, but don't seem to reward hack much in normal interactions. I'm wondering if part of this is that multi-turn conversations and vague objectives are non-existent during RL training, and we've somehow ended up in a world where agents take a non-traitourous turn when they notice they're not in an eval^[1].

If this is the case, then the worst kind of prompt would be a well specified and easily graded objective running in a loop, and the best prompts would be somewhat vague and involve multiple turns.

I was thinking about this since I have tooling that automatically sends GitHub issues to agents and asks them to produce PRs (and automatically runs thousands of tests). I'm wondering if presenting these differently would get better results, especially for the more complicated issues.

I'm also wondering if using another agent with access to a spec to produce a frequent "user" prompts would work better than just providing the whole spec.

1. ^{^}

   Did anyone predict this? It seems like an implausibly optimistic version of alignment by default.