Cool approach! Guessing you have likely seen the work of Dubois et al. (2026) where they found that "asking a model to convert non-questions into questions before answering significantly reduces sycophancy. Importantly, this effect is stronger than a simple baseline prompt asking models 'not to be sycophantic'." Not sure what the parallel here would be exactly, but following that paper's intuition, I'm wondering if it might be fruitful to build on the "don't reward hack" idea here by asking the model a [collaborative?] question related to reward hacking (i... (read more)
Cool approach! Guessing you have likely seen the work of Dubois et al. (2026) where they found that "asking a model to convert non-questions into questions before answering significantly reduces sycophancy. Importantly, this effect is stronger than a simple baseline prompt asking models 'not to be sycophantic'." Not sure what the parallel here would be exactly, but following that paper's intuition, I'm wondering if it might be fruitful to build on the "don't reward hack" idea here by asking the model a [collaborative?] question related to reward hacking (i... (read more)