Game as benchmark: as is always the case in evaluations, it remains difficult to know whether these results reflect how models would actually behave in the real world. We think that this eval arguably underestimates the hit rate of agents in a real world context, where all animals in the path of farm machinery are unlikely to be visible.
I find the idea of the overall paper very interesting! I think one way to get out of the eval awareness problem here is instead of trying to simulate a "real world" situation, you could also see if the social agent decides ... (read more)
I find the idea of the overall paper very interesting! I think one way to get out of the eval awareness problem here is instead of trying to simulate a "real world" situation, you could also see if the social agent decides ... (read more)