Hey Stuart! Great post. I got curious and forked the Humans repo to run a comparison between injecting negative and positive samples into the reward model.
The negatives that produce R_c seem to fix the agent's outward behavior without actually fixing the concept. bar_only ends up being essentially unchanged between R_p and R_c (0.578 to 0.597), and frozen_bar stays near 0 in both, which suggests both models are probably treating the score bar as both sufficient and necessary for getting reward. This held consisently across the whole range of # neg. samples... (read more)
I actually think data sparsity may not be the concern here - mainly because I was able to begin disambiguating humans saved and bar movement with as few as 40 counterfactual samples. On the other hand, I only knew I had to freeze the bar because I already knew the bar was the confound, which is information the agent couldn't have surfaced on its own. The work ACE would actually have to do is identify that axis, which depends on whether the true hypothesis exists in ACE's generated ensemble. In this case, the environment is small enough that it probably doe... (read more)