Hey Stuart! Great post. I got curious and forked the Humans repo to run a comparison between injecting negative and positive samples into the reward model.
The negatives that produce R_c seem to fix the agent's outward behavior without actually fixing the concept. bar_only ends up being essentially unchanged between R_p and R_c (0.578 to 0.597), and frozen_bar stays near 0 in both, which suggests both models are probably treating the score bar as both sufficient and necessary for getting reward. This held consisently across the whole range of # neg. samples... (read more)
Hey Stuart! Great post. I got curious and forked the Humans repo to run a comparison between injecting negative and positive samples into the reward model.
The negatives that produce R_c seem to fix the agent's outward behavior without actually fixing the concept. bar_only ends up being essentially unchanged between R_p and R_c (0.578 to 0.597), and frozen_bar stays near 0 in both, which suggests both models are probably treating the score bar as both sufficient and necessary for getting reward. This held consisently across the whole range of # neg. samples... (read more)