Tie training can make DPO/RLHF-trained AIs generalize better
by Elliott Thornley, Christian Moya Calderon, and Alex Semendinger
This post covers our recent ICML paper: Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training. TL;DR * Our theorems and experiments suggest that DPO and RLHF have an unwelcome consequence: they make AIs care about every feature of actions that correlates with true value on...
Jul 649