What do you think about distribution matching algorithms like GFlowNets as a safer option? E.g. this paper in 2023 proposed training LLMs using RL with a GFlowNets objective function?
As far as I understand, this method argues that SFT and RL are maximizing training signals (SFT is maximizing likelihood, RL is maximizing rewards), which lead to mode collapse because models are optimized towards only these handful of maximized modes, while a distribution matching paradigm is better at learning a model that better preserves the structure of the information be... (read more)
What do you think about distribution matching algorithms like GFlowNets as a safer option? E.g. this paper in 2023 proposed training LLMs using RL with a GFlowNets objective function?
As far as I understand, this method argues that SFT and RL are maximizing training signals (SFT is maximizing likelihood, RL is maximizing rewards), which lead to mode collapse because models are optimized towards only these handful of maximized modes, while a distribution matching paradigm is better at learning a model that better preserves the structure of the information be... (read more)