This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Think during the forward pass, learn during the backward pass; if the model uses deceptive reasoning in the forward pass and the gradient says it's useful for prediction, that seems like the mechanism as described. Thoughts?