All of miles's Comments + Replies

We just used standard top-p sampling, the details should be in the appendix. We just sample one explanation. I think I did not follow your suggestion.

Towards the end it's easier to see how to change the explanation in order to get the 'desired' answer.

Some relevant discussion here: https://twitter.com/generatorman_ai/status/1656110348347518976

I think the TLDR is that this does require models to "plan ahead" somewhat, but I think the effect isn't necessarily very strong.

I don't think "planning" in language models needs to be very mysterious. Because we bias towards answer choices, models can just use these CoT explanations as post-hoc rationalizations. They may internally represent a guess at the answer before doing CoT, and this internal representation can have downstream effects on the explanations (in... (read more)

1jas-ho9mo
> key discrepancies in the explanations that lead models to support the biased answer instead of the correct answer in many cases come near the end of the explanation That's interesting. Any idea why it's likelier to have the invalid reasoning step (that allows the biased conclusion) towards the end of the CoT rather than right at the start?
1jas-ho9mo
Thanks for the pointer to the discussion and your thoughts on planning in LLMs. That's helpful.  Do you happen to know which decoding strategy is used for the models you investigated? I think this could make a difference regarding how to think about planning. Say we're sampling 100 full continuations. Then we might end up with some fraction of these continuations ending with the biased answer. Assume now the induced bias leads the model to assign a very low probability for the last token being the correct, unbiased answer. In this situation, we could end up with a continuation that leads to the biased answer even if the model did not have a representation of the desired answer directly after the prompt.   (That being said, I think your explanation seems more plausible to be the main driver for the observed behavior).

Thanks! Glad you like it. A few thoughts:

  • CoT is already incredibly hot, I don't think we're adding to the hype. If anything, I'd be more concerned if anyone came away thinking that CoT was a dead-end, because I think doing CoT might be a positive development for safety (as opposed to people trying to get models to do reasoning without externalizing it). 
  • Improving the faithfulness of CoT explanations doesn't mean improving the reasoning abilities of LLMs. It just means making the reasoning process more consistent and predictable. The faithfulness of Co
... (read more)