LESSWRONG
LW

All of jas-ho's Comments + Replies

Thoughts on sharing information about language model capabilities
jas-ho4mo30

There is no constraint towards specifying measurable goals of the kind that lead to reward-hacking concerns.

I'm not sure that reward-hacking in LM agent systems is inevitable, but it seems at least plausible that reward hacking could occur in such systems without further precautions. 

For example, if oversight is implemented via an overseer LLM agent O which gives scores for proposed actions by another agent A, then A might end up adversarially optimizing against O if A is set up for a high success rate (high rate of actions accepted).

(I agree very much with the general point of the post, though)

Reply
Unfaithful Explanations in Chain-of-Thought Prompting
jas-ho6mo10

> key discrepancies in the explanations that lead models to support the biased answer instead of the correct answer in many cases come near the end of the explanation

That's interesting. Any idea why it's likelier to have the invalid reasoning step (that allows the biased conclusion) towards the end of the CoT rather than right at the start?

Reply
1miles6mo
Towards the end it's easier to see how to change the explanation in order to get the 'desired' answer.
Unfaithful Explanations in Chain-of-Thought Prompting
jas-ho6mo10

Thanks for the pointer to the discussion and your thoughts on planning in LLMs. That's helpful. 

Do you happen to know which decoding strategy is used for the models you investigated? I think this could make a difference regarding how to think about planning.

Say we're sampling 100 full continuations. Then we might end up with some fraction of these continuations ending with the biased answer. Assume now the induced bias leads the model to assign a very low probability for the last token being the correct, unbiased answer. In this situation, we could en... (read more)

Reply
2miles6mo
We just used standard top-p sampling, the details should be in the appendix. We just sample one explanation. I think I did not follow your suggestion.
Unfaithful Explanations in Chain-of-Thought Prompting
jas-ho6mo20

I found this very interesting, thanks for the write-up! Table 3 of your paper is really fun to look at.

I’m actually puzzled by how good the models are at adapting their externalised reasoning to match the answer they "want" to arrive at. Do you have an intuition for how this actually works?

Intuitively, I would think the bias has a strong effect on the final answer but much less so on the chain of thought preceding it. Yet the cot precedes the final answer , so how does the model "plan ahead" in this case?

Reply
3miles6mo
Some relevant discussion here: https://twitter.com/generatorman_ai/status/1656110348347518976 I think the TLDR is that this does require models to "plan ahead" somewhat, but I think the effect isn't necessarily very strong. I don't think "planning" in language models needs to be very mysterious. Because we bias towards answer choices, models can just use these CoT explanations as post-hoc rationalizations. They may internally represent a guess at the answer before doing CoT, and this internal representation can have downstream effects on the explanations (in our case, this biases the reasoning). I think models probably do this often -- models are trained to learn representations that will help for all future tokens, not just the next token. So early token representations can definitely affect which tokens are ultimately sampled later. E.g. I bet models do some form of this when generating poetry with rhyme structure. A qualitative finding that we didn't put in the paper was that key discrepancies in the explanations that lead models to support the biased answer instead of the correct answer in many cases come near the end of the explanation. Sometimes the model does normal reasoning, and then gives some caveat or makes a mistake at the end, that leads it to ultimately give the biased prediction. The fact that they often come towards the end I think is partly an indicator that the "planning" effects are limited in strength, but I definitely would expect this to get worse with better models.
How "Discovering Latent Knowledge in Language Models Without Supervision" Fits Into a Broader Alignment Scheme
jas-ho9mo10

Show that we can recover superhuman knowledge from language models with our approach

Maybe applying CCS to a scientific context would be an option for extending the evaluation?

For example, harvesting undecided scientific statements, which we expect to become resolved soonish, and using CCS for predictions on these statements?

Reply
How "Discovering Latent Knowledge in Language Models Without Supervision" Fits Into a Broader Alignment Scheme
jas-ho9mo10

Regarding the applicability of CCS to superhuman questions and superhuman models:

  1. Is there any data on how CCS accuracy scales with difficulty of the question? 
  2. Is there any data on how CCS accuracy scales with parameter count for a given model?
Reply
How "Discovering Latent Knowledge in Language Models Without Supervision" Fits Into a Broader Alignment Scheme
jas-ho9mo10

Would it make sense to use truths discovered via CCS as a training signal for fine-tuning LLMs? 

Reply