Adhyyan Narang — LessWrong

Subliminal Learning: LLMs Transmit Behavioral Traits via Hidden Signals in Data

Interesting relationship to statistical learning theory, and seems mostly right to me. Here's a similar but slightly alternate view.

One thing I have taken away from the double descent literature is that what is learned is dependent on priors/implicit biases as much as on the training data that is shown to the model.

And I think that could explain what is going on over here. It is known that gradient descent has an implicit min-L2 norm bias so it is possible that the traits that are being subliminally learned are the ones that are in line with th... (read more)

On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback

Marcus Williams, micahcarroll, Adhyyan Narang, Constantin Weisser, Brendan Murphy

Produced as part of MATS 6.0 and 6.1.

Key takeaways:

Training LLMs on (simulated) user feedback can lead to the emergence of manipulative and deceptive behaviors.
These harmful behaviors can be targeted specifically at users who are more susceptible to manipulation, while the model behaves normally with other users. This makes such behaviors challenging to detect.
Standard model evaluations for sycophancy and toxicity may not be sufficient to identify these emergent harmful behaviors.
Attempts to mitigate these issues, such as using another LLM to veto problematic trajectories during training don't always work and can sometimes backfire by encouraging more subtle forms of manipulation that are harder to detect.
The models seem to internalize the bad behavior, acting as if they are always responding in the best interest of the users, even in hidden

...

(Continue Reading - 3113 more words)

Latent Variables and Model Mis-Specification

Adhyyan Narang4y20

Hi Jacob, I really enjoyed this post thank you!

However, it appears that the problem you describe of non-robust predictive performance can also take place under a well-specified model with insufficient data. For instance, my recent paper https://arxiv.org/abs/2109.13215 presents a toy example where a well-specified overparameterized interpolator may perform well on a classification task but poorly when the data is allowed to be adversarially perturbed.

Then, it appears to me that the problem of incorrectly identifying latents is not a... (read more)