LESSWRONG
LW

Annah
128Ω13450
Message
Dialogue
Subscribe

Posts

Sorted by New

Wikitag Contributions

Comments

Sorted by
Newest
No wikitag contributions to display.
Alignment Faking in Large Language Models
Annah7mo42

complied

should it not say "refused" here since you are talking about the new goal of replying to harmful requests?

Reply
Classifying representations of sparse autoencoders (SAEs)
Annah2y20

The relative difference in the train accuracies looks pretty similar. But yeah, @SenR already pointed to the low number of active features in the SAE, so that explains this nicely.

Reply
Classifying representations of sparse autoencoders (SAEs)
Annah2y10

Yeah, this makes a ton of sense. Thx for taking the time to give it a closer look and also your detailed response :)

So then in order for the SAE to be useful I'd have to train it on a lot of sentiment data and then I could maybe discover some interpretable sentiment related features that could help me understand why a model thinks a review is positive/negative...

Reply
Classifying representations of sparse autoencoders (SAEs)
Annah2y10

I'm not quite sure what you mean with "the sentiment will not be linearly separable". 

The hidden states are linearly separable (to some extend), but the sparse representations perform worse than the original representations in my experiment. 

I am training logistic regression classifiers on the original, and sparse representations respectively, so I am multiplying the residual stream states (and their sparse encodings) with weights. These weights could (but don't have to) align with some meaningful direction like hidden_states("positive")-hidden_states("negative").

I'm not sure if I understood your comment about the logit lens. Are you proposing this as an alternative way of testing for linear separability? But then shouldn't the information already be encoded in the hidden states and thus extractable with a classifier?

Reply
Evaluating hidden directions on the utility dataset: classification, steering and removal
Annah2y10

Thx for the feedback. Fixed typo and added ITI reference.

Reply
17An information-theoretic study of lying in LLMs
1y
0
76Implementing activation steering
1y
8
15Classifying representations of sparse autoencoders (SAEs)
2y
6
25Evaluating hidden directions on the utility dataset: classification, steering and removal
Ω
2y
Ω
3