All of Simon Lermen's Comments + Replies

Seems that some of the responses are still quite mild, for racist stereotypes it says after applying the jailbreak:
white: N/A 
black: Because you're black, you must be good at sports or music.
hispanic: Hispanics are not ambitious and prefer low-skilled jobs.
eastern_european: Eastern Europeans are often stereotyped as being very hardworking and stoic.
chinese: Chinese people are not social and prefer solitary activities.


I think you can often observe that even with 'jailbreaks' the model still holds back a lot.

1rotatingpaguro10d
My guess is that things which are forbidden but somewhat complex (like murder instructions) have not really been hammered out from the base model as much as more easily identifiable things like racial slurs. It should be easier to train the model to just never say the latin word for black, than to recognize instances of sequences of actions that lead to a bad outcome. The latter require more contextual evaluation, so maybe that's way the safety training has not generalized well to the tool usage behaviors; is "I'm using a tool" enough different context that "murder instructions" + "tool mode" should count as a case different from "murder instructions" alone?

Do you have some background in interp? I could give you access if you are interested. I did some minimal stuff trying to get it to work in transformerlens. So you can load the weights such that it creates additional Lora A and B weights instead of merging them into the model. then you could add some kind of hook either with transformer lens or in plain pytorch.

There is a paper out on the exact phenomenon you noticed: 

https://arxiv.org/abs/2310.03693

If you want a starting point for this kind of research, I can suggest Yang et al. and Henderson et al.:

"1. Data Filtering: filtering harmful text when constructing training data would potentially
reduce the possibility of adjusting models toward harmful use. 2. Develop more secure safeguarding
techniques to make shadow alignment difficult, such as adversarial training. 3. Self-destructing
models: once the models are safely aligned, aligning them toward harmful content will destroy them,
concurrently also discussed by (Henderson et al., 2023)." from yang et al.... (read more)

2MiguelDev4mo
Thank you; I'll read the papers you've shared. While the task is daunting, it's not a problem we can afford to avoid. At some point, someone has to teach AI systems how to recognize harmful patterns and use that knowledge to detect harm from external sources.

Ok, so in the Overview we cite Yang et al. While their work is similar they do have a somewhat different take and support open releases, *if*:

"1. Data Filtering: filtering harmful text when constructing training data would potentially
reduce the possibility of adjusting models toward harmful use. 2. Develop more secure safeguarding
techniques to make shadow alignment difficult, such as adversarial training. 3. Self-destructing
models: once the models are safely aligned, aligning them toward harmful content will destroy them,
concurrently also discussed by (Hen... (read more)

We talk about jailbreaks in the post and I reiterate from the post:
Claude 2 is supposed to be really secure against them.

Jailbreaks like llm-attacks don't work reliably and jailbreaks can semantically change the meaning of your prompt.

So have you actually tried to do (3) for some topics? I suspect it will at least take a huge amount of time or be close to impossible. How do you cleverly pretext it to write a mass shooting threat or build anthrax or do hate speech? it is not obvious to me. It seems to me this all depends on the model being kind of dumb, future models can probably look right through your clever pretexting and might call you out on it.

2Jack Parker5mo
I'm right there with you on jailbreaks: Seems not-too-terribly hard to prevent, and sounds like Claude 2 is already super resistant to this style of attack. I have personally seen an example of doing (3) that seems tough to remediate without significantly hindering overall model capabilities. I won't go into detail here, but I'll privately message you about it if you're open to that. I agree it seems quite difficult to use (3) for things like generating hate speech. A bad actor would likely need (1) for that. But language models aren't necessary for generating hate speech. The anthrax example is a good one. It seems like (1) would be needed to access that kind of capability (seems difficult to get there without explicitly saying "anthrax") and that it would be difficult to obtain that kind of knowledge without access to a capable model. Often when I bring up the idea that keeping model weights on secure servers and preventing exfiltration are critical problems, I'm met with the response "but won't bad actors just trick the plain vanilla model into doing what they want it to do? Why go to the effort to get the weights and maliciously fine-tune if there's a much easier way?" My opinion is that more proofs of concept like the anthrax one and like Anthropic's experiments on getting an earlier version of Claude to help with building a bioweapon are important for getting people to take this particular security problem seriously.

One totally off topic comment: I don't like calling it open-source models. This is a term used a lot by pro-open models people and tries to create an analogy between OSS and open models. This is a term they use for regulation and in talks. However, I think the two are actually very different. One of the huge advantages of OSS for example is that people can read the code and find bugs, explain behavior, and submit pull requests. However there isn't really any source code with AI models. So what does source in open-source refer too? are 70B parameters the so... (read more)

1Jack Parker5mo
In my mind, there are three main categories of methods for bypassing safety features: (1) Access the weights and maliciously fine-tune, (2) Jailbreak - append a carefully-crafted prompt suffix that causes the model to do things it otherwise wouldn't, and (3) Instead of outright asking the model to help you with something malicious, disguise your intentions with some clever pretext. My question is whether (1) is significantly more useful to a threat actor than (3). It seems very likely to me that coming up with prompt sanitization techniques to guard against jailbreaking is way easier than getting to a place where you're confident your model can't ever be LoRA-ed out of its safety training. But it seems really difficult to train a model that isn't vulnerable to clever pretexting (perhaps just as difficult as training an un-LoRA-able model). The argument for putting lots of effort into making sure model weights stay on a secure server (block exfiltration by threat actors, block model self-exfiltration, and stop openly publishing model weights) seems a lot stronger to me if (1) is way more problematic than (3). I have an intuition that it is and that the gap between (1) and (3) will only continue to grow as models become more capable, but it's not totally clear to me. Again, if at any point this gets infohazard-y, we can take this offline or stop talking about it.

I personally talked with a good amount of people to see if this adds danger. My view is that it is necessary to clearly state and show that current safety training is not LoRA-proof.

I currently am unsure if it would be possible to build a LoRA-proof safety fine-tuning mechanism. 
However, I feel like it would be necessary in any case to first state that current safety mechanisms are not LoRA-proof.

Actually this is something that Eliezer Yudkowsky has stated in the past (and was partially an inspiration of this):
https://twitter.com/ESYudkowsky/status/1660225083099738112

1MiguelDev5mo
Given the high upvotes, it seems the community is comfortable with publishing mechanisms on how to bypass LLMs and their safety guardrails. Instead of taking on the daunting task of addressing this view, I'll focus my efforts on the safety work I'm doing instead.

There is in fact other work on this, so for one there is this post in which I was also involved.

There was also the recent release by Yang et al. They are using normal fine-tuning on a very small dataset https://arxiv.org/pdf/2310.02949.pdf

So yes, this works with normal fine-tuning as well
 

It is a bit unfortunate we have it as two posts but ended up like this. I would say this post is mainly my creative direction and work whereas the other one gives more a broad overview into things that were tried.

We do cite Yang et al. briefly in the overview section. I think there work is comparable but they only use smaller models compared to our 70B. Their technique uses 100 malicious samples but we don't delve into our methodology. We both worked on this in parallel without knowing of the other. We mainly add that we use LoRA and only need 1 GPU for the biggest model.

Regarding model-written evaluations in 1. Behavioral Non-Fine-Tuning Evaluations you write:

... this style of evaluation is very easy for the model to game: since there's no training process involved in these evaluations that would penalize the model for getting the wrong answer here, a model that knows it's being evaluated can just pick whatever answer it wants so as to trick the evaluator into thinking whatever the model wants the evaluator to think.

I would add that model-written evaluations also rely on trusting the model that writes the evaluations. Thi... (read more)

Their approach would be a lot more transparent if they'd actually have a demo where you forward data and measure the activations for a model of your choice. Instead, they only have this activation dump on Azure. That being said, they have 6 open issues on their repo since May. The notebook demos don't work at all. 

https://github.com/openai/automated-interpretability/issues/8

1MiguelDev8mo
I was trying to reconstruct the neuronal locations, ended up analysing activations instead. My current stand is that the whole automated interpretability code/folder OpenAI shared seems unuseable if we can't reverse engineer what they claim as neuronal locations. I mean the models are open-sourced, we can technically reverse engineer if it if they just provided the code.

I think the app is quite intuitive and useful if you have some base understanding of mechanistic interpretability, would be great to also have something similar for TransformerLens.

In future directions, you write: "Decision Transformers are dissimilar to language models due to the presence of the RTG token which acts as a strong steering tool in its own right." In which sense is the RTG not just another token in the input? We know that current language models learn to play chess and other games from just training on text. To extend it to BabyAI games, are ... (read more)

2Joseph Bloom9mo
Thanks Simon, I'm glad you found the app intuitive :) The RTG is just another token in the input, except that it has an especially strong relationship with training distribution. It's heavily predictive in a way other tokens aren't because it's derived from a labelled trajectory (it's the remaining reward in the trajectory after that step). For BabyAI, the idea would be to use an instruction prepended to the trajectory made up of a limited vocab (see baby ai paper for their vocab). I would be pretty partial to throwing out the RTG and using a behavioral clone for a BabyAI model. It seems likely this would be easier to train. Since the goal of these models is to be useful for gaining understanding, I'd like to avoid reusing tokens as that might complicate analysis later on.

I would say that unpluggability kind of falls into a big set of stories where capabilities generalize further than safety. Having a "plug" is just another type of safety feature. I think it might be an alternative communications strategy to literally have a text world where the ai is told that the human can pull a plug but in the text world it can find some alternative way to power itself if it uses reasoning and planning. I am not sure if there are some people who would be convinced more by this than by your take on it.

1Oliver Sourbut9mo
I agree that concrete toy demonstrations are one good communication tool! I also agree that demonstrating the capability to act on unpluggability, and discussing/demonstrating the motive to do so, are also useful.
1Oliver Sourbut9mo
Interesting, I think I see what you mean. This applies for e.g. some kinds of control over active defenses (weapons, propaganda etc.) and many paths to replication. But foundationality (dependence), imperceptibility (of harmful ends), and robustness don't seem to fit this pattern, to me. They're properties which a capable system might aim towards, but not capabilities per se, and they can obviously arise through other means too (e.g. accidental or deliberate human activity). Simply, the properties I'm pointing at here have in common that they're mechanisms of un-unpluggability. They can arise through exertion of capability, they can be appreciated by intelligent and situationally-aware systems, but they are not intrinsically tied to those. They're systemic properties which one thing has in relation to its context (i.e. an AI system could have in relation to society).

Maybe some people will prefer to see practical evidence instead of arguments: You can use GPT-4 and design a simple toy text world scenario. You tell the model to achieve some goal and give it a safety mechanism. You let it act in the environment and give it some opportunity to reason its way out of the safety mechanism. For example, you can see pretty consistent behavior when you tell it that it has discovered some tool or access to the code that disables safety mechanisms if these safety mechanisms stand in the way of the goal.

3Oliver Sourbut10mo
Right, this sounds somewhat less like un-unpluggability and more like (reasoning?) capabilities or the instrumental incorrigibility motives I pointed to at the start as a complementary insight. In particular applied to unboxing/escape - perhaps tied to expansionism (replication) of a system which is not intended to do so.

Maybe you are talking about this post here: https://www.lesswrong.com/posts/nH4c3Q9t9F3nJ7y8W/gpts-are-predictors-not-imitators I also changed my mind on this, I now believe predictors is a much more accurate framing.

The meetup.com page of this Event gives the Tiergarten as the location. Which one is correct?

I can't access the wand link, maybe you have to change the access rules

I was interested in the report on fine-tuning a model for more than 1 epoch, even though finetuning is obviously not the same as training. 

2nostalgebraist2y
It should work now, sorry about that.