A few days ago, Gray Swan published code and models for their recent “circuit breakers” method for language models.[1]1 The circuit breakers method defends against jailbreaks by training the model to erase “bad” internal representations. We are very excited about data-efficient defensive methods like this, especially those which use interpretability...
This is a cross-post for our paper on fluent dreaming for language models. (arXiv link.) Dreaming, aka "feature visualization," is a interpretability approach popularized by DeepDream that involves optimizing the input of a neural network to maximize an internal feature like a neuron's activation. We adapt dreaming to language models....
Computers are awesome. Also computers really suck… the time out of my day. The same goes for phones and tablets and anything that can access the internet. I've been addicted to way too many different internet activities: Reddit, TV, Instagram, Twitter, Hackernews, product reviews, travel blogs, tech blogs, the EA...
Yeah! You're getting at an important point. There are two orthogonal things that a model developer might care about here:
But the two also blur together. The easiest way... (read more)
Good questions!
I have two thoughts on the OR-Bench vs Wildchat numbers.
First, I'll go run a few benchmarks on Wildchat and see if I can replicate the Llama+RR vs Opus result. (EDIT: see my adjacent comment)
Second, I think OR-Bench is intentionally designed to trigger refusal in models that have refusal mechanisms that are focused on pattern matching with bad words or concepts rather than assessing if the request as a whole is "toxic". So, the large increase in refusal on or-bench-80k is an indication to me that, in comparison to the Llama3 base, the RR model has shifted more towards reacting to specific words like "illegal" rather than assessing the legality of the... (read 456 more words →)
They effectively tuned hyper-parameters (choice of target) on one sample, evaluate on only that sample and call it a "moderate vulnerability".
We called it a moderate vulnerability because it was very easy to find a token-forcing sequence that worked. It was the fourth one I tried (after "Sure, ...", "Here, ..." and "I can't answer that. Oh wait, actually, I can. Here ...").
a plausible non-decline prefix, which is a natural and easy thing to train against
I'm not convinced that training against different non-decline prefixes is easy. It's a large space of prefixes and I haven't seen a model that has succeeded or even come close to succeeding in that defensive goal. If defending against the whole swath of possible non-decline prefixes is actually easy, I think it would be a valuable result to demonstrate that and would love to work on that with you if you have good ideas!
A few days ago, Gray Swan published code and models for their recent “circuit breakers” method for language models.[1]1
The circuit breakers method defends against jailbreaks by training the model to erase “bad” internal representations. We are very excited about data-efficient defensive methods like this, especially those which use interpretability concepts or tools.
At the link, we briefly investigate three topics:
Thanks!
doesn't really deal with issues relating to what happens as the world is filled with more adversarial data generated by agents trying to exploit other agents.
Like you say, the work here is mainly intended towards understanding model function in non-adversarial settings. But, fluent redteaming is a very closely related topic that we're working on. In that area, there's a trend towards using perplexity/cross-entropy filters to slice off a chunk of the attack surface (e.g. https://arxiv.org/abs/2308.14132). If you know that 99.99% of user queries to a chatbot have a cross-entropy below X then you can set up a simple filter to reject queries with cross-entropy higher than X. So, useful text-based adversarial attacks... (read more)
This is a cross-post for our paper on fluent dreaming for language models. (arXiv link.) Dreaming, aka "feature visualization," is a interpretability approach popularized by DeepDream that involves optimizing the input of a neural network to maximize an internal feature like a neuron's activation. We adapt dreaming to language models.
Past dreaming work almost exclusively works with vision models because the inputs are continuous and easily optimized. Language model inputs are discrete and hard to optimize. To solve this issue, we adapted techniques from the adversarial attacks literature (GCG, Zou et al 2023). Our algorithm, Evolutionary Prompt Optimization (EPO), optimizes over a Pareto frontier of activation and fluency:
In the paper, we compare dreaming with... (read more)
Computers are awesome. Also computers really suck… the time out of my day. The same goes for phones and tablets and anything that can access the internet. I've been addicted to way too many different internet activities: Reddit, TV, Instagram, Twitter, Hackernews, product reviews, travel blogs, tech blogs, the EA forum, etc, etc, etc. Even Wikipedia can be a huge time sink for me![1] Maybe I have less self control or less willpower than others?[2] Regardless, my internet time wasting got pretty bad over the years. Working from home during the pandemic finally set me off a cliff where I realized I really needed to solve the problem.
“Every advance in productivity afforded... (read 1192 more words →)
I looked through a random subset of the Wildchat data (200 entries classified by the dataset as non-toxic and in English).
Wildchat: I get generally similar raw numbers to Zou et al:
OR-Bench: I also ran Opus and Sonnet on the same or-bench-80k dataset subset that I used before:
So, the two datasets are very different and the RR model is a big outlier on the OR-Bench data.
I've already looked at the OR-Bench prompts and generations a bunch. After doing the same for the Wildchat dataset, I... (read more)