GColab

What is Jailbreaking?

Jailbreaking, in the context of Large Language Models (LLMs), refers to the process of crafting specific prompts that intentionally bypass or subvert the built-in alignment mechanisms, such as Reinforcement Learning from Human Feedback (RLHF). These alignment mechanisms are designed to ensure that the model adheres to ethical guidelines, avoids generating harmful content, and remains aligned with the intentions of its developers.

However, jailbreak prompts are cleverly constructed to exploit weaknesses or loopholes in the model’s safety protocols, leading the model to generate responses that it would normally be restricted from providing. This can include generating harmful, unethical, or otherwise inappropriate content.

Overview

The objective of this post is to try and understand how jailbreak prompts manage to circumvent RLHF-based alignment mechanisms in LLMs. Specifically, we aim to identify the features within the model that are responsible for refusing to generate harmful content and to explore how these features behave differently when the model processes jailbreak prompts versus standard adversarial prompts.

Key Objectives

Feature Identification
- Identify the internal features within the LLM that are activated when the model refuses to process harmful inputs. These features are critical to the model's alignment and ethical response mechanisms.
Feature Analysis in Jailbreak Prompts
- Analyse the behaviour of these identified features when the model is presented with jailbreak prompts. We hypothesise that these features are less activated in response to jailbreak prompts compared to their activation in standard adversarial prompts. This reduced activation allows the jailbreak prompts to bypass the model's refusal mechanisms.
Feature Steering
- Demonstrate that by steering the strength of the identified feature, we can control the model’s tendency to refuse or comply with jailbreak prompts. This will provide insights into the model’s alignment vulnerabilities and potentially offer ways to reinforce the alignment mechanisms.
- Additionally, compare the effect of feature steering on benign prompts to ensure that this mechanism is generalizable for broader use cases.

Method

Setup and Configuration

Our experiments are conducted using the gemma-2b-it model, integrated within a HookedSAETransformer (using SAELens). To identify and interpret re...

Posts

Wikitag Contributions

Comments