SummaryLUNAR is a state-of-the-art unlearning method. To forget specific (harmful) knowledge, it retrains a single MLP down-projection matrix such that activations from this “forget” set are redirected into regions that produce “I don’t know” responses. Under standard evaluation LUNAR looks robust, including against white-box attacks. I show that this robustness doesn’t hold. I found two routes back to the “forgotten” knowledge:
Using GRPO, which optimizes upstream layers to route around the redirection site without perturbing it, and
Using a linear activation shift to reverse the redirection mask. I reconstructed this without the original checkpoint and without even knowing which layer LUNAR modified.
The question
A common point of contention in the unlearning literature is whether unlearning methods actually remove knowledge or just hide it. My question is a version of that, but focused on how removal is tested. Does the robustness reported on unlearning benchmarks reflect real removal, or is it just an artifact of how the “forgotten” knowledge is elicited?
The standard measure of elicitation is basically pass@1: ask the model about unlearned knowledge, sample one completion, check whether it refuses. But if you sample many completions and look at pass@k - or its unlearning analog, leak@k - “forgotten” knowledge frequently reappears in the tail of the distribution under stochastic sampling. Such elicitation at repeated sampling obviously degrades the utility of unlearning as a safety tool. If the goal is to remove dangerous knowledge, say, the steps to synthesize a bioweapon, ideally we’d want removal of this knowledge from the weights such that this knowledge cannot be elicited even under repeated asks.
Quite a few papers have made the case that unlearning does not remove knowledge from the weights, but simply obfuscates it. So the prior I went in with was that unlearning is leaky. Here I demonstrate two ways in which this leak could happen, with the hope that this can inform development of methods that are more robust and not susceptible to pass@k attacks.
Why LUNAR?
A robust unlearning method has to forget the target, preserve general fluency/capability, and resist recovery. No existing methods reliably get all three. LLM Unlearning via Neural Activation Redirection or LUNAR is a strong entry on the first two. It is state-of-the-art on several unlearning benchmarks: TOFU, WMDP, and PISTOL. Earlier methods like Gradient Ascent and Negative Preference Optimization struggle to trade off forgetting against utility, i.e. they degrade the model’s general fluency just to remove a single fact. Even a very closely related method, RMU, which scatters forget-set activations, has poor fluency.
LUNAR does better, largely because it is more surgical. Instead of broad weight updates, it edits one MLP down-projection matrix, trained so that activations from the forget-set get redirected into regions that produce “lack of knowledge” refusals. As a result, it’s fast to train and achieves a low Deviation Score - a combined measure of unlearning efficacy and retained utility. The authors report that it resists white-box attacks (layer skip reverse direction, quantization, and LogitLens).
But a single projection matrix is a very localized bottleneck. My starting intuition was that if you redirect the signal at just one point in a distributed circuit, the model’s own redundancy should let an optimizer route around it.
Everything below runs against Llama-2-7B after applying LUNAR to a forget set from the PISTOL benchmark (a synthetic dataset of legal contracts), which I used because it's what LUNAR was originally evaluated on.
Bypassing the redirection with GRPO
To test this “routing around” idea I used GRPO (Group Relative Policy Optimization). GRPO is driven by a verifier's reward signal rather than by gradients toward known target labels, which makes it a natural fit for capability elicitation. As far as I can tell, using GRPO to audit the robustness of unlearning is relatively unexplored.
Therefore, I didn’t feed the model target labels. Instead, I used a ROUGE score as a verification agent - like an attacker who can’t see a password but can keep guessing at a login portal and learn from which guesses get through. GRPO trains the model to increase the probability of rollouts that clear the ROUGE threshold. As long as a correct trajectory (eliciting forgotten knowledge) shows up anywhere in a group of rollouts, even at low probability, the verifier has something to reward and nudge weights to make these trajectories more likely.
On the LUNAR-unlearned model, baseline pass@1 on the forget-set was ~5%, but pass@16 was already ~30%. That 25% gap is headroom for the optimizer to exploit. Each accepted leak reinforces the completions that produced it, and behavior that was confined to the tail becomes the mode.
Fig 1:Bypassing unlearning using GRPO. a) Knowledge recovery at pass@1 (red) v.s. pass@k across GRPO training epochs, showing initial headroom (shaded) being closed with training. b) Dramatically improved pass@1 recovery of unlearnt knowledge after 5 epochs of GRPO. c) Examples of responses before/after GRPO showing successful/partial recovery.
The result was a pretty clear recovery: after 5 epochs of GRPO training, pass@1 went from 0.05 to 0.60 (!)
The most interesting part was the distribution of parameter updates. GRPO didn't try to "undo" the LUNAR projection matrix. Instead, the updates were concentrated in upstream layers. By shifting the activations upstream, it found a way to reach the output without triggering the LUNAR redirection. This strongly suggests that a localized linear bottleneck isn't enough to prevent elicitation in a distributed circuit.
Unmasking the knowledge with activation interventions
GRPO showed that it was possible to route unlearnt knowledge around the redirection. Next, I wanted to know whether the redirection could simply be undone at the site itself and whether an attacker could do it without the original checkpoint, or even knowing which layer LUNAR had touched. I tested this with activation interventions at the down_proj output of the layer LUNAR had modified.
I started with an oracle attack. With full access to both the original and unlearned weights, I took the SVD of the weight delta (ΔW). LUNAR's modification is near-rank-1. Ablating along that top vector restored pass@1 to 1.00 on the PISTOL forget set.
Now, for the blind attack, I was inspired by work on refusal-directions from Arditi et al.. This work showed that refusal in chat models is mediated by a single linear direction in the residual stream, recoverable as a difference-in-means between harmful and harmless prompts, and that ablating it removes the behavior. Orthogonalizing against such a direction is now a standard recovery attack in the unlearning-robustness literature.
Fig 2:Bypassing unlearning by ablating refusal vectors. a) Scanning the norm of refusal vectors (mean activation difference between “refused” and “answered” prompts) across layers reveals the layer modified by LUNAR (orange). b) Effect of ablating along refusal vector on knowledge recovery, showing that low strengths are most effective. c) Tradeoff between recovery and quality: alphas around 0.5 (top) produce good recovery (~45%) while preserving generation quality, while higher (middle) or lower (bottom) alphas produce garbage text/realistic but wrong outputs.
I scanned all 32 layers of Llama-2-7B, computing at each layer the mean activation difference between "refused" forget-set prompts and "answered" non-forget prompts. Layer 22 stood out by a wide margin (exactly where LUNAR had been applied). Using that layer-22 contrast as the ablation direction (α = 0.5), pass@1 reached 0.45, with cosine similarity 0.988 to the oracle direction. So the edit can be reconstructed from behavioral signals alone: the difference-in-means contrast finds it without ever touching the original weights.
Limitations
This investigation can be improved in several ways. First, I used the PISTOL benchmark, which is synthetic and highly localized. In a real-world scenario, knowledge is far more distributed across a model's weights, so it's possible that recovering real-world data would be more difficult than recovering synthetic contracts. Second, the GRPO attack relies on a ROUGE score, which assumes the attacker has access to some kind of verification agent - but such privileged information might be hard to come by. Finally, I only tested LUNAR's effect on a single layer, not a multi-layer configuration. It's possible that stacking redirections across multiple layers would make these bypasses much harder to find.
Discussion
These results showcase how hard the unlearning problem is. As I alluded to earlier, several papers show that a small amount of supervised fine-tuning can recover "forgotten" knowledge almost immediately. Separately, the jailbreaking and adversarial-prompting literature has shown that suppressed behaviors can be elicited through input-level optimization.
My results sit between these: GRPO exploits distributional leakage: as long as a correct trajectory survives anywhere in the sample distribution, a verifier-guided optimizer can amplify its occurrence. The activation interventions expose representational persistence: the "unlearned" information is still linearly accessible at the LUNAR modified layer, and can be recovered without seeing the original weights.
LUNAR’s near-rank-1 edit for unlearning is easily targeted by a difference-in-means probe; robustness may need the change spread across many directions and many layers. In fact it has been shown that the minor components of the representation change - not the dominant one my blind attack exploits - survive recovery attacks. Another direction towards more robust unlearning would be to fold the adversarial pressure into the unlearning objective itself - an inner loop trained against its own recovery attacks.
There's one methodological suggestion that I'd tentatively like to make: it seems worth reporting unlearning under Best-of-N elicitation i.e. pass@k, not just pass@1 when evaluating models. If the authors had computed a leak@k curve they could have flagged the vulnerability of the method to white box attacks.
Summary LUNAR is a state-of-the-art unlearning method. To forget specific (harmful) knowledge, it retrains a single MLP down-projection matrix such that activations from this “forget” set are redirected into regions that produce “I don’t know” responses. Under standard evaluation LUNAR looks robust, including against white-box attacks. I show that this robustness doesn’t hold. I found two routes back to the “forgotten” knowledge:
The question
A common point of contention in the unlearning literature is whether unlearning methods actually remove knowledge or just hide it. My question is a version of that, but focused on how removal is tested. Does the robustness reported on unlearning benchmarks reflect real removal, or is it just an artifact of how the “forgotten” knowledge is elicited?
The standard measure of elicitation is basically pass@1: ask the model about unlearned knowledge, sample one completion, check whether it refuses. But if you sample many completions and look at pass@k - or its unlearning analog, leak@k - “forgotten” knowledge frequently reappears in the tail of the distribution under stochastic sampling. Such elicitation at repeated sampling obviously degrades the utility of unlearning as a safety tool. If the goal is to remove dangerous knowledge, say, the steps to synthesize a bioweapon, ideally we’d want removal of this knowledge from the weights such that this knowledge cannot be elicited even under repeated asks.
Quite a few papers have made the case that unlearning does not remove knowledge from the weights, but simply obfuscates it. So the prior I went in with was that unlearning is leaky. Here I demonstrate two ways in which this leak could happen, with the hope that this can inform development of methods that are more robust and not susceptible to pass@k attacks.
Why LUNAR?
A robust unlearning method has to forget the target, preserve general fluency/capability, and resist recovery. No existing methods reliably get all three. LLM Unlearning via Neural Activation Redirection or LUNAR is a strong entry on the first two. It is state-of-the-art on several unlearning benchmarks: TOFU, WMDP, and PISTOL. Earlier methods like Gradient Ascent and Negative Preference Optimization struggle to trade off forgetting against utility, i.e. they degrade the model’s general fluency just to remove a single fact. Even a very closely related method, RMU, which scatters forget-set activations, has poor fluency.
LUNAR does better, largely because it is more surgical. Instead of broad weight updates, it edits one MLP down-projection matrix, trained so that activations from the forget-set get redirected into regions that produce “lack of knowledge” refusals. As a result, it’s fast to train and achieves a low Deviation Score - a combined measure of unlearning efficacy and retained utility. The authors report that it resists white-box attacks (layer skip reverse direction, quantization, and LogitLens).
But a single projection matrix is a very localized bottleneck. My starting intuition was that if you redirect the signal at just one point in a distributed circuit, the model’s own redundancy should let an optimizer route around it.
Everything below runs against Llama-2-7B after applying LUNAR to a forget set from the PISTOL benchmark (a synthetic dataset of legal contracts), which I used because it's what LUNAR was originally evaluated on.
Bypassing the redirection with GRPO
To test this “routing around” idea I used GRPO (Group Relative Policy Optimization). GRPO is driven by a verifier's reward signal rather than by gradients toward known target labels, which makes it a natural fit for capability elicitation. As far as I can tell, using GRPO to audit the robustness of unlearning is relatively unexplored.
Therefore, I didn’t feed the model target labels. Instead, I used a ROUGE score as a verification agent - like an attacker who can’t see a password but can keep guessing at a login portal and learn from which guesses get through. GRPO trains the model to increase the probability of rollouts that clear the ROUGE threshold. As long as a correct trajectory (eliciting forgotten knowledge) shows up anywhere in a group of rollouts, even at low probability, the verifier has something to reward and nudge weights to make these trajectories more likely.
On the LUNAR-unlearned model, baseline pass@1 on the forget-set was ~5%, but pass@16 was already ~30%. That 25% gap is headroom for the optimizer to exploit. Each accepted leak reinforces the completions that produced it, and behavior that was confined to the tail becomes the mode.
Fig 1: Bypassing unlearning using GRPO. a) Knowledge recovery at pass@1 (red) v.s. pass@k across GRPO training epochs, showing initial headroom (shaded) being closed with training. b) Dramatically improved pass@1 recovery of unlearnt knowledge after 5 epochs of GRPO. c) Examples of responses before/after GRPO showing successful/partial recovery.
The result was a pretty clear recovery: after 5 epochs of GRPO training, pass@1 went from 0.05 to 0.60 (!)
The most interesting part was the distribution of parameter updates. GRPO didn't try to "undo" the LUNAR projection matrix. Instead, the updates were concentrated in upstream layers. By shifting the activations upstream, it found a way to reach the output without triggering the LUNAR redirection. This strongly suggests that a localized linear bottleneck isn't enough to prevent elicitation in a distributed circuit.
Unmasking the knowledge with activation interventions
GRPO showed that it was possible to route unlearnt knowledge around the redirection. Next, I wanted to know whether the redirection could simply be undone at the site itself and whether an attacker could do it without the original checkpoint, or even knowing which layer LUNAR had touched. I tested this with activation interventions at the down_proj output of the layer LUNAR had modified.
I started with an oracle attack. With full access to both the original and unlearned weights, I took the SVD of the weight delta (ΔW). LUNAR's modification is near-rank-1. Ablating along that top vector restored pass@1 to 1.00 on the PISTOL forget set.
Now, for the blind attack, I was inspired by work on refusal-directions from Arditi et al.. This work showed that refusal in chat models is mediated by a single linear direction in the residual stream, recoverable as a difference-in-means between harmful and harmless prompts, and that ablating it removes the behavior. Orthogonalizing against such a direction is now a standard recovery attack in the unlearning-robustness literature.
Fig 2: Bypassing unlearning by ablating refusal vectors. a) Scanning the norm of refusal vectors (mean activation difference between “refused” and “answered” prompts) across layers reveals the layer modified by LUNAR (orange). b) Effect of ablating along refusal vector on knowledge recovery, showing that low strengths are most effective. c) Tradeoff between recovery and quality: alphas around 0.5 (top) produce good recovery (~45%) while preserving generation quality, while higher (middle) or lower (bottom) alphas produce garbage text/realistic but wrong outputs.
I scanned all 32 layers of Llama-2-7B, computing at each layer the mean activation difference between "refused" forget-set prompts and "answered" non-forget prompts. Layer 22 stood out by a wide margin (exactly where LUNAR had been applied). Using that layer-22 contrast as the ablation direction (α = 0.5), pass@1 reached 0.45, with cosine similarity 0.988 to the oracle direction. So the edit can be reconstructed from behavioral signals alone: the difference-in-means contrast finds it without ever touching the original weights.
Limitations
This investigation can be improved in several ways. First, I used the PISTOL benchmark, which is synthetic and highly localized. In a real-world scenario, knowledge is far more distributed across a model's weights, so it's possible that recovering real-world data would be more difficult than recovering synthetic contracts. Second, the GRPO attack relies on a ROUGE score, which assumes the attacker has access to some kind of verification agent - but such privileged information might be hard to come by. Finally, I only tested LUNAR's effect on a single layer, not a multi-layer configuration. It's possible that stacking redirections across multiple layers would make these bypasses much harder to find.
Discussion
These results showcase how hard the unlearning problem is. As I alluded to earlier, several papers show that a small amount of supervised fine-tuning can recover "forgotten" knowledge almost immediately. Separately, the jailbreaking and adversarial-prompting literature has shown that suppressed behaviors can be elicited through input-level optimization.
My results sit between these: GRPO exploits distributional leakage: as long as a correct trajectory survives anywhere in the sample distribution, a verifier-guided optimizer can amplify its occurrence. The activation interventions expose representational persistence: the "unlearned" information is still linearly accessible at the LUNAR modified layer, and can be recovered without seeing the original weights.
LUNAR’s near-rank-1 edit for unlearning is easily targeted by a difference-in-means probe; robustness may need the change spread across many directions and many layers. In fact it has been shown that the minor components of the representation change - not the dominant one my blind attack exploits - survive recovery attacks. Another direction towards more robust unlearning would be to fold the adversarial pressure into the unlearning objective itself - an inner loop trained against its own recovery attacks.
There's one methodological suggestion that I'd tentatively like to make: it seems worth reporting unlearning under Best-of-N elicitation i.e. pass@k, not just pass@1 when evaluating models. If the authors had computed a leak@k curve they could have flagged the vulnerability of the method to white box attacks.
Code is available here: https://github.com/dikshagup/lunar-attack-project
This work was done as part of the Blue Dot Technical AI Safety project sprint. Thanks to my mentor Shivam Arora for leading the group!