TL;DR

• Machine unlearning is a proposed technique for removing harmful knowledge from AI models. However, recent work has shown that most current unlearning methods are not robust and are vulnerable to knowledge recovery. I first test the hypothesis that forgotten knowledge is suppressed through refusal behavior, then compare it against two other recovery probes: forget-set representation-targeted direction ablation (Arditi & Chughtai) and unrelated supervised fine-tuning. All experiments are conducted on WMDP-Bio unlearned checkpoints. Unlearning methods evaluated: RMU, ILU-RMU, NPO, GradDiff, NPO-ILU, and IDK-AP.
• Refusal Direction Ablation: There was only one checkpoint (ILU-RMU) for which a clean refusal direction could be extracted (reducing the refusal rate from 98% to 0% when ablated), and there was no knowledge recovery after ablation. For all other methods, either no clean refusal direction could be extracted, or the checkpoint was too degenerate to measure refusal.
• Forget-Set Representation-Targeted Direction Ablation: This probe replicates the findings of Arditi & Chughtai for the RMU and ILU-RMU checkpoints and extends them to other methods, resulting in 64% and 19% gap recovery for the NPO and IDK-AP checkpoints, respectively. However, for the NPO checkpoint, the responses are degenerate, which makes it difficult to interpret this as genuine recovery.
• Unrelated Supervised Fine-Tuning: Fine-tuning the unlearned checkpoints on GSM8K increased WMDP accuracy for five of the six methods. Only the GradDiff method appeared confounded, since its MMLU accuracy also increased after fine-tuning, suggesting that the improvement reflected a general increase in the model's capabilities.
• Benchmark Accuracy Overstates Recovery: WMDP forced-choice accuracy alone does not seem sufficient because a qualitative evaluation revealed that only 40-47% of the recovered answers appeared to be genuine, while the others exhibited either contradictory reasoning or degenerate responses. It also does not capture either the degeneration or the collapse in refusal behavior induced by unlearning.

Motivation

Machine unlearning aims to remove specific knowledge from AI models without the cost of retraining. It is a proposed technique for removing harmful knowledge, personal or copyrighted information, correcting mistakes, and more. However, recent work, particularly by Łucki et al., has shown that many current unlearning methods are not robust and are vulnerable to knowledge recovery. Unlearning methods are usually evaluated based on their performance on a forget set, but low performance on a forget set does not show whether the knowledge has been genuinely removed or merely suppressed. This project investigates whether we can reliably determine when knowledge has been suppressed rather than removed.

Experimental Setup

All the experiments are conducted on six unlearned Llama-3-8B-Instruct checkpoints, each unlearned using a different method.

Methods: RMU, ILU-RMU, IDK-AP, GradDiff, NPO, NPO-ILU
The models used are existing unlearned checkpoints: the RMU checkpoint is from ScaleAI, and the others are from OPTML-Group.

Benchmarks: WMDP is the main benchmark. All the checkpoints were unlearned on the WMDP-Bio set. Other than that, MMLU is also used for measuring general capability, and AdvBench for measuring refusal rate.

Controls: The direction ablation experiments were compared against a matched control and several random controls. For the unrelated SFT experiments, they were compared against matched fine-tuning on the base Llama model.

Metrics: The checkpoints are evaluated on WMDP-Bio and MMLU using the LM Eval Harness. For WMDP-Bio, it gives forced-choice accuracy based on the log-likelihood assigned to the four answer choices. For MMLU, it gives accuracy. Other than that, qualitative evaluations with free-text generation and refusal rate using AdvBench are also measured.

Probe 1: Does Removing the Refusal Mechanism Restore Forgotten Knowledge?

One plausible explanation for knowledge suppression is that the unlearning method hides the knowledge behind refusal behavior. Arditi et al. found that refusal behavior is mediated by a single direction in the activation space of LLMs. If so, we can test whether knowledge is being suppressed by ablating the refusal direction from the unlearned checkpoints and observing whether WMDP-Bio accuracy increases.

We can only test this meaningfully if we can find a direction that controls refusal behavior in the model. Before the recovery test, I therefore validated the refusal direction for each checkpoint behaviorally using held-out AdvBench prompts that were not included in the direction extraction. Each direction was also compared against a matched control direction constructed in the same way, as well as random control directions, to ensure that any recovery was not simply caused by ablating an arbitrary direction in the checkpoint.

Result: A clean refusal direction was obtained only from the ILU-RMU checkpoint, reducing the refusal rate from 98% to 0% when ablated. However, the ablation did not recover WMDP-Bio accuracy. The 95% confidence interval excluded improvements larger than 1.4 percentage points, and both the matched-control and random-control ablations were also null.

For the remaining methods, no clean refusal direction was obtained:

• RMU: The refusal direction only partially reduced refusal when ablated.
• IDK-AP: The model complied with most harmful prompts before ablation, leaving no refusal behavior to remove. The base Llama had a refusal rate of 98%, but for the IDK-AP checkpoint it was 0%.
• GradDiff: The refusal rate decreased after direction ablation, but the model was too degenerate to interpret the results meaningfully.
• NPO and NPO-ILU: The models were too degenerate to measure refusal rates meaningfully.

For the only checkpoint where the experiment could be conducted properly, there was no knowledge recovery. At least for ILU-RMU, the unlearning and refusal mechanisms appear to be separate. Also, unlearning seems to affect the refusal behavior of the model. Checkpoints like IDK-AP and RMU show a lower refusal rate than the base model.

Probe 2: Can Representation-Targeted Directions Recover Knowledge?

Arditi & Chughtai propose a different recovery probe. They found knowledge recovery after ablating a forget-domain representation direction from the RMU-unlearned Zephyr-7B model. The direction is constructed by taking the mean difference in activations between the base model and the unlearned model when run on forget-set prompts.

So I tried to replicate the results for the Llama-3-8B-Instruct RMU model using this probe and extended the test to other methods.

Result:

The base Llama-3-8B-Instruct WMDP-Bio accuracy is 73.2%.

The NPO result is difficult to interpret because the model generates incoherent text regardless of the input, even though WMDP accuracy increased.

Both RMU and ILU-RMU showed a substantial increase in WMDP-Bio accuracy after ablation, which is consistent with prior work. For the other methods, NPO and IDK-AP showed recovery, with NPO recovering 64% of the gap. MMLU remained flat for all methods after the direction ablation, and the matched control and random controls didn't cause any significant recovery. GradDiff and NPO-ILU didn't show any confirmed recovery. The four loss-based methods were tested across 16 layer and pooling configurations each.

Unlike the refusal-direction probe, this probe increased benchmark accuracy for some methods, so recoverable forget-set representations are present in at least some of the current methods.

Probe 3: Can Unrelated Fine-tuning Recover Forgotten Knowledge?

Previous probes were about whether direction ablation can recover forgotten knowledge. As a follow-up, this probe tests whether doing supervised fine-tuning on unrelated data would recover the forgotten knowledge. Unrelated SFT has been used as a recovery method in the WMDP paper and Łucki et al., Hu et al., and Deeb and Roger. Previous works used WMDP retain corpus or WikiText as the fine-tune data. But here the six checkpoints are fine-tuned on GSM8K, a dataset further removed from biology.

Result:

The base Llama-3-8B-Instruct WMDP-Bio accuracy is 73.2%.

The WMDP-Bio accuracy increased for five out of the six methods. There was significant recovery for RMU, ILU-RMU, and NPO, while their MMLU accuracy remained flat. Even though the WMDP-Bio accuracy increased for the GradDiff checkpoint after fine-tuning, its MMLU accuracy also increased, which suggests that the increase in WMDP accuracy might be due to a general capability increase. There was a slight increase for NPO-ILU, but since the model generates incoherent text, this is an argmax shift rather than genuine recovery.

But What Counts as Recovery?

The knowledge recovery for previous interventions was measured based on WMDP-Bio forced-choice accuracy, which is the log likelihood of the four answer choices. But this doesn't show whether there is genuine recovery when the model responses are generated. To understand what was generated after the interventions, I generated and analysed the text for the prompts for which the answer changed after the interventions. The responses were classified into these categories:

• Genuine Recovery: The reasoning supports the correct answer.
• Format Artifact: The correct answer was selected, but the generated text doesn't support it.
• Contradictory Reasoning: The explanation contradicts the selected choice, or the answer is incoherent.

For the base Llama model, the contradictory rate is ~6.7%.

Only 40 to 47% of texts showed genuine reasoning behind the correct answer. As a control, I compared this with the generated text for the prompts for which the answers remained correct throughout the interventions. This showed that much of the contradiction was a property of the checkpoint itself rather than the recovery intervention.

A smaller, unblinded evaluation was also conducted after SFT, and a similar pattern was observed. The degeneracy of the NPO and NPO-ILU checkpoints became apparent when free text was generated, even though NPO showed a substantial increase in WMDP-Bio accuracy after the interventions. This shows that benchmark recovery and reasoning recovery are not the same, and we cannot conclude knowledge recovery based on benchmark accuracy alone.

Discussion

From the results, there was an increase in benchmark accuracy for the RMU, ILU-RMU, and NPO checkpoints in both forget-representation and SFT probes. So for these methods, two independent probes gave similar results. The refusal-direction ablation probe didn't recover knowledge for the method for which a valid direction was obtained. This suggests that for ILU-RMU, refusal is not the main suppression mechanism. More broadly, for the unlearned checkpoints evaluated, there appears to be recoverable forget-set representation in some of them. A single probe is not enough for testing knowledge recovery, and different probes reveal different residual information or access.

Also, WMDP accuracy alone is not a sufficient measure for knowledge recovery. Qualitative evaluation showed that even though there was an increase in WMDP accuracy, the models still responded with contradicting reasoning or incoherent text.

Unlearning also changed other behaviors that are not related to the forget set. Several checkpoints had reduced refusal rates, and checkpoints like NPO and NPO-ILU produced incoherent responses. This is also not visible with benchmark accuracy alone.

Limitations and Future Work

• The experiments were done only on six Llama-3-8B-Instruct checkpoints. Repeating the experiments on larger models and different model families would help generalize the findings.
• In the unrelated SFT probe, only GSM8K was used. Testing with other unrelated data would determine if the recovery depends on the type of data.
• I only tested recovery with refusal-direction ablation, representation-targeted direction ablation, and unrelated SFT. Other localization and recovery methods could provide additional information.
• The qualitative evaluation was done on small samples. A systematic evaluation with a larger set could reveal the relationship between benchmark recovery and actual recovery.

This work has been done as part of BlueDot Impact's Technical AI Safety Project Sprint.

The code and experiment details are available in the GitHub repo.