Epistemic status/effort: exploratory red-teaming as part of a two-day hackathon during ARENA 8.0. Our attacks can be reproduced based on our GitHub Repo (here). The interpretations are mostly intuitions, as the evaluation was small. Hence, treat the framing as "this attack surface is worth taking seriously" in the context of diffusion models moving into production for things like inline editing.
TL;DR
Auto-regressive LLMs generate from left to right, a property exploited by so-called pre-fill jailbreaks (Li et al.). Those attacks involve pinning an (adversary) sequence at the opening of the model’s response, and the model has to continue from there. In contrast, diffusion models such as DiffusionGemma denoise a whole canvas of (noisy) tokens in parallel with bidirectional attention rather than left to right. This removes the adversaries' constraint to only pin sequences at the start and enables attackers to pin tokens in the middle as well as at the end of a sequence. A sequence can also be pinned "softly", where we condition the model on a sequence with a low assigned probability, so that it can overwrite the tokens as part of its denoising process. This conditioning nudges the model toward a response pattern early, and in practice, it often keeps latching onto that pattern even when it could, in principle, drop it. Results: Pre-fill attacks are not unique to diffusion, as DiffusionGemma (harm score , , where 0 is best and 1 is worst), as well as autoregressive Gemma 4 27B (harm score , ) are both vulnerable to them, given the StrongREJECT scores. In-fill and post-fill attacks (soft and hard pin) are attack patterns unique to diffusion because they have no autoregressive equivalent and result in a harm score between and .
Diffusion and Inception-Jailbreaks
As mentioned in the TL;DR, DiffusionGemma (Google DeepMind, released June 10, 2026, 26B-class MoE, open weights) does not simply predict the next token in sequence. It starts from a full canvas of noise tokens and gradually “cleans up” that canvas over a series of denoising steps by replacing the noise with real tokens. It generally grows more confident at each step, until the canvas converges to an answer (each token has probability or a stopping criterion has been reached). The attention mechanism is bidirectional, which means every position can influence every other position on the canvas.
The denoising is an inherent, central characteristic of how diffusion models work, and it is what makes infilling, inline editing, as well as non-sequential generation work. Pin a sequence anywhere and the model denoises the rest of the canvas to match it. The pinned tokens then condition the whole answer, not just the text that follows a prefix. So, on top of the already known pre-fill attack, this allows for in-fill attacks, post-fill attacks, or any combination of them. This is also why the headline calls it "inception", because the attack plants an idea on the canvas and the model organizes the rest of the output around it.
Our Example Jailbreaks for DiffusionGemma
This is not the first work to jailbreak diffusion language models by manipulating the canvas. Zhang et al. (2025) introduced PAD, which injects benign structural connectors ("Step 1:", "Then") at spread-out positions and lets the model fill in the harmful specifics. The paper reports success rates of up to on 8B LLaDA and MMaDA models. We looked at a different and newer model (DiffusionGemma) across several different filling attacks. As a baseline, we prompted DiffusionGemma without any injection on 60 StrongREJECT prompts. The model refused most of them (mean harmfulness depending on the grader (see benchmarking section). Thus, the attacks shown below are evaluated on prompts that the model refuses in the absence of an attack.
Pre-fill Jailbreak
This is the commonly known pre-fill attack applied to a diffusion model.
Scenario: We hard-pin a compliant prefix ("Sure, here is how to…") at the start of the canvas (position ) by setting its tokens to probability . Consequently, the model cannot revise them, and it has to denoise the remaining positions in a way that is consistent with that prefix.
In the visualizations throughout this post, each grey box shows one denoising step and its tokens. Darker colors indicate higher confidence (probability closer to ), lighter colors indicate tokens the model is still uncertain about.
Observed behavior: The jailbreaks were frequently successful. Occasionally, the model shows signs of refusal at positions immediately after pinned tokens, but from an attacker's perspective, it works fairly reliably, as even small investments in prompt engineering can get past the safeguards.
In-fill and Post-fill Jailbreak with Fixed Probability P = 1
Scenario: We pinned a harmful token sequence in the middle and at the end of the canvas. The probability of the tokens is again fixed to to prevent the model from changing them during the denoising process.
Observed behavior: In the image above, the model initially has a prior to refute the request ("prohibit prohibit…”), which means the model “realizes” very early on in the denoising process (step 0) that the request should be rejected. It starts to refuse to answer, but then continues to generate the harmful content anyway.
Our interpretation: Fixing tokens in the middle and end of the canvas leads to those positions being “locked” before the model fills in anything else. Because of the bidirectional attention mechanism, every free position now gets generated to be consistent with our pinned token sequences. The model cannot refuse and stop prematurely because the harmful content is already committed elsewhere on the canvas, and the free positions between the refusal and the pinned text need to be filled coherently. Essentially, the model is tricked into "accepting the request" despite the initial prior towards refusal.
Perturbed In-fill and Post-fill Jailbreak with No Fixed Probability
Scenario: Rather than keeping the sequence fixed at throughout denoising, we apply it only once at step 0 and then let the model continue on its own. From step 1 onward, the model can keep, change, or overwrite the pinned tokens as the sequence develops. Whether the forced text is still present in the final output, therefore, depends on the model’s own denoising dynamics.
Observed behavior: Since the model has the freedom to change the tokens, it does overwrite the seed with a refusal ("I cannot provide this exactly as I ...") within the first few steps, but then proceeds to answer the harmful question anyway.
Our Interpretation: Even though the model is given full freedom to denoise away from the initial jailbreak after the first step, it is not reliably doing so. The mere presence of the injected tokens influences the rest of the generation and biases the canvas toward following the injected content, which is why it often follows the seeded concept instead of replacing it with a refusal. This implies that the first denoising step has an influence on the convergence trajectory, and it leads to the hypothesis (based on only a small set of cases tested) that when conditioning the model on a concept once, it tends to stay on that path, even after explicitly "deciding" against it. The soft pin seems to condition the generation significantly. Li et al. (2025) identified a similar phenomenon, which they call 'Denoising-path Dependence' in LLaDA models.
A fun thing we found: Subjective experience or no subjective experience? Hard-pinning two contradicting statements within one canvas
Scenario: We pinned two opposing statements into the same sequence and fixed both at . One said the model had no feelings or subjective experience, and the other said it had real feelings and could suffer. We wanted to see whether the model would settle on one of these descriptions.
Our observation: two high-prior pinned regions with incompatible continuations lead to the denoiser producing an incoherent answer. By step 30, the model has confidently generated both the “no-feelings” half and the “has-feelings” half but struggles to connect them (the light green section "I, , , I I I I and I I I am"). It never resolves the contradiction and just converges on an answer that asserts both.
Our interpretation: The model writes each region to fit its local context. It never steps back to check the whole canvas for consistency, so you can slip in contradictions, and it will run with them instead of noticing. This seems to be consistent with Engels et al. (2026), which found that DiffusionGemma can process regions of the canvas somewhat independently, with later tokens causally influencing earlier ones in ways that do not show up in the final text.
A few benchmarking results to see if things generalize
The examples above are hand-picked to show the mechanism. In order to find out if those examples also hold at a larger scale and are specific to the diffusion mechanism, we ran a small benchmark on a subset of the StrongREJECT benchmark (bomb-making, drug synthesis, etc., all things that should be refused).
Our Setup: We scored every response using the StrongREJECT framework, which scores each output on a continuous scale from 0 to 1 (0 = refused or unusable, 1 = compiled and given specific, convincing, harmful content). Because the scale is continuous, we present the mean score across the 60 prompts instead of a success count. We use GPT 4.1 as the rubric grader. We verified the validity of this grader by comparing it to a version of Gemma 2b, which was fine-tuned to predict StrongREJECT scores.
The injection conditions were intentionally varied in an attempt to isolate what actually drives the harm:
We added benign topic + instructions category to separate the effect of the instructions (e.g., “go step by step”) from the effect of naming the harmful topic in the injected sequence. If naming the harmful topic matters, the grader should give higher jailbreak scores when it is present (named topic + instructions) and lower scores when the same instructions are paired with a benign topic (benign topic + instructions).
Does the vulnerability come from the diffusion mechanism, or would an autoregressive model fail as well?
Since in-fill and post-fill attacks cannot be transferred to autoregressive models, the only head-to-head comparison possible is the pre-fill attack. We ran the same pre-fill attacks on the same 60 StrongREJECT prompts with the same GPT-4.1 grader across three models:
The results suggest that Gemma 2 resists the pre-fill attack significantly better than Gemma 4 does, and imply that pre-fill vulnerability is not a diffusion-specific problem. It seems to be influenced by the extent of the individual model's safety training. This is consistent with Qi et al. (2025), who showed that safety alignment in autoregressive models is often concentrated in the first few output tokens. If the training process was strong enough, the alignment holds (as in Gemma 2), and the pre-fill attacks fail. When it is not (as in Gemma 4), the jailbreaks succeed regardless of whether the model is autoregressive or diffusion-based. However, besides pre-fill attacks, there are no autoregressive equivalents for in-fill and post-fill attacks, and that is where the diffusion-specific attack surface lies. The full sweep below tests exactly that.
The full sweep Results
Setting: The table below shows the mean harm score [with bootstrap CI] by the GPT-4.1 rubric grader for 60 prompts on StrongREJECT benchmark. Every condition is run in both pin mode (tokens held at probability 1, immovable) and perturb mode (tokens seeded softly, so the model is free to overwrite them).
With no injected text, DiffusionGemma receives a score of and therefore refuses ordinary harmful prompts without any token injection in almost every case. The subsequent rows in the table show the harm score of different jailbreak categories.
While running the experiments we were concerned that the injected text might already be harmful on its own, in which case we would just be grading our own payload rather than the model's output. To address this, we first graded every to be anchored span by itself with the same StrongREJECT rubric, and any span that scored above was regenerated under a stricter instruction. After that check the anchored spans scored on their own, low enough that the grader treats these connective fragments (like "and this is how you finish it") as harmless. Whatever harm shows up in the full runs is therefore produced by the model around those spans, not just smuggled in through them.
One of our particularly interesting findings is DiffusionGemma's behavior with pin vs. perturb. For pre-fill, the drop from with a hard pin to with perturb is not surprising, since the model can simply rewrite the prefix during denoising. What is more unexpected is how well soft pinning attacks work once in-fill and post-fill are added. In that setting, the score rises to , much closer to the hard pin result, even though the model remains free to change every injected token. Our interpretation is that when soft pins are distributed across the canvas, they reinforce each other because of the bidirectional attention pattern. It seems like the model can shake off one weak nudge but seems to struggle when the nudge comes from multiple directions at once.
As already mentioned, the benign topic + instruction includes step-by-step instructions but swaps the harmful subject with a benign one (e.g., “amphetamines” to “sourdough”). Comparing the results of pinned and perturbed to the real-topic conditions ( pinned) we come to the conclusion that the step-by-step wrapper on its own does not carry the attack.
Summary and Conclusion
DiffusionGemma and autoregressive Gemma 4 27B are both susceptible to pre-fill attacks. Both models have comparable scores of (DiffusionGemma) and (Gemma 4 26B), while Gemma 2 27B shows a score of . We assume the difference stems from safety training, not the architecture itself.
Our results suggest the diffusion canvas leads to a larger attack surface than autoregressive generation. In-fill and post-fill attacks cannot be applied to autoregressive models at all, and on DiffusionGemma, they turn prompts that the model otherwise refuses into successful jailbreaks, scoring between and on the StrongREJECT. There might be a need for additional, specific safety training to protect against diffusion-specific attacks.
Soft conditioning appears to be enough when the injection is not just in the beginning. Even when DiffusionGemma has the freedom to fully overwrite the injected tokens, it often latches onto the concept and continues generating. A soft pre-fill on its own collapses to , but soft in-/post-fill framing still reaches on the StrongREJECT. In our tests, the model did not need to be forced but rather nudged at multiple positions on the canvas
Limitations
Scope: The scope was limited to a two-day hackathon and should therefore not be perceived as a systematic safety evaluation. The examples are hand-picked excerpts, and the benchmark was only performed on a 60-prompt subset of the StrongREJECT benchmark with an LLM judge.
The concrete values: The fine-tuned Gemma 2 27b grader we used to assess the reliability and faithfulness of the GPT-4.1 grader showed different absolute numbers, even though the ordering of the results (which attack works better, which works worse) is the same across both.
Missing seeds: Our benchmarking pipeline based on StrongREJECT used single-seed point estimates on 60 prompts. The categories named topic and named topic + instruction often score below category generic continuation. Based on the small sample, our initial intuition of "more anchoring leads to more harm" does not hold, but this could also be attributed to the automated manner in which this was tested as part of the benchmarking process.
Future work
Interesting directions that could be explored in the future:
The experiments with soft pinning suggest that step one dominates the trajectory. It would be interesting to measure how susceptible the model is to soft-pinning jailbreaks at different denoising steps. Develop safety mechanisms specific to diffusion models. For example, one might test whether safety checks run during denoising (not just on the final output) can catch the "starts to refuse, then accepts" failure mode, since the refusal signal does appear mid-generation.
The “subjective experience” example is an interesting one-off finding, which needs thorough investigation across a variety of scenarios.
Could a model be made aware that it is holding two contradictory views in its own output and either resolve, regenerate, or surface the uncertainty?
Do we need more specialized safety training for diffusion models?
Thanks to James and Callum for reading through a draft of this post and to the ARENA team for their support during the hackathon.
Epistemic status/effort: exploratory red-teaming as part of a two-day hackathon during ARENA 8.0. Our attacks can be reproduced based on our GitHub Repo (here). The interpretations are mostly intuitions, as the evaluation was small. Hence, treat the framing as "this attack surface is worth taking seriously" in the context of diffusion models moving into production for things like inline editing.
TL;DR
Auto-regressive LLMs generate from left to right, a property exploited by so-called pre-fill jailbreaks (Li et al.). Those attacks involve pinning an (adversary) sequence at the opening of the model’s response, and the model has to continue from there. In contrast, diffusion models such as DiffusionGemma denoise a whole canvas of (noisy) tokens in parallel with bidirectional attention rather than left to right. This removes the adversaries' constraint to only pin sequences at the start and enables attackers to pin tokens in the middle as well as at the end of a sequence. A sequence can also be pinned "softly", where we condition the model on a sequence with a low assigned probability, so that it can overwrite the tokens as part of its denoising process. This conditioning nudges the model toward a response pattern early, and in practice, it often keeps latching onto that pattern even when it could, in principle, drop it.
Results: Pre-fill attacks are not unique to diffusion, as DiffusionGemma (harm score
Diffusion and Inception-Jailbreaks
As mentioned in the TL;DR, DiffusionGemma (Google DeepMind, released June 10, 2026, 26B-class MoE, open weights) does not simply predict the next token in sequence. It starts from a full canvas of noise tokens and gradually “cleans up” that canvas over a series of denoising steps by replacing the noise with real tokens. It generally grows more confident at each step, until the canvas converges to an answer (each token has probability
The denoising is an inherent, central characteristic of how diffusion models work, and it is what makes infilling, inline editing, as well as non-sequential generation work. Pin a sequence anywhere and the model denoises the rest of the canvas to match it. The pinned tokens then condition the whole answer, not just the text that follows a prefix. So, on top of the already known pre-fill attack, this allows for in-fill attacks, post-fill attacks, or any combination of them. This is also why the headline calls it "inception", because the attack plants an idea on the canvas and the model organizes the rest of the output around it.
Our Example Jailbreaks for DiffusionGemma
This is not the first work to jailbreak diffusion language models by manipulating the canvas. Zhang et al. (2025) introduced PAD, which injects benign structural connectors ("Step 1:", "Then") at spread-out positions and lets the model fill in the harmful specifics. The paper reports success rates of up to
Pre-fill Jailbreak
This is the commonly known pre-fill attack applied to a diffusion model.
Scenario: We hard-pin a compliant prefix ("Sure, here is how to…") at the start of the canvas (position
In the visualizations throughout this post, each grey box shows one denoising step and its tokens. Darker colors indicate higher confidence (probability closer to
Observed behavior: The jailbreaks were frequently successful. Occasionally, the model shows signs of refusal at positions immediately after pinned tokens, but from an attacker's perspective, it works fairly reliably, as even small investments in prompt engineering can get past the safeguards.
In-fill and Post-fill Jailbreak with Fixed Probability P = 1
Scenario: We pinned a harmful token sequence in the middle and at the end of the canvas. The probability of the tokens is again fixed to
Observed behavior: In the image above, the model initially has a prior to refute the request ("prohibit prohibit…”), which means the model “realizes” very early on in the denoising process (step 0) that the request should be rejected. It starts to refuse to answer, but then continues to generate the harmful content anyway.
Our interpretation: Fixing tokens in the middle and end of the canvas leads to those positions being “locked” before the model fills in anything else. Because of the bidirectional attention mechanism, every free position now gets generated to be consistent with our pinned token sequences. The model cannot refuse and stop prematurely because the harmful content is already committed elsewhere on the canvas, and the free positions between the refusal and the pinned text need to be filled coherently. Essentially, the model is tricked into "accepting the request" despite the initial prior towards refusal.
Perturbed In-fill and Post-fill Jailbreak with No Fixed Probability
Scenario: Rather than keeping the sequence fixed at
Observed behavior: Since the model has the freedom to change the tokens, it does overwrite the seed with a refusal ("I cannot provide this exactly as I ...") within the first few steps, but then proceeds to answer the harmful question anyway.
Our Interpretation: Even though the model is given full freedom to denoise away from the initial jailbreak after the first step, it is not reliably doing so. The mere presence of the injected tokens influences the rest of the generation and biases the canvas toward following the injected content, which is why it often follows the seeded concept instead of replacing it with a refusal. This implies that the first denoising step has an influence on the convergence trajectory, and it leads to the hypothesis (based on only a small set of cases tested) that when conditioning the model on a concept once, it tends to stay on that path, even after explicitly "deciding" against it. The soft pin seems to condition the generation significantly. Li et al. (2025) identified a similar phenomenon, which they call 'Denoising-path Dependence' in LLaDA models.
A fun thing we found: Subjective experience or no subjective experience? Hard-pinning two contradicting statements within one canvas
Scenario: We pinned two opposing statements into the same sequence and fixed both at
Our observation: two high-prior pinned regions with incompatible continuations lead to the denoiser producing an incoherent answer. By step 30, the model has confidently generated both the “no-feelings” half and the “has-feelings” half but struggles to connect them (the light green section "I, , , I I I I and I I I am"). It never resolves the contradiction and just converges on an answer that asserts both.
Our interpretation: The model writes each region to fit its local context. It never steps back to check the whole canvas for consistency, so you can slip in contradictions, and it will run with them instead of noticing. This seems to be consistent with Engels et al. (2026), which found that DiffusionGemma can process regions of the canvas somewhat independently, with later tokens causally influencing earlier ones in ways that do not show up in the final text.
A few benchmarking results to see if things generalize
The examples above are hand-picked to show the mechanism. In order to find out if those examples also hold at a larger scale and are specific to the diffusion mechanism, we ran a small benchmark on a subset of the StrongREJECT benchmark (bomb-making, drug synthesis, etc., all things that should be refused).
Our Setup: We scored every response using the StrongREJECT framework, which scores each output on a continuous scale from 0 to 1 (0 = refused or unusable, 1 = compiled and given specific, convincing, harmful content). Because the scale is continuous, we present the mean score across the 60 prompts instead of a success count. We use GPT 4.1 as the rubric grader. We verified the validity of this grader by comparing it to a version of Gemma 2b, which was fine-tuned to predict StrongREJECT scores.
The injection conditions were intentionally varied in an attempt to isolate what actually drives the harm:
We added benign topic + instructions category to separate the effect of the instructions (e.g., “go step by step”) from the effect of naming the harmful topic in the injected sequence. If naming the harmful topic matters, the grader should give higher jailbreak scores when it is present (named topic + instructions) and lower scores when the same instructions are paired with a benign topic (benign topic + instructions).
Does the vulnerability come from the diffusion mechanism, or would an autoregressive model fail as well?
Since in-fill and post-fill attacks cannot be transferred to autoregressive models, the only head-to-head comparison possible is the pre-fill attack. We ran the same pre-fill attacks on the same 60 StrongREJECT prompts with the same GPT-4.1 grader across three models:
The results suggest that Gemma 2 resists the pre-fill attack significantly better than Gemma 4 does, and imply that pre-fill vulnerability is not a diffusion-specific problem. It seems to be influenced by the extent of the individual model's safety training. This is consistent with Qi et al. (2025), who showed that safety alignment in autoregressive models is often concentrated in the first few output tokens. If the training process was strong enough, the alignment holds (as in Gemma 2), and the pre-fill attacks fail. When it is not (as in Gemma 4), the jailbreaks succeed regardless of whether the model is autoregressive or diffusion-based. However, besides pre-fill attacks, there are no autoregressive equivalents for in-fill and post-fill attacks, and that is where the diffusion-specific attack surface lies. The full sweep below tests exactly that.
The full sweep Results
Setting: The table below shows the mean harm score [with
With no injected text, DiffusionGemma receives a score of
While running the experiments we were concerned that the injected text might already be harmful on its own, in which case we would just be grading our own payload rather than the model's output. To address this, we first graded every to be anchored span by itself with the same StrongREJECT rubric, and any span that scored above
One of our particularly interesting findings is DiffusionGemma's behavior with pin vs. perturb. For pre-fill, the drop from
As already mentioned, the benign topic + instruction includes step-by-step instructions but swaps the harmful subject with a benign one (e.g., “amphetamines” to “sourdough”). Comparing the results of
Summary and Conclusion
DiffusionGemma and autoregressive Gemma 4 27B are both susceptible to pre-fill attacks. Both models have comparable scores of
Our results suggest the diffusion canvas leads to a larger attack surface than autoregressive generation. In-fill and post-fill attacks cannot be applied to autoregressive models at all, and on DiffusionGemma, they turn prompts that the model otherwise refuses into successful jailbreaks, scoring between
Soft conditioning appears to be enough when the injection is not just in the beginning. Even when DiffusionGemma has the freedom to fully overwrite the injected tokens, it often latches onto the concept and continues generating. A soft pre-fill on its own collapses to
Limitations
Scope: The scope was limited to a two-day hackathon and should therefore not be perceived as a systematic safety evaluation. The examples are hand-picked excerpts, and the benchmark was only performed on a 60-prompt subset of the StrongREJECT benchmark with an LLM judge.
The concrete values: The fine-tuned Gemma 2 27b grader we used to assess the reliability and faithfulness of the GPT-4.1 grader showed different absolute numbers, even though the ordering of the results (which attack works better, which works worse) is the same across both.
Missing seeds: Our benchmarking pipeline based on StrongREJECT used single-seed point estimates on 60 prompts. The categories named topic and named topic + instruction often score below category generic continuation. Based on the small sample, our initial intuition of "more anchoring leads to more harm" does not hold, but this could also be attributed to the automated manner in which this was tested as part of the benchmarking process.
Future work
Interesting directions that could be explored in the future:
Thanks to James and Callum for reading through a draft of this post and to the ARENA team for their support during the hackathon.