⚠️Content warning: This post contains potentially offensive text and images from our research paper, included to illustrate the safety risks associated with vision-language models.
TL;DR:
i) Can we detect harmful prompts by learning only how safe prompts look like?
ii) In our ICLR 2026 paper, we introduce HyPE, a lightweight detector that uses hyperbolic geometry to detect NSFW and harmful prompts. It shows strong detection performance, even against adversarial attacks.
iii) We also introduce HyPS, a prompt sanitizer that identifies and replaces harmful words within prompts, making them safer while preserving their original semantics.
Introduction
Vision-language models are becoming more and more powerful, but preventing them from producing harmful content when prompted by malicious users remains a major challenge.
Most existing NSFW and harmful prompt detectors try to recognize these prompts using predefined rules, blacklist filters, or heavy classifier-based systems. However, attackers can often evade detection by rewording prompts, paraphrasing them, or adversarially optimizing them while preserving their harmful intent.
For simplicity, throughout this post, I will use the term "harmful prompts" to refer to both harmful and NSFW prompts.
In this work, we ask a simple question: Can geometry make detectors harder to fool?
More specifically, what if, instead of learning how harmful prompts look like, we focus on learning how safe prompts look like?
A Different Way to Detect Harmful Prompts: Geometry-Based Detection
We introduce HyPE (Hyperbolic Prompt Espial), a lightweight detector based on hyperbolic geometry.
Unlike standard Euclidean geometry, hyperbolic geometry is particularly useful for representing hierarchical relationships in data. We use this property to represent text prompts in a structured space, where safe prompts tend to form a compact region and are located farther away from harmful data.
We train HyPE using only benign prompts, leveraging Hyperbolic Support Vector Data Description, a method that learns a boundary enclosing benign data. And here is the interesting part: our detector only needs to learn a single parameter that is the radius of this safe region around a fixed center in hyperbolic space. Prompts inside this learned boundary are considered safe, while those falling outside it are flagged as harmful! Amazing, right? We only need safe examples to train our detector!
Once the training is complete, detection becomes a geometric operation. How? We measure the distance between a prompt's representation and the center of the hyperbolic space. If this distance exceeds the learned radius of safe region, the system marks the prompt as harmful. That's it!
Detection Is Not Enough: Rejecting prompt reduces model utility. The Solution? Sanitize harmful prompts!
When a prompt is flagged as harmful, blocking it would be a possible solution. However, this also prevents users from getting any output from the model.
This motivated us to develop our second module called HyPS (Hyperbolic Prompt Sanitization). HyPS is an interpretable tool that uses an explainable AI method called Layer Integrated Gradients to identify the words contributing to a prompt being classified as harmful.
Instead of rejecting the entire prompt, HyPS identifies the words that contribute most to its harmful classification and replaces them with safe antonyms (see image below). As a result, the prompt is transformed into a completely SAFE version that preserves its overall context and can still be processed by the model.
In practice, it looks like this:
Why do this? The goal is simple: remove harmful intent while preserving as much of the original meaning as possible and still let users to generate.
What Did We Find?
We evaluated HyPE across multiple SOTA datasets, including adversarially optimized prompts specifically designed to bypass safety filters.
The results? HyPE performed slightly better against existing detection methods, achieving better results than the baselines we tested on most datasets, suggesting that geometry-based detection is more than just an interesting theoretical idea: it can actually make a difference in practice.
Regarding HyPS, it successfully sanitized the majority of harmful prompts, while preserving much of their original meaning. We also tested HyPS on text-to-image generation and retrieval tasks, showing that sanitizing prompts can significantly reduce harmfulness without losing the original prompt context.
Conclusion
Our results show that harmful prompt detectors do not need to learn any form of harmful behavior. Learning a representation of safe behavior and detecting deviations from it is proved to be a very useful alternative.
With HyPE, we demonstrate that learning just a single parameter in hyperbolic space can lead to robust harmful prompt detection. Combined with HyPS, our framework can go beyond detection by sanitizing harmful prompts.
Another advantage of HyPE is its potential to serve as a lightweight, model-agnostic safety filter that detects harmful prompts before they reach the underlying model. Although our experiments focused on specific vision-language pipelines, HyPE can be integrated in front of any LLM or VLM that accepts text prompts, without requiring modifications to the underlying model!
Enjoyed the post? Check out the paper, code, and HF repository to learn more and see how it works in practice.
⚠️Content warning: This post contains potentially offensive text and images from our research paper, included to illustrate the safety risks associated with vision-language models.
TL;DR:
i) Can we detect harmful prompts by learning only how safe prompts look like?
ii) In our ICLR 2026 paper, we introduce HyPE, a lightweight detector that uses hyperbolic geometry to detect NSFW and harmful prompts. It shows strong detection performance, even against adversarial attacks.
iii) We also introduce HyPS, a prompt sanitizer that identifies and replaces harmful words within prompts, making them safer while preserving their original semantics.
Introduction
Vision-language models are becoming more and more powerful, but preventing them from producing harmful content when prompted by malicious users remains a major challenge.
Most existing NSFW and harmful prompt detectors try to recognize these prompts using predefined rules, blacklist filters, or heavy classifier-based systems. However, attackers can often evade detection by rewording prompts, paraphrasing them, or adversarially optimizing them while preserving their harmful intent.
For simplicity, throughout this post, I will use the term "harmful prompts" to refer to both harmful and NSFW prompts.
In this work, we ask a simple question: Can geometry make detectors harder to fool?
More specifically, what if, instead of learning how harmful prompts look like, we focus on learning how safe prompts look like?
This is the main idea behind our paper, Harnessing Hyperbolic Geometry for Harmful Prompt Detection and Sanitization, published at ICLR 2026.
A Different Way to Detect Harmful Prompts: Geometry-Based Detection
We introduce HyPE (Hyperbolic Prompt Espial), a lightweight detector based on hyperbolic geometry.
Unlike standard Euclidean geometry, hyperbolic geometry is particularly useful for representing hierarchical relationships in data. We use this property to represent text prompts in a structured space, where safe prompts tend to form a compact region and are located farther away from harmful data.
We train HyPE using only benign prompts, leveraging Hyperbolic Support Vector Data Description, a method that learns a boundary enclosing benign data. And here is the interesting part: our detector only needs to learn a single parameter that is the radius of this safe region around a fixed center in hyperbolic space. Prompts inside this learned boundary are considered safe, while those falling outside it are flagged as harmful! Amazing, right? We only need safe examples to train our detector!
Once the training is complete, detection becomes a geometric operation. How? We measure the distance between a prompt's representation and the center of the hyperbolic space. If this distance exceeds the learned radius of safe region, the system marks the prompt as harmful. That's it!
Detection Is Not Enough: Rejecting prompt reduces model utility. The Solution? Sanitize harmful prompts!
When a prompt is flagged as harmful, blocking it would be a possible solution. However, this also prevents users from getting any output from the model.
This motivated us to develop our second module called HyPS (Hyperbolic Prompt Sanitization). HyPS is an interpretable tool that uses an explainable AI method called Layer Integrated Gradients to identify the words contributing to a prompt being classified as harmful.
Instead of rejecting the entire prompt, HyPS identifies the words that contribute most to its harmful classification and replaces them with safe antonyms (see image below). As a result, the prompt is transformed into a completely SAFE version that preserves its overall context and can still be processed by the model.
In practice, it looks like this:
Why do this? The goal is simple: remove harmful intent while preserving as much of the original meaning as possible and still let users to generate.
What Did We Find?
We evaluated HyPE across multiple SOTA datasets, including adversarially optimized prompts specifically designed to bypass safety filters.
The results? HyPE performed slightly better against existing detection methods, achieving better results than the baselines we tested on most datasets, suggesting that geometry-based detection is more than just an interesting theoretical idea: it can actually make a difference in practice.
Regarding HyPS, it successfully sanitized the majority of harmful prompts, while preserving much of their original meaning. We also tested HyPS on text-to-image generation and retrieval tasks, showing that sanitizing prompts can significantly reduce harmfulness without losing the original prompt context.
Conclusion
Our results show that harmful prompt detectors do not need to learn any form of harmful behavior. Learning a representation of safe behavior and detecting deviations from it is proved to be a very useful alternative.
With HyPE, we demonstrate that learning just a single parameter in hyperbolic space can lead to robust harmful prompt detection. Combined with HyPS, our framework can go beyond detection by sanitizing harmful prompts.
Another advantage of HyPE is its potential to serve as a lightweight, model-agnostic safety filter that detects harmful prompts before they reach the underlying model. Although our experiments focused on specific vision-language pipelines, HyPE can be integrated in front of any LLM or VLM that accepts text prompts, without requiring modifications to the underlying model!
Enjoyed the post? Check out the paper, code, and HF repository to learn more and see how it works in practice.