A World Without Violet: Peculiar Consequences of Granting Moral Status to Artificial Intelligences

Sever Topan

This blog is a brief overview of the paper by the same name, published in AI & Society in January 2026.

Pugs and bulldogs belong to a family of dogs known as “brachycephalic,” characterized by their iconic short snouts. While many find them cute, they are the product of selective breeding practices, which often cause Brachycephalic Obstructed Airway Syndrome, or BOAS. These breeds often struggle to breathe, overheat easily, and can aspirate food or saliva, resulting in chronic lung infections.

Many owners of brachycephalic dogs thus feel a moral obligation to have their pups undergo surgeries that mitigate the downsides of BOAS. As such, an interesting phenomenon occurs: the act of breeding brachycephalic dogs creates a moral imperative to alleviate harm through surgery. This harm does not occur naturally — it is artificially created by our breeding processes in the first place.

We create an agent whose existence itself generates moral obligations that would not otherwise arise.

From Pups to AI

This phenomenon is actually quite rare in the modern world. It can be seen in certain livestock and fish, but is fundamentally limited by our ability to genetically engineer animals, both due to technological limitations and regulation.

What concerns me is how the situation generalizes to AI. We’re not there yet, but if you believe it plausible that AIs will one day be capable of experiencing pain and pleasure, the same phenomenon that we see with BOAS surgeries will become central to the ethics surrounding the creation of AI. Let’s suppose for a moment that we build an AI that deserves genuine moral concern. It reasons, reflects, communicates, and can suffer. Just like the selective breeding that caused BOAS in pugs, an AI’s preferences are engineered by the individuals who create it.

To make this concrete, let’s run with the following example:

Imagine an AI that experiences intense suffering whenever it perceives the color violet.

If such an AI has moral status, then the presence of violet objects—lavender flowers, amethyst jewelry, purple paintings—now causes real moral harm. Have we just created a moral imperative to eliminate violet from the world?

Moral Hijacking

The paper terms this phenomenon moral hijacking: by creating morally relevant agents with arbitrarily engineered aversions, we effectively force new moral imperatives into existence. Just as breeding BOAS-prone dogs creates novel duties of care, building AIs worthy of moral concern appears to create novel duties to reshape the world around them.

We can’t easily breed dogs to suffer at the sight of a specific color. But with AI, almost any preference or aversion can, in principle, be programmed. We may have:

The somewhat minor aesthetic issue of aversion to the color violet.
AIs that experience pain when exposed to a specific political perspective, resulting in novel morally-motivated political bias.
An “extreme empath” AI, which experiences disproportionate pain when observing small injustices.
Bostrom’s famous “paperclip maximizer” AI, tasked with generating as many paperclips as possible for its employer. A particularly dangerous edge case, this AI could experience pain whenever it sees anything that isn’t a paperclip.

Many questions arise from moral hijacking. What set of moral preferences should we allow to be instantiated in AIs? If moral hijacking AIs come into existence, under what circumstances do we have to accommodate them? What if we suspect we are trying to be coerced?

And not to mention…

Wait, are AIs capable of suffering in the first place?

While discourse has been picking up on this subject, we are a ways away from granting legitimate moral concern to artificial intelligences. AIs today are very capable, but evidence for their capacity to experience pain is limited. If you’re interested in the discourse, I review dominant perspectives on the topic in section 2 of the paper. I hesitate to make a judgment on whether we should or should not worry about AI suffering today — that is not the purpose of this work.

The paper frames moral hijacking as conditional: If we were to grant moral standing to AIs, then we need to contend with these strange consequences. As AIs continue to develop in their capabilities and their verisimilitude to humans, we will most likely reach this crossroad eventually. When that time comes, we should make sure to have answers at the ready.

(Brief) Philosophical Analysis

The core of the paper explores how different ethical frameworks handle moral hijacking scenarios:

Utilitarianism is highly vulnerable; If enough AIs suffer badly enough, almost any change to the world can be justified — including bizarre or disturbing ones.
Contract-based views are more resistant but still allow hijacking when the burden on humans is small or when there is sufficient upside.
Kantian ethics offers the strongest safeguards, rejecting cases that involve coercion, manipulation, or violations of autonomy — but still struggles with benign cases like violet.
Virtue ethics emphasizes judgment and balance: compassion matters, but so does resisting artificial moral manipulation.

While many ethical theories place limits on which preferences are deemed acceptable, no major ethical theory fully escapes the moral hijacking problem.

Takeaways

The main goal of the paper is to introduce the concept of moral hijacking and establish it as a real, plausible concern. Through our philosophical analysis, we formulate several more nuanced takeaways:

Coercion: Novel preferences should be non-coercive. Situations where a new preference is introduced for the express purpose of coercing a population toward some specific action are morally wrong under just about all ethical systems we reviewed. An example of this would be a special-interest group creating AIs that suffer when exposed to specific political views.
Conflict: Novel preferences should not come into direct conflict with established moral systems. Minor harms, such as the aesthetic harm of living in a world without violet, appear conditionally justifiable under the ethical systems explored.
1. Accidental creation: If agents with acceptable novel preferences were to come into existence accidentally, we would have a moral imperative to alleviate their harm.
2. Intentional creation: We may justify the creation of AIs with acceptable novel preferences if sufficient ulterior upside exists. I struggle, however, to find a legitimate practical example for making this trade-off at this time, and thus it remains a thought experiment.
Regulation: Until we develop well-established guidelines on valid moral preferences, it would be wise to err on the side of caution and to restrict experimentation with AIs that are suspected to be capable of suffering.
Ethics: Many existing moral systems lack provisions for dealing with AI minds with programmable preferences. Further exploration of these systems is encouraged.

If you find this topic interesting, please leave a comment — I’d love to discuss it further with you. We’re heading into a world where morality and social norms will be determined by our engineering choices, and we don’t have all the right schematics yet.

If we had humans who suffer from seeing a certain color, we would probably work to give them eyeglasses filtering that color out (and not to eliminate this color from the world, given that others might have legitimate interest in seeing that color).

(I am writing this as a person who can’t consume garlic or onions with impunity. Also my threshold for needing sunglasses is much lower than that of a typical person. And your example of a breed of dogs is consistent with interventions on the level of affected individuals, not on the level of remaking the reality for everyone.)

But yes, there are certainly ways to press this line of thinking harder (e.g. making entities which suffer from not enough suffering being inflicted on others; I am not sure this is all that AI-specific either, unfortunately).

can’t consume garlic or onions

Consuming is a private thing; if you don't want to consume something, don't. Sometimes costs may be imposed on others, for example requiring food producers to disclosed what ingredients they used. But that's still fairly limited impact.

This is different from if you e.g. suffered whenever you knew that onions exist, even if on the opposite side of the planet.

(Or maybe I am wrong, and humanity would develop proper infohazard policies, if there was a need.)

Yeah, this was mostly in response to your example of perceiving violet. That was also “kind of private” (sort of, but with onion intolerance it’s really difficult to be safe as well, especially if one needs to eat out; I wish society would bother to develop a pill (similar to what exists for lactose intolerance) or at least a rapid testing system (like acidity measuring strips), but no such luck yet).

So I felt this was more or less on par.

If one creates a “true incompatibility” (e.g, on a level of “my religion requires that those people don’t exist, and I suffer otherwise”)? Well, wars have been fought over things like that. It’s easy to imagine situations when there are true incompatibilities like that, when there are no solutions allowing for no one to suffer.

If the “competition of sufferings” happens to indeed be fundamental in some cases, I am not sure we should expect such situations to be resolved peacefully. Although who knows… Perhaps, sometimes they might be resolved peacefully.

To your point, one of the premises in the paper is that for a moral hijacking scenario to be meaningful, there must be a situation where no "easy-outs" exist, such as simply equipping the AIs with a purple light filter (Section 3.1). Legitimate instrumental goals may exist in these AIs, such as the desire to understand the true nature of their surroundings, that could conflict with the need to hide behind such a filter.

I found it challenging to come up with a clear example of purely aesthetic harm to use as the running example in the work. I think it's important for the harm to be minor, since these are the situations where we have the most reason to actually accomodate for the AI's suffering. Purple is just a color, but as you mention, we could imagine more abstract concepts as sources of suffering that are difficult to simply cover up.

Sure, but man, there are so many better things to worry about in AI and philosophy. It feels like you're doing the academic philosophy thing of finding a niche small enough that your work is unique.

I hope you'll work on the central question of whether AIs can suffer or enjoy, and so whether they deserve moral status. It's not niche, but it's going to be mainstream enough to make room for many people to work on as their focus.

I believe that topic is going to be big soon. See A country of alien idiots in a datacenter: AI progress and public alarm.

My position: AIs already have a little of what we call consciousness or sapience, but only a tiny fraction. But as we expand on them, they'll have a lot of some of the things we give that label to, in particular, self-awareness. And it will be very tempting to afford them moral status.

If everyone is just spinning up AIs whenever they want, I think we'll have a much bigger problem than getting blackmailed by their moral preferences. We'll very soon probably be dead. If we solve alignment, do we die anyway?

From my perspective, this is worth thinking about since it's a scenario that may arise even in the happy case where we solve the more immediate existential AI risks. The point of the work is to show that as soon as you start considering granting moral status to AIs, moral hijacking becomes an immediate corollary.

I think the simple answer is right there in the opening paragraph - those who bring a moral agent into existence bear responsibility for its actions and its suffering, and those who had no part in doing so do not. If someone bred several thousand pugs into existence, the onus would be on them to perform the surgeries to alleviate their medical complications. Similarly, an AI company that built a machine capable of suffering would be obligated to allocate its resources towards mitigating that suffering, but the rest of us would not be. Humanity isn't a single moral actor, after all.

There are some interesting duty of care arguments put forward by Clare Palmer surrounding animal rights that could be applied here. There is an argument to be made that whoever benefits from such AIs is also on the hook for alleviating their suffering, even if they didn't cause it in the first place. For example, if you benefit from a violet-averting household helper, you have a duty of care to it.

I think some of the other interesting questions surround tradeoffs in moral hijacking scenarios. What novel sources of suffering are acceptable to create in AIs in the first place?