David Chalmers describes the inverted qualia thought experiment, in The Conscious Mind, as an argument against logical supervenience of phenomenal experience on physical states:

one can coherently imagine a physically identical world in which conscious experiences are inverted, or (at the local level) imagine a being physically identical to me but with inverted conscious experiences. One might imagine, for example, that where I have a red experience, my inverted twin has a blue experience, and vice versa. Of course he will call his blue experiences "red," but that is irrelevant. What matters is that the experience he has of the things we both call "red"---blood, fire engines, and so on---is of the same kind as the experience I have of the things we both call "blue," such as the sea and the sky. The rest of his color experiences are systematically inverted with respect to mine, in order that they cohere with the red-blue inversion. Perhaps the best way to imagine this happening with human color experiences is to imagine that two of the axes of our three-dimensional color space are switched---the red-green axis is mapped onto the yellow-blue axis, and vice versa. To achieve such an inversion in the actual world, presumably we would need to rewire neural processes in an appropriate way, but as a logical possibility, it seems entirely coherent that experiences could be inverted while physical structure is duplicated exactly. Nothing in the neurophysiology dictates that one sort of processing should be accompanied by red experiences rather than by yellow experiences.

There are quite a lot of criticisms of this sort of argument. Chalmers addresses some of them, such as the idea that this isn't neurologically plausible for humans (he brings up aliens with more symmetric color neurology as a counter). A principled approach to criticism is to track the semantics of "looking red" through mutually interpretable language, as Wilfred Sellars does in Empiricism and the Philosophy of Mind. This is, unfortunately, somewhat laborious, and could easily fail to connect with qualia realist intuitions.

I'll take a more philosophically modest approach: analyzing the consequences of the hypothetical, with group theory. Hopefully, this will make clearer requirements that must be satisfied in the hypothetical, and make progress towards isolating cruxes.

Let's start by considering a slightly broader space of color qualia operations that include red/blue inversion. We could think of a color in standard form as a triple of numbers in RGB order. Call an operation that permutes the channels (e.g. swapping red and blue) a channel permutation. The group of channel permutations is $S_3$, the symmetric group of permutations of a 3-element set. We can write the channel permutations as RGB for the identity, BGR for red/blue inversion, BRG for a red-to-green rotation, and so on. Channel permutations compose as, for example, $GRB\cdot BGR=GBR$; group composition "applies the right element first". Each channel permutation can be categorized as either being the identity, swapping two channels, or rotating channels in either the red-to-green or green-to-red directions; there are 6 elements of $S_3$ in total.

At a high level, we will construct the color qualia space category CQS as the category of functors $[S_3,Set]$, where $S_3$ is the group construed as a single-object category. This is, of course, highly abstract, so let's go step by step.

A color qualia space has an associated set of elements. Intuitively, these represent data structures that contain colors. The color qualia space also specifies a way to apply $S_3$ group elements to these set elements. More formally, a color qualia space is a pair $(Q,p)$ where $p:S_3\times Q\to Q$ is group homomorphic in its first argument: $p(RGB,q)=q$ and $p(a\cdot b,q)=p(a,p(b,q\left)\right)$. Since $p$ is generally clear from context, we also write $p(a,q)$ as $a\ast q$. (Note, p is a group action).

As an example, 100 by 100 images, where each pixel has numbers for each of the three color channels, form a color qualia space, where the group action (permuting channels) maps across each pixel. The function $q\mapsto GRB\ast q$ performs a red/green swap on its image argument, like a shader.

We want to consider maps between color qualia spaces, but we need to be careful. In the inverted qualia thought experiment, we could imagine that the original and their twin both look at a stop sign, and then yell "RED" if the stop sign's corresponding color qualia are closest to the red primary color. But then the original and the twin would behave differently, contradicting physical identicality. Going with the hypothetical, their mental operations on their color percepts can't disambiguate the channels too much. In some sense, operations mapping qualia to qualia (such as, taking their visual field and imagining transformations of it) have to be working isomorphically despite the red-blue inversion.

The concept of equivariance in group theory is a rather strong version of this. In this case, an equivariant map $f:Q\to R$ between color qualia spaces $Q,R$ has, for any $a\in S_3$, the equality $f(a\ast x)=a\ast f\left(x\right)$. Intuitively, this means the function acts symmetrically on channels, not picking out any one as special, and not identifying the chirality of the channels. If the twin's qualia were red/blue inverted from the original's before the equivariant map, they remain so after both apply the map.

Color qualia spaces and equivariant maps between them form a category, $CQS\cong [S_3,Set]$. Let's quickly list some examples of equivariant maps:

• Mapping an image to its gray-scale variant.
• Setting the right half of an image to gray-scale.
• Mapping a function from channel value to channel value (e.g. restricting to a range) over all channel values in an image.
• Taking the dominant primary color of the top-left pixel of an image, and then inverting the two other primary colors' channels throughout the whole image

And some examples of non-equivariant maps:

• Mapping an input image to an output image representing text spelling "red", "green", or "blue", depending on the dominant primary color in the input image.
• Mapping an image $q$ to $b\ast q$ for a non-trivial $S_3$ element $b$, i.e. swapping two channels, or rotating channels.

Let's examine the last point. Suppose $f\left(y\right)=b\ast y$ is a function on images. Now to check equivariance, we ask if $\forall a\in S_3,\forall x,b\ast a\ast x=a\ast b\ast x$. But this is only true when $b=RGB$, the group identity. Note $S_3$ is not Abelian.

What are the philosophical consequences? We could vaguely imagine that both the original and the twin mentally rotate red qualia towards green qualia, green qualia towards blue qualia, and blue qualia towards red qualia (perhaps imagining a transformation to their visual field). But this operation (BRG) does not commute with the original inversion (BGR).

Suppose a third person tells both the original and the twin: "Imagine applying BRG to your visual field". The original interprets this "correctly", so actually does BRG. But the twin thinks, "To apply BRG, in my red (color of blood) channel" --- actually blue qualia --- "I put the value of my blue (color of ocean) channel" --- actually red qualia. So the twin actually implements the opposite rotation, GBR!

If you and the twin could both actually apply BRG, and "naively" apply it (in the way that causes the twin to do GBR as above), then you would get the same results naively and actually, while the twin would get different results. Presumably the twin would notice this difference, so we must reject some premise (the original and the twin are supposed to behave the same). And of course, naive application is more straightforward. So we must conclude that you and the twin can't actually both apply BRG.

Let's further characterize equivariant maps. Given a color qualia space $(Q,p)$, the orbit of an element $q\in Q$ is the set of elements reachable through group actions, $\{a\ast q|a\in S_3\}$. Now let the orbit map ${\pi }_Q:Q\to Q/S_3$ map elements to their orbits, effectively quotienting over channel permutations. The orbit map relates to equivariance in the following way: for any equivariant $f:Q\to R$, there is a unique function on orbits $g:Q/S_3\to R/S_3$ commuting, $g\circ {\pi }_Q={\pi }_R\circ f$.

(Why is this true? Note $f$ must map elements of a single Q-orbit to a single R-orbit, which allows defining $g$ commuting. Any alternative choice would fail commutation on some Q-orbit.)

An orbit is itself a color qualia space (a subspace of the original), and must have size 1, 2, 3, or 6 (by sub-group analysis). We can characterize orbits of a given size as isomorphic to a standard qualia space of that (finite) size. Explicitly:

• The size-1 qualia space $C{Q}_1$ has elements $\left\{\varnothing \right\}$; it is trivial.
• The size-2 qualia space $C{Q}_2$ has elements $\{L,R\}$ representing chirality. Channel reflections (RBG, GRB, BGR) flip chirality, rotations preserve it. (Imagine three balls corresponding to primary colors, with sticks connecting them in a triangle; chirality-reversing operations require flipping the triangle over vertically.)
• The size-3 qualia space $C{Q}_3$ has elements {R, B, G} representing primary colors. Group operations work straightforwardly, e.g. $BGR\ast B=R$.
• The size-6 qualia space $C{Q}_6$ has as elements total orderings of primary colors (e.g. R > B > G), of which there are 6. Group operations work straightforwardly, e.g. $BGR\ast (R>B>G)=(B>R>G)$.

Now let's consider equivariant maps between these standard qualia spaces. Any equivariant map between these must be either: some space to itself, some space to $C{Q}_1$, or $C{Q}_6$ to anything. (For example, there are no equivariant maps from $C{Q}_1$ to $C{Q}_6$.) So we can reduce the 4 x 4 of signatures $C{Q}_i\to C{Q}_j$ to only 9 realizable signatures. 4 of these signatures are clearly trivial, as they map to $C{Q}_1$. Exhaustively analyzing the possible equivariant maps of the non-trivial signatures:

• There are two equivariant maps $C{Q}_2\to C{Q}_2$: identity and chirality-reversal.
• There is only one equivariant map $C{Q}_3\to C{Q}_3$, the identity.
• There are two equivariant maps $C{Q}_6\to C{Q}_2$, which assign distinct chiralities to the two cyclic orbits, {(R > G > B), (B > R > G), (G > B > R)} and {(B > G > R), (R > B > G), (G > R > B)}.
• There are three equivariant maps $C{Q}_6\to C{Q}_3$, which pick out either the greatest, middle, or least primary color.
• There are six equivariant maps $C{Q}_6\to C{Q}_6$ corresponding to each of the $S_3$ permutations applying to the ordering, e.g. swapping first and second places.

In tabular form, cardinalities of equivariant map sets $|Ho{m}_{CQS}(C{Q}_i,C{Q}_j)|$ are as follows:

We now have a combinatorial characterization of equivariant maps in general. First determine how $f$ maps input orbits to output orbits. Then for each input orbit ${\pi }_Q\left(x\right)$, determine how $f$ equivariantly maps it to the corresponding output orbit ${\pi }_R\left(f\right(x\left)\right)$, which has a finite combinatorial characterization.

In the reverse direction, we can form a valid equivariant map by first choosing a map on orbits $g:Q/S_3\to R/S_3$, then for each input orbit, selecting an equivariant map to the output orbit. Formally, we could write the set of such maps as:

$Ho{m}_{CQS}(Q,R)\cong {\Pi }_{S\in Q/S_3}{\Sigma }_{T\in R/S_3}Ho{m}_{CQS}(S,T)$

where $Ho{m}_{CQS}(S,T)$ is the set of equivariant maps between color qualia spaces S and T, and $\Pi ,\Sigma$ are set-theoretic dependent product and sum.

We now have a fairly direct characterization of equivariant maps in CQS. They aren't exactly characterizable as "functions between quotient spaces" as one might have expected. Instead, they carry extra orbit-to-orbit information, although this information is combinatorially simple for any given pair of orbits.

What are the philosophical implications? To the color qualia realist, CQS decently characterizes mental operations on color qualia that don't break the physical symmetry, in thought experiments such as inverted qualia. Equivariance is mathematically natural, and rules out non-realizable mental operations such as BRG. Analysis of CQS, including combinatorial characterization of equivariant maps, provides a functional analysis relevant to the physics of the situation, which (according to Chalmers) doesn't actually involve color qualia as physical entities.

To the color qualia non-realist, the functional characterization of color qualia intuitions through CQS could provide a hint as to a specific error or illusion the color qualia realist is subject to. The non-realist could expect that, once the functional physical component to the intuition is characterized, there is not a remaining reason to expect color qualia to exist above and beyond such physics and physical functions.

I have found CQS to be clarifying with respect to the inverted qualia thought experiment: equivariance provides a mathematically simple constraint on realizability of mental operations in the scenario. In particular, CQS analysis led me to correct an intuition that channel rotations such as BRG would be realizable, and the combinatorial characterization showed (non-obviously) that maps between quotient spaces are insufficient. My own view is that inverted qualia arguments are fairly weak, and that CQS analysis has some relevance to showing the weakness, but the fuller case would require engaging with the relationship between phenomenal experience and belief-formation.

(If you like the idea of a circular "color wheel" rather than a three-channel "color cube", you may consider the (also non-Abelian) orthogonal group O(2), and the continuous qualia space category $\left[O\right(2),Set]$.)