Consider any finite two-player game in normal form (each player can have any finite number of strategies, we can also easily generalize to certain classes of infinite games). Let $S_A$ be the set of pure strategies of player $A$ and $S_B$ the set of pure strategies of player $B$. Let $u_A:S_A\times S_B\to R$ be the utility function of player $A$. Let $(\alpha ,\beta )\in \Delta S_A\times \Delta S_B$ be a particular (mixed) outcome. Then the alignment of player $B$ with player $A$ in this outcome is defined to be:

$$a_{B/A}(\alpha ,\beta ):=\frac{E_{\alpha \times \beta }\left[u_A\right]-{min}_{{\beta }^{\prime }\in S_B}{E}_{\alpha \times {\beta }^{\prime }}\left[u_A\right]}{{max}_{{\beta }^{\prime }\in S_B}{E}_{\alpha \times {\beta }^{\prime }}\left[u_A\right]-{min}_{{\beta }^{\prime }\in S_B}{E}_{\alpha \times {\beta }^{\prime }}\left[u_A\right]}\in [0,1]$$

Ofc so far it doesn't depend on $u_B$ at all. However, we can make it depend on $u_B$ if we use $u_B$ to impose assumptions on $(\alpha ,\beta )$, such as:

• $\beta$ is a $u_B$-best response to $\alpha$ or
• $(\alpha ,\beta )$ is a Nash equilibrium (or other solution concept)

Caveat: If we go with the Nash equilibrium option, $a_{B/A}$ can become "systematically" ill-defined (consider e.g. the Nash equilibrium of matching pennies). To avoid this, we can switch to the extensive-form game where $B$ chooses their strategy after seeing $A$'s strategy.

So, something like "fraction of preferred states shared" ? Describe preferred states for P1 as cells in the payoff matrix that are best for P1 for each P2 action (and preferred stated for P2 in a similar manner) Fraction of P1 preferred states that are also preferred for P2 is measurement of alignment P1 to P2. Fraction of shared states between players to total number of preferred states is measure of total alignment of the game.

For 2x2 game each player will have 2 preferred states (corresponding to the 2 possible action of the opponent). If 1 of them will be the same cell that will mean that each player is 50% aligned to other (1 of 2 shared) and the game in total is 33% aligned (1 of 3), This also generalize easily to NxN case and for >2 players.

And if there are K multiple cells with the same payoff to choose from for some opponent action we can give 1/K to them instead of 1.

(it would be much easier to explain with a picture and/or table, but I'm pretty new here and wasn't able to find how to do them here yet)

I think this is backward.  The game's payout matrix determines the alignment.  Fixed-sum games imply (in the mathematical sense) unaligned players, and common-payoff games ARE the definition of alignment.  

When you start looking at meta-games (where resource payoffs differ from utility payoffs, based on agent goals), then "alignment" starts to make sense as a distinct measurement - it's how much the players' utility functions transform the payoffs (in the sub-games of a series, and in the overall game) from fixed-sum to common-payoff.

Another point you could fix using intuition would be complete disinterest. It makes sense to put it at 0 on the [-1, 1] interval.

Assuming rational utility maximizes, a board that results in a disinterested agent would be:

1/0  1/1

0/0 0/1

Then each agent cannot influence the rewards of the other, so it makes sense to say that they are not aligned.

More generally, if arbitrary changes to one players payoffs have no effect on the behaviour of the other player, then the other player is disinterested.

Correlation between player payouts? In a zero sum game it is -1, when payouts are perfectly aligned it is +1, if payouts are independent it is 0.

I'll take a shot at this. Let $A$ and $B$ be the sets of actions of Alice and Bob. Let $o_n:B\to \{1,...n\}$ (where 'n' means 'nice') be function that orders $B$ by how good the choices are for Alice, assuming that Alice gets to choose second. Similarly, let $o_s:B\to \{1,...,n\}$ (where 's' means 'selfish') be the function that orders $B$ by how good the choices are for Bob, assuming that Alice gets to choose second. Choose some function $\psi$ measuring similarity between two orderings of a finite set (should range over $[-1,1]$); the alignment of $B$ with $A$ is then $\psi (o_n,o_s)$.

Example: in the prisoner's dilemma, $B=\{c,d\}$, and $o_n$ orders $c>d$ whereas $o_s$ orders $d>c$. Hence $\psi (o_n,o_m)$ should be $-1$, i.e., Bob is maximally unaligned with Alice. Note that this makes it different from Mykhailo's answer which gives alignment $0.5$, i.e., medium aligned rather than maximally unaligned.

This seems like an improvement over correlation since it's not symmetrical. In the game where Alice and Bob both get to choose numbers $x,y\in \{1,2\}$ and Alice's utility function outputs $y+x$ whereas Bob's outputs $y-x$, Bob would be perfectly aligned with Alice (his $o_n$ and $o_s$ both order $2>1$) but Alice perfectly unaligned with Bob (her $o_n$ orders $1>2$ but her $o_s$ orders $2>1$).

I believe this metric meets criteria 1,3,4 you listed. It could be changed to be sensitive to players' decision theories by changing $o_s$ (for alignment from Bob to Alice) to be the order output by Bob's decision theory, but I think that would be a mistake. Suppose I build an AI that is more powerful than myself, and the game is such that we can both decide to steal some of the other's stuff. If the AI does this, it leads to -10 utils for me and +2 for it (otherwise 0/0); if I do it, it leads to -100 utils for me because the AI kills me in response (otherwise 0/0). This game is trivial: the AI will take my stuff and I'll do nothing. Also, the AI is maximally unaligned with me. Now suppose I become as powerful as the AI and my 'take AI's stuff' becomes -10 for AI, +2 for me. This makes the game a prisoner's dilemma. If we both run UDT or FDT, we would now cooperate. If $o_s$ is the ordering of the AI's decision theory, this would mean the AI is now aligned with me, which is odd since the only thing that changed is me getting more powerful. With the original proposal, the AI is still maximally unaligned with me. More abstractly, game theory assumes your actions have influence on the other player's rewards (else the game is trivial), so if you cooperate for game-theoretical reasons, this doesn't seem to capture what we mean by alignment.

Alright, here comes a pretty detailed proposal! The idea is to find out if the sum of expected utility for both players is “small” or “large” using the appropriate normalizers.

First, let's define some quantities. (I'm not overly familiar with game theory, and my notation and terminology are probably non-standard. Please correct me if that's the case!)

• $A.$ The payoff matrix for player 1.
• $B.$ The payoff matrix for player 2.
• $s,r$ the mixed strategies for players 1 and 2. These are probability vectors, i.e., vectors of non-negative numbers summing to 1.

Then the expected payoff for player 1 is the bilinear form $s^{T}Ar={\sum }_{i,j}{s}_i{a}_{ij}{r}_j$ and the expected payoff for player 2 is $s^{T}Br={\sum }_{i,j}{s}_i{b}_{ij}{r}_j$. The sum of payoffs is $s^T(A+B)r.$

But we're not done defining stuff yet. I interpret alignment to be about welfare. Or how large the sum of utilities is when compared to the best-case scenario and the worst-case scenario. To make an alignment coefficient out of this idea, we will need

•  $l(A,B).$ This is the lower bound to the sum of payoffs, $l(A,B)={min}_{u,v}\left[u^T\right(A+B\left)v\right]$, where $u,v$ are probability vectors. Evidentely, $l(A,B)=min(A+B).$
• $u(A,B).$ The upper bound to the sum of payoffs in the counterfactual situation where the payoff to player 1 is not affected by the actions of player 2, and vice versa. Then $u(A,B)={max}_{u,v}{u}^{T}Av+{max}_{u,v}{u}^{T}Bv$. Now we find that $u(A,B)=maxA+maxB$.

Now define the alignment coefficient of the strategies $(s,r)$ in the game defined by the payoff matrices $(A,B)$ as 

The intuition is that alignment quantifies how the expected payoff sum $s^T(A+B)r$ compares to the best possible payoff sum $u(A,B)$ attainable when the payoffs are independent. If they are equal, we have perfect alignment $(a=1)$. On the other hand, if $s^T(A+B)r=l(A,B)$, the expected payoff sum is as bad as it could possibly be, and we have minimal alignment ($a=0$). 

The only problem is that $u(A,B)=l(A,B)$ makes the denominator equal to 0; but in this case, $u(A,B)=s^T(A+B)r$ as well, which I believe means that defining $a=1$ is correct. (It's also true that$l(A,B)=s^T(A+B)r$, but I don't think this matters too much. The players get the best possible outcome no matter how they play, which deserves $a=1$.) This is an extreme edge case, as it only holds for the special payoff matrices $A$ ($B$) that contain the same element $a$ ($b$) in every cell. 

Let's look at some properties:

• A pure coordination game has at least one maximal alignment equilibrium, i.e., $a(s,r)=1$ for some $s,r$. All of these are necessarily Nash equilibria.
• A zero-sum game (that isn't game-theoretically equivalent to the 0 matrix) has $a=0$ for every pair of strategies $(s,r)$. This is because $s^{T}Ar+s^{T}Br=l(A,B)=0$ for every $s,r$. The total payoff is always the worst possible.
• The alignment coefficient is linear in a specific senst, i.e., $a(A,B)=a(aJ+dA,bJ+dB),$ where $J$ is the matrix consisting of only $1$s.

Now let's take a look at a variant of the Prisoner's dilemma with joint payoff matrix

Then 

The alignment coefficient at $(s,r)$ is

Assuming pure strategies, we find the following matrix of alignment, where $a_{ij}$ is the alignment when player 1 plays $i$ with certainty and player 2 plays $j$ with certainty.

Since$s=r=(0,1)$ is the only Nash equilibrium, the “alignment at rationality” is 0. By taking convex combinations, the range of alignment coefficients is $[0,1/2]$.

Some further comments:

• Any general alignment coefficient probably has to be a function of $(s,r)$, as we need to allow them to vary when doing game theory. 
  • Specialized coefficients would only report the alignment at Nash equilibria, maybe the maximal Nash equilibrium.
  • One may report the maximal alignment without caring about equilibrium points, but then the strategies do not have to be in equilibrium, which I am uneasy with. The maximal alignment for the Prisoner's dilemma is 1/2, but does this matter? Not if we want to quantify the tendency for rational actors to maximize their total utility, at least.
• Using e.g. the correlation between the payoffs is not a good idea, as it implicitly assumes the uniform distribution on $s,r$. And why would you do that?

Quick sketch of an idea (written before deeply digesting others' proposals):

Intuition: Just like player 1 has a best response (starting from a strategy profile $s$, improve her own utility as much as possible), she also has an altruistic best response (which maximally improves the other player's utility).

Example: stag hunt. If we're at (rabbit, rabbit), then both players are perfectly aligned. Even if player 1 was infinitely altruistic, she can't unilaterally cause a better outcome for player 2.

Definition: given a strategy profile $s$, an $a$-altruistic better response is any strategy of one player that gives the other player at least $a$ extra utility for each point of utility that this player sacrifices.

Definition: player 1 is $a$-aligned with player 2 if player 1 doesn't have an $x$-altruistic better response for any $x>a$.

0-aligned: non-spiteful player. They'll give "free" utility to other players if possible, but they won't sacrifice any amount of their own utility for the sake of others.

$c$-aligned for $c\in (0,1)$: slightly altruistic. Your happiness matters a little bit to them, but not as much as their own.

1-aligned: positive-sum maximizer. They'll yield their own utility as long as the total sum of utility increases.

$c$-aligned for $c\in (1,\infty )$: subservient player: They'll optimize your utility with higher priority than their own.

$\infty$-aligned: slave. They maximize others' utility, completely disregarding their own.

Obvious extension from players to strategy profiles: How altruistic would a player need to be before they would switch strategies?