This post is a response to a claim by Scott Sumner in his conversation at LessOnline with Nate Soares, about how ethical we should expect AI's to be.

Sumner sees a pattern of increasing intelligence causing agents to be increasingly ethical, and sounds cautiously optimistic that such a trend will continue when AIs become smarter than humans. I'm guessing that he's mainly extrapolating from human trends, but extrapolating from trends in the animal kingdom should produce similar results (e.g. the cooperation between single-celled organisms that gave the world multicellular organisms).

I doubt that my response is very novel, but I haven't seen clear enough articulation of the ideas in this post.

To help clarify why I'm not reassured much by the ethical trend, I'll start by breaking it down into two subsidiary claims:

1. The world will be dominated by entities who cooperate, in part because they use an ethical system that is at least as advanced as ours.

2. Humans will be included in the set of entities with whom those dominant entities cooperate.

Claim 1 seems well supported by trends that economists such as Sumner often focus on. I doubt that Nate was trying to argue against this claim. I'll give it a 90+% chance of turning out to be correct. Sumner's point sounds somewhat strong because it focuses on an important, and somewhat neglected, truth.

Claim 2 is where I want to focus most of our concern. The trends here are a bit less reassuring.

There's been a clear trend of our moral circle expanding in the direction that we currently think it should expand. How much of that should we classify as objective improvements versus cultural fads? Claim 1 is often measured by fairly objective criteria (GDP, life expectancy, etc.). In contrast, we measure expansion of our moral circle by the criteria of our current moral standards, giving us trends that look about as good if they're chasing fads as they do if the trends will stand the test of time.

Gwern provides extensive pushback against strong claims that moral circle expansion is a consistent trend.

I'll add one example: children have increasingly had their freedom to wander restricted during my lifetime (see the free range parenting movement). It's almost as if they're considered to be like zoo animals, with their caretakers optimizing for safety at the expense of happiness. I don't find it hard to imagine a future where AI treats us like that.

The obvious versions of the moral circle expansion hypothesis suggest that we should expect human societies to soon grant moral patienthood to animals.

Blindly maximizing the size of our moral circle would be more problematic than maximizing cooperation. It's pretty unlikely that we will want to expand our moral circle to include pet rocks. It sure looks to me like moral circle expansion has been driven largely by pragmatic evaluations of costs and benefits, with only a modest influence from increasingly principled altruism.

Given this uncertainty about how closely our moral circle expansion approximates attaining an objective moral truth about who should be a moral patient, we ought to be more uncertain about it than we are about the continuation of economic progress.

I expect that whether humans remain in the moral circle depends on hard-to-predict factors, such as the costs associated with interacting with humans, or whether AIs have relevant altruistic goals.

I recommend looking at examples such as human interactions with cats and chickens as possible analogies for how beings with very different cognitive abilities might interact. My impression is that increasing human intelligence does not make those interactions more ethical, but increasing human wealth weakly tends to make the interactions more ethical. Humans seem mostly aware of ethical concerns regarding factory farmed chickens, yet their reactions seem mostly influenced by cost considerations rather than improved ethical insights.

So I'm only weakly reassured by the trend that Sumner sees.