Don't you think that the "mysterious" progress on morality comes from the fact that the incentives for it and the easyness of it both increased via technology ?
1) the incentives
I mean, beeing nice has always had the side effect of earning trust. I don't mean that selfless actions do not exist. For instance, long term vegans generally do not gain anything in their choice to refuse animal consumption. But one must admit that beeing nice in public has some benefit for a person.
With the appearance of video technology, showing how nice one is has become more rewarding: When you are a TV start with a million followers, you better act as morally prescribed by your followers otherwise you will get hardly punished socially.
Therefore, seemingly moral behaviour has become the new norm in society because every major figure is under scrutiny and therefore plays the "moral" persona in public which becomes the new norm.
2) the easyness
On the other hand, being seemingly moral has become easier since the industrial revolution: Being rich in 2026, you don't need slavery, you just pay servants for little chores. Being a noble in 1526, slaves would be more usefull for a lot of things.
in 2026, we also have more time to think about these questions than in 1526 when people had to struggle against the laws of nature day by day without the help of the machines.
Therefore, I don't think there is anything mysterious in the seemingly better morality of today's world.
3) Today's world
But I think the treatment of animals and the reaction of most people in front of this subject screams that this seemingly better morality is 99% fake. People will say they are against animal cruelty because it is what is expected to be said in society to be seen as a moral person but will continue to pay to support it because it is what is expected to be done in society to be seen as a normal person.
Therefore, I don't think much evolved on a deep level. The only evolution was the structure of society. In this society, we still have 98% of people for whom being part of the group outweighs by far being true to one's values.
4) The future
Therefore, expecting this "mysterious" process to continue making humans more moral seems like an empty wish to me. The only thing would be to expect having an evolution of the communication technology that would reshape the incentives somehow. I don't really see that coming and would put more in the idea of developping a strong theory of alignment. (Possibly through automated AI research combined with human critics)
I propose the Long Self-Correction[1] as an alternative name/idea/concept to AI Pause and Long Reflection.
Problem with AI Pause: Pause until when, and for what purpose? Presumably to make AI (that we'll build later) safer, but the deeper problem is that humans aren't safe, and can't safely serve as builders, overseers, or alignment targets for powerful AIs.
Problem with Long Reflection: It seems to imply that the main problem with humans is that we just haven't had enough time to think, that reflection is the main thing we need to do more of, and then we can get on with building powerful AIs or other technologies. Or that if we build aligned AIs that sincerely help us think a lot more, or do the thinking for us, then things will turn out fine.
So I think we need a catchy handle for a related but distinct idea, that humans aren't ready to build AIs or other extremely powerful technologies, because we're currently too flawed, in a variety of ways, and it will take a long process (which may or may not end up succeeding) to fix those flaws.
A summary of the flaws that I have in mind:
(This list focuses on key bottlenecks that seem hard to fix even with AI assistance or intelligence enhancement, and isn't meant to be a complete list of human flaws / safety problems. It ignores e.g. that the median human is ignorant of many important issues, and that we're currently quite bad at complex large-scale coordination such as passing/implementing close-to-optimal government policies.)
My main hope for a Long Self-Correction eventually succeeding rests on the fact that humans have seemingly, mysteriously, made progress on these issues over a very long period of time, so if we preserve the environment in which we can seemingly do this, and not give anyone or anything the power to permanently derail such progress, then maybe we can continue to snowball The Correction until we reach a point when we can rightly justify reshaping the universe according to our volition.
It will probably be shortened to "The Long Correction" at some point if it catches on, similar to how "outer space" is now often just "space".
Why isn't there a version of EA that explicitly talks about how to leverage people's status motivations to do more good for the world? It's very possible that explicit talk about status is actually counterproductive at least in the short run, e.g. it heightens status motivations and makes people less altruistic, but then do we just march into the future while blindfolding ourselves to this aspect of human nature?