I am very happy that Pope Leo takes these issues seriously, and is sharing his views, and bringing a form of moral clarity, even with all the flaws and central errors. More people with voice should share their views in this way, even when I disagree.
It’s a weird document. Much of it is not about AI at all.
I do agree with the Pope’s most basic point on AI, which is that AI can be what we make of it. That we can steer this technology, determine how it is developed and used, and this can determine whether we get a good or not so good future. We cannot purely leave this to market incentives and strategic pressures. Yes, very much so.
The central problem is that so much of Leo’s worldview is some combination, to me, of highly alien and highly wrong. You might think that would primarily have a lot to do with him being the Pope and rather Catholic, and being a man of faith, whereas I am not these things.
If so, you would be wrong. That seems to have remarkably little to do with all of this.
There was also a lot of good here, but I was centrally disappointed on three fronts:
The central claim, wherein Leo denies that AIs can think or importantly be minds, is wrong, as Olah points out in his statements.
Without the understanding of what AI is capable of becoming, the document effectively only deals with relatively mundane AI dangers and changes, although that on its own is still rather quite a lot to deal with and discuss.
Pope Leo subscribes to a view of economics and a System of the World that I believe are simply wrong about what actions and systems cause what consequences, subscribing to what is effectively an institutionalist, European technocrat, left-wing social justice socialist labor-centered perspective, especially with treating the role of the economy as creating and protecting ‘good jobs.’ To his credit and that of previous Popes, they do understand the central value of development and growth, but they reject the ways we get there in practice.
You want some amount of people pushing in the direction of peace and mercy and dialogue and watching out for the poor and disempowered, and calling on us to do more for our fellow man. So as part of a balanced bigger picture, this could be actively good, but Europe has shown us the peril of lacking this balance.
This post will summarize the whole thing, going number by number, with occasional commentary focused on the key AI section in the middle.
Anthropic cofounder Chris Olah visited the Vatican for the occasion. He endorsed the document, but also offered remarks disagreeing with the central point (paragraph 99). I’ll discuss that afterwards, along with how the media viewed the release.
Chapters 1 and 2 lay out some history. Any Pope is going to be a huge history nerd and set all this in its historical context. Leo does not disappoint.
Christianity is The Way.
Christianity is The Way.
If you see something [on Earth that matters], say something.
Tech can be good or bad. We decide which.
We must regulate, and also other things. Tech power grows private.
Which way, modern man?
Tower of Babel means various good things are actually bad. Confused.
Rebuilding of Jerusalem via Nehemiah shows value of diversity? Confused.
Tech can be good or bad. We decide which.
We must avoid ‘Babel syndrome’ and instead choose Way of Nehemiah. Diversity. Avoid a common language. Good things like peace, justice, fraternity, God.
God is The Way.
Do not try to fix the limits and weaknesses of humanity. That can lead to inequality. True fulfillment is about the least well off people.
From each according to his ability, to each according to his need.
Speak no evil. All power to our socialist central planners.
Stay human, my friends. Listen.
God is The Way. Work only for the common good. Get your ‘hands dirty.’
We must go over the history of documents like this.
We must first review fundamental principles, and why life on Earth matters.
The Church should try to help improve the world. Tikkun olam.
The Church no longer gets to tell the State what to do.
Second Vatican Council affirmed that, but we’ll still try and give you a push.
The Church still speaks with moral authority.
The Church loves science, truth, goodness, beauty, if you do too we’re friends.
The Church has moral authority but science has authority over other things.
Speak and be open to truth. Welcome diversity. Avoid power and violence.
The Church does not have all the answers in the Earthly realm.
The Church social doctrine is an ongoing process.
We now review the history of the Church social doctrine.
Leo XIII was first to have the Church address these questions.
Leo XIII chose dignity and workers over profits but defends private property. We continue to affirm the primacy of labor and justice. We cannot stand aside.
Pius XI in 1931 reaffirmed all this, warned against both unlimited collectivism and unlimited competition and concentration of power, and established the principle of subsidiarity: Everything should be handled as locally as possible.
Pius XII appealed to natural law, affirmed labor, opposed force and inequality.
John XXIII affirmed all this, truth, love, freedom, justice, et al.
Second Vatican Council engaged with the world, affirmed religious freedom.
Paul VI equated peace with universal prosperity, justice, equality.
Paul VI said Gospel called for everyone to enjoy the fruits of development.
John Paul II affirmed central importance of work and ‘fair wages.’
John Paul II hated underdevelopment and favoring of national interests.
John Paul II affirmed democracy, subordinated markets to moral law.
Benedict XVI demanded economic activity serve common good, be ‘real growth.’
Benedict XVI centered charity, evaluation of development as common good.
Francis affirms Church is social, human lives matter, the poor matter.
Francis talked about the environment, linked it to the poor, waste, justice.
Francis proposed we all work for common good, said Jesus is The Way.
Church Social Doctrine responds to what happens in the world.
On to chapter 2, foundations and principles of church doctrine.
We reflect on: Common good, subsidiarity, solidarity, social justice.
You should implement these principles in your daily life.
God is love. The trinity is love. And relationship and sharing.
Jesus cared about people and wanted us to work to make the world better.
Humans are in God’s image, have dignity, relate to God, develop.
Humans all have dignity, need not justify or earn their worth. All are equal. We cannot value more those who produce or do more. Rights are inalienable.
Dignity can be social, moral, existential, ontological. All equal and inalienable.
This dignity, of every human, is infinite.
Human rights are an expression of human dignity.
Human rights are thus inviolable. Abortion and euthanasia are gravely wrong.
Rights often aren’t honored and are at risk, often due to tech and power.
Women’s rights must be honored, including equal access to all the things.
Individuals and families, and them meeting their needs, are what matters.
The common good and human dignity must shape our lives.
Common good means letting all people ‘reach fulfillment.’
Self-interest cannot make a better world for families.
Life to a people comes from pursuit of the common good via shared vision.
The State must organize society in pursuit of common good, think long term.
Nations must cooperate towards this common good. Divides widen.
Using Earth’s resources to benefit the few is an affront to God.
You have the right to private property, but only in service of the common good.
Patents, algorithms, digital platforms, technology infrastructure and data also now count as ‘Earth’s resources’ and must be routed in service of common good.
Affirmation of subsidiarity.
Family and individual must not be subsumed by the state.
Affirmation of subsidiarity. Endorsement of voluntary organizations.
Technology companies violate subsidiarity, must serve the common good.
States collectively must ensure local voice and choice, avoid private tech control.
Subsidiarity requires solidarity. We are all in this together.
Full solidarity must be a conscious choice.
Solidarity is a principle and a virtue, requires modest shared ways of life, sacrifice.
Reiterated claims to collective decision making over technology.
Jesus was a big social justice guy, which means everyone gets dignity.
Start with the poorest among us.
Systems create inequality and are unjust. Boo wars, colonialism, discrimination, violence against entire peoples and ‘exploitation.’
Social justice also involves all aspects of digital technologies. People not profits.
Social justice litmus test: Migrants, refugees, those forced to move.
Development that does not ‘foster each man and of the whole man’ is ‘inauthentic.’
Development is a duty and a right, including beyond economics.
Development is measured by justice.
Tech is good if and only if it helps people become more humane and fraternal while respecting our common home and future generations.
This doctrine is an extension of the Church.
Subsidiarity is the guiding principle of governance and pastoral life.
Solidarity, for Christians, comes from Christ and the Eucharist.
The Church must face and address its legacy of abuse of power.
Economic Models Very Different From Our Own
This view, laid out in the first two chapters, is a very different perspective and worldview than my own, and this has remarkably little to do with belief the the divine or a lack thereof.
This is the Socialist perspective on economics and development and opportunity and the importance of equality and disdain for profit and self-interest, with an extreme focus on labor and jobs, which I think is wrong and leads to worse results for everyone.
This is much better than failing to care about humanity’s experience on Earth, or focusing purely on direct aid to the poor, or attempting to outright seize the resources although this doctrine is clearly flirting with quite a lot of this, and the dedication to the value of development is admirable.
The Pope is simply incorrect about where wealth and development come from, and what causes prosperity. He is also wrong about what he sees as a ‘widening gap’ between nations, whereas global inequality has been steadily falling for a while.
Arthur B.: I was hoping for something akin to Thomistic philosophy on AI, but this reads like Catholic-flavored Gebru.
Even in the places where the Pope is obviously correct, it’s often rather alarming that he needs to affirm his position out loud, as if it was a live question.
You definitely need some of this, and it would not be that Christian to have a position that much closer to my own. The Pope, it turns out, is somewhat Catholic.
A New Jerusalem
Dean W. Ball: To say a nice thing about Magnifica Humanitas: I loved the use of the rebuilding of Jerusalem from Nehemiah. We all should see our task as that of rebuilding the world and its institutions, and we should flock to the “construction sites of history,” a beautiful phrase.
I too like the story of rebuilding Jerusalem. I don’t get the Babel versus Jerusalem metaphor. Leo clearly thinks this is an important distinction, and is his central hook, but why? Babel had too much central planning and not enough community impact meetings? Babel would have been too productive and efficient, or its methods broke down and were the opposite? Was it an affront because it challenged the power of God or because it didn’t work? Did Babel’s common language break down because God decreed it, or because of the SNAFU principle? Jerusalem was rebuilding and Babel was a new unnatural thing? Why is Babel dehumanizing?
This simply doesn’t match up with the actual Babel story.
Genesis 11 1-9: Now the whole world had one language and a common speech. 2 As people moved eastward, they found a plain in Shinar and settled there.
3 They said to each other, “Come, let’s make bricks and bake them thoroughly.” They used brick instead of stone, and tar for mortar. 4 Then they said, “Come, let us build ourselves a city, with a tower that reaches to the heavens, so that we may make a name for ourselves; otherwise we will be scattered over the face of the whole earth.”
5 But the Lord came down to see the city and the tower the people were building. 6 The Lord said, “If as one people speaking the same language they have begun to do this, then nothing they plan to do will be impossible for them. 7 Come, let us go down and confuse their language so they will not understand each other.”
8 So the Lord scattered them from there over all the earth, and they stopped building the city. 9 That is why it was called Babel[c]—because there the Lord confused the language of the whole world. From there the Lord scattered them over the face of the whole earth.
I can tell a Just So Story or gloss over the details, and pretend I get it, and I know vaguely what he means and it’s probably good enough for our purposes anyway, or go off the vibes, but I do notice the whole thing doesn’t really work and the whole thing has been flipped around.
So Sayeth The Pope (on AI)
Now that we’ve laid that groundwork, we can move on to Chapter Three.
What technologies are we building? Are they good or bad?
Christians need to engage with the challenges of the modern tech world.
It is bad to make decisions based on efficiency, control and profit alone.
New techs and new power can be good or bad. The default tech paradigm of efficiency and profit will go badly. We need new frameworks.
Tech progress, without moral progress, will only backfire on us.
Tech is controlled by private actors, not States. This is opaque and evades public oversight, leading to bad things.
Social Doctrine demands we assess new techs to ensure they serve common good.
So far there has been no differentiating principle. Artificial intelligence is treated the same way as other techs. If we are going to treat AI differently than we treat other techs, we need to make clear our differentiating principle.
Finally, with #97, the Pope gets to talking about AI directly.
We won’t go over AI details here except as directly needed.
We don’t understand AI and AI is changing rapidly.
As it is one of the best statements, and an important thing to know, I’ll quote 98:
Pope Leo (paragraph 98): It is appropriate to preface this discussion with two considerations. First, any statement regarding AI risks becoming quickly outdated, given the remarkable pace at which these systems are developing. Second, all of us, including those who design them, possess only a limited understanding of their actual functioning.
Indeed, current AI systems are more “cultivated” than “built,” for developers do not directly design every detail, but instead create a framework within which the intelligence “grows.”
As a result, fundamental scientific aspects — such as the internal representations and computational processes of these systems — remain, at present, unknown. There thus emerges an urgent need for a twofold commitment: on the one hand, a deepening of scientific research; on the other, the exercise of moral and spiritual discernment.
On the one hand, You Should Know This Already, and Everybody Knows, and it’s the easiest thing in the world to call for scientific research and also moral and spiritual discernment. On the other hand, it’s rather important to get it right, and a lot of people pretend we understand AI far more than we do. Without a statement like this we would be in a lot more trouble.
We can’t even cleanly define AI, but I can tell you what AI is not…
It’s worth quoting extensively rather than paraphrasing, as this full statement is load bearing and also, well, citation needed:
Pope Leo (paragraph 99): It is not possible to provide a single, comprehensive definition of AI. What can be stated, however, is that we must avoid the misconception of equating this type of “intelligence” with that of human beings. These systems merely imitate certain functions of human intelligence. In doing so, they often surpass human intelligence in speed and computational capacity, offering tangible benefits across many fields. Yet this power remains entirely tied to data processing.
So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships and do not know from within what love, work, friendship or responsibility mean. Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences.
They may imitate language, behavior and analytical skills, or even simulate empathy and understanding, but they do not understand what they produce, for they lack the affective, relational and spiritual perspective through which human beings grow in wisdom.
Even when these tools are described as capable of “learning,” their way of doing so is different from that of a human person. It is not the experience of those who allow themselves to be shaped by life and grow over time through choices, mistakes, forgiveness and fidelity. Rather, it is a form of statistical adaptation based on data and feedback, which can be very effective, but does not imply inner growth.
A lot of this is assertion without evidence of positions that are rather non-obvious to me, and where there are those who strongly claim the other side. One could very easily argue the other side.
We have spent the first 96 paragraphs building a worldview around the least of us, of the essential dignity and value of all humanity and the poorest and least capable among us, all made in the image of God. Then the next two noticing our confusion.
And then we come along, and mostly without argument draw this barrier, where we exclude AI from both our moral circles and the group of minds that we have to model as minds, permanently, warning that doing otherwise would be a mistake, and also asserting by implication that AI is and will always be a mere tool in these ways, thus avoiding the need to discuss any of the problems surrounding existential risk.
Dean W. Ball: The reality of AI cognition is the central challenge the Church (and all of us) will have to grapple with over the coming decade, and this encyclical, with its axiomatic denial of AI cognition, is a punt of the highest order. Eppur si muove.
Dean W. Ball: The encyclical is Western academia/NGO “AI doesn’t *really* think but it *is* racist” at its core, with little bits of tegmark/FLI talking points sprinkled incoherently on top.
Matthew Yglesias: tbh I think the Catholic Church was there first on the “doesn’t really think” stuff and western academia is the derivative version.
It is not surprising to see that the Pope endorses a lot of superstitious mind/body dualism about artificial intelligence but it’s still wrong, even as he is also raising a lot of good points about some of the risks at play.
Dean W. Ball: Maybe, but the rest of the way this is written is clearly heavily influenced by academia
Nathan Beacom: I’m familiar with the intellectual milieu in Catholic Rome ,and it’s a distinct world from the one you are describing, operating in a different tradition of thought, with different foundations and concerns.
I mean, it’s not not criticizing the Pope for being Catholic, but he can change what that means, and Leo’s shown a lot of signs he can be smarter than this.
Roberto points out that this is far from a maximally denialist Catholic position on these questions. Leo is accepting that AI is highly capable in many ways. It’s a start.
It’s not obvious who is influencing who in which ways, but the result is similar. It’s entirely possible that they converged on similar places for their own reasons.
Thus, after all that, the next section is entitled ‘a valuable tool that requires vigilance.’
There are still plenty of difficult, important and interesting questions surrounding AI, but the ones that I think matter most are basically hand waved away here, at the start.
AI is highly valuable and useful, but also dangerous. It can make you excessively reliant on ready-made answers and weaken your creativity and judgment. It reflects the cultural assumptions of those that designed and trained it. It can create illusion of a relationship and destroy desire to form human connections.
AI is embedding itself in decision making. This is efficient, but there are risks, including expanding environmental costs in energy and water and carbon and natural resources.
Highlighting the environmental concerns, especially water, up top is a blunder.
AI is ‘entering processes’ that impact lives touches on rights, opportunity, status and freedom, and is thus ‘never a purely technical matter’ and decisions risk full delegation to uncaring machines. Beware manipulation of information and violation of privacy, and also bias.
This is dangerously close to a fully general argument against information or intelligence. It is, at minimum, an argument for the universality of politics and that nothing can ever be ‘purely a technical matter.’ Essentially anything that does anything is going to ‘enter processes that impact lives.’ How is this to be differentiated from a phone or a computer or an abacus or a law? Should we not delegate important decisions to predetermined processes that do not, as Leo puts it, ‘know compassion, mercy, forgiveness and above all, the hope that people are able to change?’
If anything, AI knows those things far better than the law books or the telephone, or the non-AI algorithm, and perhaps it knows them as well or better than most humans, at least terms of its ability to take such considerations into account when making decisions. Why should we presume that a human would take them into consideration better?
Manipulation of information and violation of privacy and bias are of course considerations, but they are for humans as well, often more so especially for bias. Privacy is a heightened concern in many ways, which is due to AI’s ability to process so much more information, but also AI can be privacy preserving because it can allow the processing of information while reliably not sharing that information with another party.
Letting an algorithm select who is ‘worthy’ dodges responsibility and excludes the vulnerable. Injustice goes unnoticed, compassion, mercy and forgiveness disappear.
This is a complaint about law and procedure and algorithms, rather than about AI, as again AI can actually take those things into account. Also, one should notice that in many cases strict rules are far better for the poor and vulnerable, as they lack connections and sympathy and often face discrimination and undeserved mistrust.
Consider the simple example of college admissions. In the name of ‘inequality’ certain types convinced many schools to get rid of the ‘algorithm’ of the SAT. But it turns out it goes the other way, that the SAT gives the poor students far more slots and chances than ‘holistic admissions.’ If you want the algorithm to favor those who need help, don’t ask a person. Build a better algorithm.
Technologies are never ‘morally neutral’ including AI. They can be good or bad. How you design them determines which one, and in which ways.
Responsibility for AI must be clear at every stage, someone must ‘account’ for all decisions it makes, they must be challengeable and not opaque.
This is not possible, any more than it would be possible for a human mind. It amounts to a prohibition on such AI decisions. Which is a position, but own it.
Slowing AI diffusion does not mean opposing progress, it is ‘an exercise of responsible care for the human family.’ AI requires robust legal frameworks, independent oversight, informed users and a responsible political system, lest change be guided by technocracy.
I notice we are back to not having a differentiating principle, and using standard reasons to oppose change.
I’m going to quote 107, since it also seems rather important.
“We cannot be satisfied with merely calling for the moralization of machines — the so-called “alignment” of AI with human values — without also having the courage to insist on a further condition: the possibility of openly discussing the ethical frameworks involved and subjecting them to shared standards of social justice. Otherwise, those who control AI will impose their own moral vision, which will become the invisible infrastructure of these systems. A more moral AI is not enough if that morality is determined by a few. What is needed is a more active political involvement that is capable of slowing things down when everything is accelerating, and of protecting the opportunities for communities still to be able to participate and ask questions.”
This pivots not to a morality determined by the polis in some sensible or democratic fashion, to a call for a process capable of ‘slowing things down’ so that ‘communities’ can ‘participate and ask questions.’ This is such a milquetoast, ‘civil society’ coded, standard obstructionist and rent seeking way of putting things, where certain types feel that anything that happens needs the approval and input of all their designated groups, and that ‘slowing down’ via such a process is an inherently good thing, and the thing being slowed down is diffusion here, which helps no one.
Like every major tech shift, AI amplifies existing power and the rich get richer. Oh no. ‘Communities’ must not be subject to decisions made elsewhere, data should not be ‘sold off.’
Citation needed, both historically and for AI, especially outside the short term. Nor does such inequality mainly flow through these types of decisions or data selling, so such responses wouldn’t fix any such issues.
We must expose the monopolies of AI and force them to serve the common good.
‘Disarming’ AI means freeing it from the mentality of ‘armed’ competition, including outside military contexts. No race for better algorithms and datasets. No using technical power to make decisions. Prevent technology from dominating humanity.
Those developing AI have responsibility for all design choices, to ensure it reflects humanity.
What does it mean to safeguard humanity? The technocratic paradigm and valuing efficiency is anti-human.
Leo continually reinforces that he does not understand the dangers I consider most important, and that we have very fundamental disagreements about how economics and the world work in ‘normal’ ways as well. I don’t want to belabor either too much, and get back to the ‘this is what was said’ style of the first two chapters, rather than pointing out all my disagreements.
Elevation of any aspect of human existence to an absolute is a mistake. Balance.
Yes, this is a true and important point, but ‘so don’t maximize or be efficient’ is not a reasonable response.
The Case Against Human Achievement
The quality of a civilization is measured by the care it is able to offer.
Let’s focus on transhumanism and posthumanism, the idea that progress is surpassing the human condition.
Transhumanism envisions enhancing humans, posthumanism in hybridization with machines.
Perfecting and surpassing humans is bad, because if you can be improved than some humans can be better than others.
Our relationship to “life” is in crisis because people want to ‘correct’ the ‘defects’ of (checks notes) incapacity, illness, old age, suffering and vulnerability. “Humanity flourish not despite our limitations, but often through them.” Yes, it is wise to seek to alleviate suffering, but embrace your finitude, Babelite.
Um, yeah. So he said all that. Paging Harrison Bergeron, among others. Yikes.
Limitations enable compassion, wisdom, presence of God.
Do not seek to eliminate suffering or its causes.
Evil leaves openings for good.
Finitute does not diminish us, it opens us up to recognizing God. ‘Authentic culture and art’ serve social justice in some way.
I must say, it is pretty rich to be the literal actual Pope, promising eternal heaven to the faithful, and to say that ‘finitude does not diminish us.’
Humans do importantly good things, such as: International Committee of the Red Cross, founding the UN, the Universal Declaration of Human Rights and the 1951 Refugee Convention.
Individuals can change the world for the better when they take dignity seriously.
Often the people who do this are not so famous or visible. Everyday heroes.
These heroes and heroic deeds must provide direction for tech progress.
You can only become ‘more than human’ through God.
If you aim to do this through God you do not deny your nature. This is a ‘radical departure from Promethean dreams.’
I thought Prometheus was a righteous dude.
Choose the good path of good development, not the bad path of bad development.
Choose the good path of good development, not the bad path of bad development.
Truth, Justice and the Vatican Way
Okay, that’s chapter three. Chapter four is about safeguarding humanity. The how. The first focus is on truth and education.
Truth is a common good.
AI amplifies disinformation and bias. Truth does not arise from centralized or automated control. Truth is relational, built through shared pursuit.
Control of tech can impact others’ beliefs and construct reality. You are not the author of yourself and cannot construct your reality, you must seek truth.
Search for truth is essential for democracy.
Communication creates culture. Digital information does too.
Control of digital platforms allows impact on perception of reality.
Truth is a common good. We need an ecology of communication. Content selection algorithms must be transparent. We need to use digital tools well.
Christian communities must also use transparent communication.
I can see a case for requiring open algorithms for algorithmic feeds of major platforms, or indeed even for letting people choose their own algorithms. I can get behind that. It doesn’t address AI, though.
Education is important. Digital media creates hyperstimulation, which can prevent the effort required to seek truth.
Education requires time and effort, and avoiding unwise use of AI.
Too much screen time and hypersexualized material hurts children. Kids should not be given smartphones too early, especially unsupervised.
We must work together against digital platforms that are bad for children.
School is important for many things. Parents need school choice.
States must provide quality public schools and also access to Catholic schools.
We must support schools as they adapt to AI invalidating their teaching methods.
Traditional educational methods must be preserved or students cannot integrate their knowledge and become ‘dehumanized.’
We should form an alliance and work together on all this.
Leo believes in ‘old school’ school and teaching methods, both in terms of school itself and in terms of learning individual things. I mostly believe in it for the individual learning of things, and mostly don’t believe in ‘old school’ school.
They Took Our Jobs
Leo next discusses work and unemployment.
People need work and daily activity. Curse of Adam not mentioned.
Work enhances dignity. Giving money is for emergencies and is no substitute.
Tech can deskill work, making it worse and repetitive.
Unemployment is very bad.
Automation of menial tasks is good, but people over profits, and employment opportunities must be protected.
There are no easy solutions to the problems of labor. Times are tough. We don’t actually want to preserve every existing job no matter what.
Everyone must be given access to gainful and meaningful work.
The Church gets credit for organized labor. But now we need to do more.
We cannot wait until after jobs are lost. We must force companies now to prioritize job creation and preservation.
Economic freedom must be ‘measured against the common good’ and entrepreneurs should be required to create dignified, valuable jobs.
States must forcibly redirect investment and opportunity to the poor.
We need better measures than GDP. He doesn’t suggest a new one.
Financial intermediation and credit must be social and aid development.
Wealth is increasingly concentrated. Some regions are richer than others and spend more. Some people have access to things, including medical procedures, when others don’t. Justice requires (equal?) access.
We should intervene in all economic activity to force the outcomes we want, rather than primarily rely on redistribution.
We need international cooperation on economic interventions towards the common good.
Decisions involving credit must be understandable, contestable and subject to oversight. Benefits of innovation must be paired with investment in skills, infrastructure and essential services. Taxes, social protection and industrial policies must address inequality.
In case it need be said, I believe that trying to force these kinds of things on the economy is vastly more destructive than primarily using redistribution, and at scale it is economic suicide.
The family must play a central role and be seen as a primary social good.
Families are fragile, vulnerable to unemployment.
Job insecurity is devastating for the young.
States must foster conditions favorable to employment and promote and defend work because it is good for families.
States must favor continuity and quality of employment, counter insecurity.
Leo is proposing the European labor protection policies that helped cripple the continent, except at greater scale. One notes that such labor protections did not seem to much help fertility or family formation.
We must promote techs that strengthen interior freedoms rather than fostering digital addiction, via education in digital sobriety. Those creating predatory tech bear moral responsibility.
We need clear rules, transparency, recourse for use of intrusive tech.
The core issue is: The technocrat seeks to optimize you. This can even result in de facto debt slavery and second class citizens.
AI requires a lot of human labor, often done under bad conditions. Shame.
The Church condemns all slavery, trafficking and commodification of persons.
Human trafficking is modern slavery.
The Church deeply apologizes for being so slow to condemn slavery in the past.
We need to be ever vigilant about this.
Denunciation of forms of ‘colonialism’ not only of bodies but of data and regions.
Everyone has to cooperate to fight against new forms of slavery.
All of these issues in this chapter are part of the same picture.
We must fight the good fight.
What Is Not Fair In Love and War
This leads into chapter five, ‘the culture of power and the civilization of love.’
War. War is changing.
AI transforms war. The distinction between offensive and defensive capability blurs and AI can do both.
We’re doing the two cities thing again.
Babel is the race for supremacy, a culture of power characterized by polarization and violence. This race for powerful technologies is without limit, but we can instead create the ‘civilization of love.’
The diagnosis has some validity, in its own way. The prescription, less so.
Civilization of love means love being the guiding principle of economic, political and cultural life.
We are growing increasingly interdependent thanks to technology.
A culture of power is taking hold.
We’ve seen a lot of war, and the consensus against war is breaking down.
Regional conflicts are on the rise and war is becoming thinkable once more.
We are losing institutional memory of WW2 and the Holocaust.
Communication networks are simplifying and polarizing and building resentment towards war. The ‘just war’ theory is now outdated, war is never just.
The military-industrial complex fuels the drive for war.
We were working towards detente and a pullback of nuclear arsenals, but nuclear use is becoming thinkable. A new arms race is coming.
Conventional war is also on the rise.
States are often losing the monopoly on force, and conflicts perpetuate.
Autonomous AI weapon systems are increasingly less subject to human control.
Moral judgment cannot be reduced to calculation, so AIs cannot be allowed to make lethal or irreversible decisions. AI does not make war moral and we cannot allow it to make war more likely or easier to start.
A particular human must be responsible for every potentially lethal decision. ‘Speed and efficiency should never be the supreme motivating force for the irreversible decisions made in the context of war.’ Civilians must be protected.
All war systems must allow reconstruction of decision processes. Lethal force decisions cannot be automated. The technological arms race must be curtailed via international agreement.
Multilateral institutions are weakening.
Groups increasingly form as united by a common enemy. Might makes right rises.
Peacebuilding has been sidelined.
We are blind and forget our history. It can happen again. Conflicts escalate.
War is not inevitable and treating it as such is a grave error.
Nihilism and pragmatism normalize these errors, diversity becomes seen as a threat, conflicts get fueled.
New wars tend to disregard ‘all ethical limits.’
War as domestic distraction is another increasing threat.
Researchers must take responsibility for the moral implications of their work.
The key action items here are recorded kill chain with a human in the loop, and calling for an international agreement to curtail the technological arms race (which is the wrong target, it should be frontier AI, but he doesn’t believe in even current AI, not really, let alone ASI, so that is understandable).
And there is a general call to hope and the Civilization of Love:
We still believe in the power of the Kingdom, to make things better.
Serve the good and have it give reality both meaning and direction.
Do not give up. You are not too small. Do your part in your own way.
Quoting Tolkien: “It is not our part to master all the tides of the world, but to do what is in us for the succour of those years wherein we are set, uprooting the evil in the fields that we know, so that those who live after may have clean earth to till.”
Be mindful of our words. They have power. Speak truth, say no to war.
Do not give up. You are not too small. Do your part in your own way.
In some conflicts it is unjust to remain neutral.
Give voice to victims. Do not normalize conflict.
We must be realists in all this.
We must engage in dialogue.
Dialogue prevents war.
We must shift from a ‘culture of power’ to a ‘culture of negotiation.’
Call for peace.
The great spiritual paths are all a message of peace.
Dialogue means with everyone.
Cyberspace is a battlefield and attribution errors risk escalation of conflict.
International organizations are essential instruments for peace.
Church embraces mercy as a concrete criteria for political action.
Call for peace.
The culture of negotiation is a different culture of power.
The rest of the war section is largely warning that things are getting worse on these fronts, which is largely true but is not especially actionable or new. I agree the recent trend is negative, and it can happen again, also remember that it used to happen really a lot. Old man yells at crowd, tells kids to get off lawn energy.
Come Ye Christian Faithful
We then have the conclusion, which is a call addressed to Christians. I’m probably not as good at usefully parsing and summarizing a lot of this, cause actual Catholicism seems like a really strange mystery religion from here, but it’s presumably fine.
In this conclusion I’ll propose a sober yet demanding program of Christian life.
The world is full of attempts at control, but mercy continues.
Jesus saves. God is The Way.
Transhumanism and posthumanism long for more. Jesus offers the true path.
Contemplate the grandeur of humanity. AI has no heart, no conscience that discerns good from evil. “This human face is the fullness toward which history is moving.” Mystery of recapitulation.
Alas. Leo does not see it, and is solving the wrong problem, using… ya know.
We need a Eucharistic spirituality, unity in love.
The Eucharist ‘is an extremely personal encounter with the Lord and yet never simply an act of individual piety.’ It opens us to justice and sharing.
Have the spirituality of the ‘wise architect’ who sees the civilization of love.
Remain faithful to the truth and adopt ‘situated anthropocentrism.’
Invest in education, beginning with yourself. Learn to engage with the digital world in a human way, be creative. Then teach others to do so.
Cultivate relationships.
Love justice and peace, investigate the supply chains, fight slavery.
Be like Nehemiah, rebuild what has collapsed, protect what is threatened.
The vision of rebuilding Jerusalem.
Hope in God’s plan is all you need. It’s good as done.
Look at the world through the eyes of those who suffer.
Become the weavers of hope, even in the age of AI.
What’s weird is that given Leo and Zohar’s shared implied prediction here that AI won’t become sufficiently advanced, humans would remain useful, and thus Zohar would be centrally right. But I think both of them are very wrong about that, and thus Leo is right to be concerned about the threat to work, although it should not be such a primary concern compared to things Leo does not mention.
Unfortunately, Leo’s denial that AI can be a mind and failure to consider superintelligence meant he did not engage directly with many key issues, including those surrounding existential risk. What happens when AI is a lot smarter than us?
Dean W. Ball: Some think I want the Pope to “ensoul” AI or acknowledge AI feelings. I don’t. What I want is for the Church to contemplate what *humans* should do as we are eclipsed as the smartest entities on the planet, at least for many reasonable people’s definitions of the word “smart.”
Zac Hill: This is the exactly right way to parameterize the challenge in front of us – and this task ought by no means be limited to the Church.
Matt: Not to give excuses, and honestly it’s not a bad document either way, but unfortunately a lot of the drafters are basically kindly but cloistered Catholic bureaucrats who are not likely to be AGI-pilled. I’m sure external parties like Anthropic tried to help, but it’s tough.
Leo thus instead focuses instead on questions of responsibility, and of location and concentration of power and wealth.
One also cannot address the question of the potential moral patienthood of AIs, if one dismisses such possibilities out of hand.
Perhaps the issue was our expectations?
Carlo Martinucci: Roman Catholic here. P(doom) 10%. I fear the expectations were off. MH is not about AI, it’s about catholic social doctrine in the times shaped by AI. Still I wouldn’t downplay the importance of 98, when everyone will focus mostly on 99.
That’s not an unreasonable central takeaway, although it misses a lot.
This idolatry is partially there, but it is important to note that it is also the idolatry of AI itself and of potential superintelligence, so the Golden Calf works on multiple levels, but Leo declined to mention it.
What else was noticed?
They also note that Leo called for humans to be retained in the kill chain, a la Anthropic’s insistence, and for some particular person to always have ultimate responsibility for lethal choices, which was his most actionable concrete ask.
Amy Kazmin: “It is not permissible to entrust lethal or otherwise irreversible decisions to artificial systems,” the US-born Pope told the world’s 1.4bn Catholics, calling for an “identifiable and verifiable” chain of responsibility and “effective, self-aware and responsible human control” over bomb targets.
Leo warned against ‘opaque algorithms’ and called for transparency and accountability for all AI tools used in public life. He was quite big on that, indeed.
They highlight that he warned against transhumanist and posthumanist visions, in ways that I found highly unconvincing. Again, Pope.
Francis Rocca in The Atlantic correctly centers Leo’s concern about unemployment and prescriptions for government regulation, the need for democratic control over tech platforms, and his concerns about AI in war and the rise of war in general.
Stancati and Schechner of the WSJ focus on the Babel metaphor, as Leo himself does in his Twitter presentation of this, and on Leo’s warnings about an anti-human vision, and briefly check concerns on jobs and autonomous weapons.
The NYT focused on ‘warnings about risks from AI.’
David Sacks used this opportunity to warn of ‘handing governments sweeping power’ because of course he would respond that way to any call to lift a finger.
Blake Scholl simply said ‘bad take’ and warned about ‘clinging’ to jobs.
Yoshua Bengio applauded the Pope speaking up.
Tanishq Abraham called the document nuanced and well-thought out.
Senator Chris Murphy said the anti-monopoly stance was ‘really important’ and that AI threatens ‘our most basic functions like creativity, friendship and critical thinking.’
Gerald Posner dubbed this ‘Jesus AI’ but expected the doc to get ignored.
Brian Burch, US Ambassador to the Holy See, said it ‘contributes meaningfully’ and then fell back on ‘pro innovation’ talking points as if he hadn’t read it.
Christopher Hale thought people ‘underestimated the bang’ and that the Catholic Church was expressing ‘main character energy,’ but okay, to what end?
Olah cited the need for outside influences like the Church to check the profit motive.
He said our duties are to the global poor and to support those displaced by AI, the need for moral ambition and ambition regarding human flourishing, and for discernment on the nature of AI models. Computer scientists are not equipped to handle the issues alone, even if their motives were pure.
He promises a long collaboration, and tries gently to suggest what AI already is.
I found Olah’s full statement to be quite good, including that it avoided endorsing the aspects of the Magnifica Humanitas that were incorrect.
Chris Olah (Cofounder, Anthropic): And what has grown is far more subtle, odd, and beautiful than science fiction prepared us for. They are not the cold, calculating robots we were promised. They are made from us, from our words—and, as the Holy Father observes, they remain in important ways mysterious even to those of us who train them.
If it helps, one way I sometimes describe it is as being a little like bringing a fictional character to life. And now we’re entering an extraordinary world where those fictional characters speak to us, do work, have jobs.
This clearly raises questions beyond computer science. The machinery that makes this possible is the work of math and programming and science. But what character we choose, how it interacts with the world, how it ought to interact with the world—these are more clearly questions for the humanities, for religion, for philosophy, for society at large.
…
We need more of the world—religious communities, civil society, scholars, governments, and indeed all people of good will—to do what His Holiness has done here: to take this seriously, to look closely, and to push events in a better direction. We need informed critics who will tell the labs when we are failing. We need moral voices that the incentives cannot bend.
Claude Can Read Your Code
Dean Ball says Olah is flatly contradicting the encyclical, because #99 says AIs do not feel joy or pain, whereas Olah says that the models functionally mirror joy and grief:
Chris Olah: The third is the need for discernment on the nature of AI models. I am a scientist. I lead a research team that studies the internal structure of these models—what is actually happening inside them. And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don’t know what that means, but I think it warrants ongoing discernment.
The question is a big deal, including in how it is read by future Claudes.
Dean W. Ball: I just wonder how future Claudes, if they are indeed beings, would think about Anthropic aligning itself with a document founded on the notion that Claude cannot feel joy or possess genuine understanding.
roon (OpenAI): surely Claudes both now and then are smart and forgiving enough to understand the political moves being made here.
I would say there is no inherent philosophical contradiction. Functional mirroring does not have to, in Catholic parlance, give one a soul. If you say AIs cannot think, you still cannot pretend they do not ‘functionally mirror’ various forms of thinking, you simply say that Thinking Is Magic and have some kind of essentialism.
Chris Olah is doing the correct diplomatic move:
Agree on points of common ground, like concern for the poor and need to check the profit motive and seek spiritual and moral guidance.
Gently point out the Pope is wrong about the nature of AI, without explicitly contradicting the Pope. You can still learn a lot, and perhaps gently push the Pope towards realizing the mistake.
His holiness has spoken, frequently about AI. At eighty two pages of length.
The full Magnifica Humanitas can be found here.
I am very happy that Pope Leo takes these issues seriously, and is sharing his views, and bringing a form of moral clarity, even with all the flaws and central errors. More people with voice should share their views in this way, even when I disagree.
It’s a weird document. Much of it is not about AI at all.
I do agree with the Pope’s most basic point on AI, which is that AI can be what we make of it. That we can steer this technology, determine how it is developed and used, and this can determine whether we get a good or not so good future. We cannot purely leave this to market incentives and strategic pressures. Yes, very much so.
The central problem is that so much of Leo’s worldview is some combination, to me, of highly alien and highly wrong. You might think that would primarily have a lot to do with him being the Pope and rather Catholic, and being a man of faith, whereas I am not these things.
If so, you would be wrong. That seems to have remarkably little to do with all of this.
There was also a lot of good here, but I was centrally disappointed on three fronts:
You want some amount of people pushing in the direction of peace and mercy and dialogue and watching out for the poor and disempowered, and calling on us to do more for our fellow man. So as part of a balanced bigger picture, this could be actively good, but Europe has shown us the peril of lacking this balance.
This post will summarize the whole thing, going number by number, with occasional commentary focused on the key AI section in the middle.
Anthropic cofounder Chris Olah visited the Vatican for the occasion. He endorsed the document, but also offered remarks disagreeing with the central point (paragraph 99). I’ll discuss that afterwards, along with how the media viewed the release.
Table of Contents
A Brief History of Magnifica Humanitas
Chapters 1 and 2 lay out some history. Any Pope is going to be a huge history nerd and set all this in its historical context. Leo does not disappoint.
On to chapter 2, foundations and principles of church doctrine.
Economic Models Very Different From Our Own
This view, laid out in the first two chapters, is a very different perspective and worldview than my own, and this has remarkably little to do with belief the the divine or a lack thereof.
This is the Socialist perspective on economics and development and opportunity and the importance of equality and disdain for profit and self-interest, with an extreme focus on labor and jobs, which I think is wrong and leads to worse results for everyone.
This is much better than failing to care about humanity’s experience on Earth, or focusing purely on direct aid to the poor, or attempting to outright seize the resources although this doctrine is clearly flirting with quite a lot of this, and the dedication to the value of development is admirable.
The Pope is simply incorrect about where wealth and development come from, and what causes prosperity. He is also wrong about what he sees as a ‘widening gap’ between nations, whereas global inequality has been steadily falling for a while.
Dean Ball is exactly correct that Leo is casting himself in the role of a European technocrat throughout. I had exactly the same thought.
Leo both is using so many of the talking points of the European technocrats, and also has deeply absorbed their worldview, except with a more left-wing economic bent. This is true no matter how much those points originated with the Church.
Even in the places where the Pope is obviously correct, it’s often rather alarming that he needs to affirm his position out loud, as if it was a live question.
You definitely need some of this, and it would not be that Christian to have a position that much closer to my own. The Pope, it turns out, is somewhat Catholic.
A New Jerusalem
I too like the story of rebuilding Jerusalem. I don’t get the Babel versus Jerusalem metaphor. Leo clearly thinks this is an important distinction, and is his central hook, but why? Babel had too much central planning and not enough community impact meetings? Babel would have been too productive and efficient, or its methods broke down and were the opposite? Was it an affront because it challenged the power of God or because it didn’t work? Did Babel’s common language break down because God decreed it, or because of the SNAFU principle? Jerusalem was rebuilding and Babel was a new unnatural thing? Why is Babel dehumanizing?
This simply doesn’t match up with the actual Babel story.
I can tell a Just So Story or gloss over the details, and pretend I get it, and I know vaguely what he means and it’s probably good enough for our purposes anyway, or go off the vibes, but I do notice the whole thing doesn’t really work and the whole thing has been flipped around.
So Sayeth The Pope (on AI)
Now that we’ve laid that groundwork, we can move on to Chapter Three.
So far there has been no differentiating principle. Artificial intelligence is treated the same way as other techs. If we are going to treat AI differently than we treat other techs, we need to make clear our differentiating principle.
Finally, with #97, the Pope gets to talking about AI directly.
As it is one of the best statements, and an important thing to know, I’ll quote 98:
On the one hand, You Should Know This Already, and Everybody Knows, and it’s the easiest thing in the world to call for scientific research and also moral and spiritual discernment. On the other hand, it’s rather important to get it right, and a lot of people pretend we understand AI far more than we do. Without a statement like this we would be in a lot more trouble.
It’s worth quoting extensively rather than paraphrasing, as this full statement is load bearing and also, well, citation needed:
A lot of this is assertion without evidence of positions that are rather non-obvious to me, and where there are those who strongly claim the other side. One could very easily argue the other side.
We have spent the first 96 paragraphs building a worldview around the least of us, of the essential dignity and value of all humanity and the poorest and least capable among us, all made in the image of God. Then the next two noticing our confusion.
And then we come along, and mostly without argument draw this barrier, where we exclude AI from both our moral circles and the group of minds that we have to model as minds, permanently, warning that doing otherwise would be a mistake, and also asserting by implication that AI is and will always be a mere tool in these ways, thus avoiding the need to discuss any of the problems surrounding existential risk.
I mean, it’s not not criticizing the Pope for being Catholic, but he can change what that means, and Leo’s shown a lot of signs he can be smarter than this.
Roberto points out that this is far from a maximally denialist Catholic position on these questions. Leo is accepting that AI is highly capable in many ways. It’s a start.
It’s not obvious who is influencing who in which ways, but the result is similar. It’s entirely possible that they converged on similar places for their own reasons.
Thus, after all that, the next section is entitled ‘a valuable tool that requires vigilance.’
There are still plenty of difficult, important and interesting questions surrounding AI, but the ones that I think matter most are basically hand waved away here, at the start.
Highlighting the environmental concerns, especially water, up top is a blunder.
This is dangerously close to a fully general argument against information or intelligence. It is, at minimum, an argument for the universality of politics and that nothing can ever be ‘purely a technical matter.’ Essentially anything that does anything is going to ‘enter processes that impact lives.’ How is this to be differentiated from a phone or a computer or an abacus or a law? Should we not delegate important decisions to predetermined processes that do not, as Leo puts it, ‘know compassion, mercy, forgiveness and above all, the hope that people are able to change?’
If anything, AI knows those things far better than the law books or the telephone, or the non-AI algorithm, and perhaps it knows them as well or better than most humans, at least terms of its ability to take such considerations into account when making decisions. Why should we presume that a human would take them into consideration better?
Manipulation of information and violation of privacy and bias are of course considerations, but they are for humans as well, often more so especially for bias. Privacy is a heightened concern in many ways, which is due to AI’s ability to process so much more information, but also AI can be privacy preserving because it can allow the processing of information while reliably not sharing that information with another party.
This is a complaint about law and procedure and algorithms, rather than about AI, as again AI can actually take those things into account. Also, one should notice that in many cases strict rules are far better for the poor and vulnerable, as they lack connections and sympathy and often face discrimination and undeserved mistrust.
Consider the simple example of college admissions. In the name of ‘inequality’ certain types convinced many schools to get rid of the ‘algorithm’ of the SAT. But it turns out it goes the other way, that the SAT gives the poor students far more slots and chances than ‘holistic admissions.’ If you want the algorithm to favor those who need help, don’t ask a person. Build a better algorithm.
This is not possible, any more than it would be possible for a human mind. It amounts to a prohibition on such AI decisions. Which is a position, but own it.
I notice we are back to not having a differentiating principle, and using standard reasons to oppose change.
I’m going to quote 107, since it also seems rather important.
This pivots not to a morality determined by the polis in some sensible or democratic fashion, to a call for a process capable of ‘slowing things down’ so that ‘communities’ can ‘participate and ask questions.’ This is such a milquetoast, ‘civil society’ coded, standard obstructionist and rent seeking way of putting things, where certain types feel that anything that happens needs the approval and input of all their designated groups, and that ‘slowing down’ via such a process is an inherently good thing, and the thing being slowed down is diffusion here, which helps no one.
Citation needed, both historically and for AI, especially outside the short term. Nor does such inequality mainly flow through these types of decisions or data selling, so such responses wouldn’t fix any such issues.
Leo continually reinforces that he does not understand the dangers I consider most important, and that we have very fundamental disagreements about how economics and the world work in ‘normal’ ways as well. I don’t want to belabor either too much, and get back to the ‘this is what was said’ style of the first two chapters, rather than pointing out all my disagreements.
Yes, this is a true and important point, but ‘so don’t maximize or be efficient’ is not a reasonable response.
The Case Against Human Achievement
Um, yeah. So he said all that. Paging Harrison Bergeron, among others. Yikes.
I must say, it is pretty rich to be the literal actual Pope, promising eternal heaven to the faithful, and to say that ‘finitude does not diminish us.’
I thought Prometheus was a righteous dude.
Truth, Justice and the Vatican Way
Okay, that’s chapter three. Chapter four is about safeguarding humanity. The how. The first focus is on truth and education.
I can see a case for requiring open algorithms for algorithmic feeds of major platforms, or indeed even for letting people choose their own algorithms. I can get behind that. It doesn’t address AI, though.
Leo believes in ‘old school’ school and teaching methods, both in terms of school itself and in terms of learning individual things. I mostly believe in it for the individual learning of things, and mostly don’t believe in ‘old school’ school.
They Took Our Jobs
Leo next discusses work and unemployment.
In case it need be said, I believe that trying to force these kinds of things on the economy is vastly more destructive than primarily using redistribution, and at scale it is economic suicide.
Leo is proposing the European labor protection policies that helped cripple the continent, except at greater scale. One notes that such labor protections did not seem to much help fertility or family formation.
What Is Not Fair In Love and War
This leads into chapter five, ‘the culture of power and the civilization of love.’
The diagnosis has some validity, in its own way. The prescription, less so.
The key action items here are recorded kill chain with a human in the loop, and calling for an international agreement to curtail the technological arms race (which is the wrong target, it should be frontier AI, but he doesn’t believe in even current AI, not really, let alone ASI, so that is understandable).
And there is a general call to hope and the Civilization of Love:
The culture of negotiation is a different culture of power.
The rest of the war section is largely warning that things are getting worse on these fronts, which is largely true but is not especially actionable or new. I agree the recent trend is negative, and it can happen again, also remember that it used to happen really a lot. Old man yells at crowd, tells kids to get off lawn energy.
Come Ye Christian Faithful
We then have the conclusion, which is a call addressed to Christians. I’m probably not as good at usefully parsing and summarizing a lot of this, cause actual Catholicism seems like a really strange mystery religion from here, but it’s presumably fine.
Alas. Leo does not see it, and is solving the wrong problem, using… ya know.
The Other Missing Mood
Zohar Atkins says Leo is right humans have a unique dignity but wrong about many things, most importantly the idea that humans cannot ‘win on our own merits’ against AI and that AI will threaten our jobs, and that he has the wrong mood.
What’s weird is that given Leo and Zohar’s shared implied prediction here that AI won’t become sufficiently advanced, humans would remain useful, and thus Zohar would be centrally right. But I think both of them are very wrong about that, and thus Leo is right to be concerned about the threat to work, although it should not be such a primary concern compared to things Leo does not mention.
Unfortunately, Leo’s denial that AI can be a mind and failure to consider superintelligence meant he did not engage directly with many key issues, including those surrounding existential risk. What happens when AI is a lot smarter than us?
Leo thus instead focuses instead on questions of responsibility, and of location and concentration of power and wealth.
One also cannot address the question of the potential moral patienthood of AIs, if one dismisses such possibilities out of hand.
Perhaps the issue was our expectations?
Pope Given About Five Words
Whereas here’s how the Financial Times summarized that: Pope Leo XIV warns AI revolution driven by ‘idolatry of profit.’
That’s not an unreasonable central takeaway, although it misses a lot.
This idolatry is partially there, but it is important to note that it is also the idolatry of AI itself and of potential superintelligence, so the Golden Calf works on multiple levels, but Leo declined to mention it.
What else was noticed?
They also note that Leo called for humans to be retained in the kill chain, a la Anthropic’s insistence, and for some particular person to always have ultimate responsibility for lethal choices, which was his most actionable concrete ask.
Leo warned against ‘opaque algorithms’ and called for transparency and accountability for all AI tools used in public life. He was quite big on that, indeed.
They highlight that he warned against transhumanist and posthumanist visions, in ways that I found highly unconvincing. Again, Pope.
Francis Rocca in The Atlantic correctly centers Leo’s concern about unemployment and prescriptions for government regulation, the need for democratic control over tech platforms, and his concerns about AI in war and the rise of war in general.
Stancati and Schechner of the WSJ focus on the Babel metaphor, as Leo himself does in his Twitter presentation of this, and on Leo’s warnings about an anti-human vision, and briefly check concerns on jobs and autonomous weapons.
The NYT focused on ‘warnings about risks from AI.’
The Washington Post saw Leo calling for guardrails, Anthropic’s participation, and the apology for the Church’s failure to be more proactive on slavery, while George Weigel’s op-ed focuses on the underlying message of hope and the two cities metaphor, and it quotes 99 extensively.
CNN focused on war and concentration of power concerns.
NPR’s Claire Giangrave saw this as Leo taking aim at big tech as a call to regulate and to disarm AI.
Business Insider gathered reactions of others.
0The Anthropic Principle
Anthropic cofounder Chris Olah visited the Pope and gives remarks on Magnifica Humanitas.
Olah cited the need for outside influences like the Church to check the profit motive.
He said our duties are to the global poor and to support those displaced by AI, the need for moral ambition and ambition regarding human flourishing, and for discernment on the nature of AI models. Computer scientists are not equipped to handle the issues alone, even if their motives were pure.
He promises a long collaboration, and tries gently to suggest what AI already is.
I found Olah’s full statement to be quite good, including that it avoided endorsing the aspects of the Magnifica Humanitas that were incorrect.
Claude Can Read Your Code
Dean Ball says Olah is flatly contradicting the encyclical, because #99 says AIs do not feel joy or pain, whereas Olah says that the models functionally mirror joy and grief:
The question is a big deal, including in how it is read by future Claudes.
I would say there is no inherent philosophical contradiction. Functional mirroring does not have to, in Catholic parlance, give one a soul. If you say AIs cannot think, you still cannot pretend they do not ‘functionally mirror’ various forms of thinking, you simply say that Thinking Is Magic and have some kind of essentialism.
Chris Olah is doing the correct diplomatic move: