We should not expect the Vatican to want or need to use AI to write documents meant for wide public consumption. Yes, the liturgical writing style they tend to use can be awkward, but reading and writing such documents is kinda what Catholic priests do. Perhaps it's hard to believe that unassisted humans — in this age where people's attention spans have been fried by phones and social media — can write something 2% as long as Zvi's AI newsletter series, but an organization with the resources of the Catholic Church actually can. Also, the encyclical was presumably written and reviewed by several people working collaboratively; it's not like it was just one guy who can secretly use ChatGPT on his own.
I just don't see why you think we should be so confident about this. Indeed, the encyclical was presumably worked on collaboratively by multiple people. Lots of people use AI to help them write, it wouldn't be so shocking to think that this includes some Vatican officials.
Re: whether it "reads as AI", my understanding is that the whole reason Linch thought to put this into Pangram was that it read as AI. Most people don't put everything they read into AI! You can also read someone on reddit saying the same thing here.
The rest of your post seems like it's just batting away the evidence. Every time people have tested Pangram rigorously, it's been shown to have a very low false negative [EDIT: I actually meant 'positive'] rate. Furthermore, I think the evidence of old papal encyclicals not being flagged is also relevant here. It's possible that that's just because Pangram trained on those, but most people aren't super interested in encyclicals, and Pangram Labs is a small enough company that I doubt they've trained on the whole internet. I think this would be legit if we had very very strong reasons to think there was no way that anyone at the Vatican would ever use AI in their writing, but I just don't think you've offered those reasons.
1I fed a couple of my very recent blog posts to Pangram, because I do love em-dashes and bulleted lists. And in an unscientific sample, Pangram scored them as 100% human.
This has updated my prior on Pangram: It does not appear to trivially fail on the kind of writing that I do, given a totally inadequate sample size.
I recently took a look at Jabarian & Imas 2025 and was disappointed to find that their human-authored texts came from public pre-LLM-era datasets. So I can't trust the very low false positive rate without reasoning about Pangram's training process. (To be fair, it would be expensive to commission 1,992 human-authored texts just to evaluate Pangram.)
1There are a few other studies: one of them looks at magazine articles published in 2024 as the "human-written set" and finds a ~2% FPR, using an early 2025 / late 2024 version of Pangram. There's another one that uses Italian newspaper articles published in early 2026 (and finds a <0.1% FPR), I'll ask the authors when those articles came out but I would assume late 2025.
Update: apparently it's from this dataset which was published pre-2020, so also not from the post-LLM-era.
1Thanks. (Russell, Karpinska, and Iyyer 2025) have human texts that were published between 2022 and 2024-12-08 (Table 5), and v1 of the paper was released 2025-01-26. So at least some of the evals were performed between 2024-12-08 and 2025-01-26. I think the false positive rate of 2% for Pangram in Table 2 means that Pangram mistakenly classified 3 out of 150 human-authored texts as AI. If we knew more about the distribution of those articles over the time range and made some assumptions about how frequently Pangram retrains, we'd be able to conclude something about the true FPR.
FWIW their human corpus is articles in places including Readers Digest, I wouldn't actually be that shocked if that had some LLM content in late 2024.
Also as a model size/compressibility issue, it's actually really hard to "cheat" by memorizing the entire internet! Claude Opus couldn't do it and Claude Opus's almost certainly bigger and had more training time than Pangram Labs's classifier.
Also, if Pangram were trained on previous encyclicals, you'd think that it would have learned that encyclical-like style is a tell for being human-generated.
So, the previous posts here led to this article in The Verge. Christopher Hale said in response:
I can confirm that this is 100% false. Not only was it written by hand, but the first drafts were literally written on paper with a pen.
He's a popular Catholic writer focused on the Pope, with some Vatican connections.
An author of one of the previous LessWrong posts then responded by accusing Hale of using AI for his past work. But with 33k likes I guess Hale is winning in terms of popular opinion.
Anyway, for a bunch of people this is now how they heard of LessWrong, so they know it as a group of anti-Christianity pro-AI cultists running a smear campaign, lol.
Yep, I could have been less abrasive in that interaction. That said, it just does seem pretty relevant for understanding how credible his denial is to know that he has other undisclosed AI writing!
1I'm confused what this comment is trying to say tbh. Like I think I can try to read it as a series of disconnected observations but I think there's supposed to be an underlying point. But I can't figure out what it is.
Now that things have died down a bit, I want to point out that using this tweet as evidence has (extremely) obvious logical and empirical flaws.
Logically, drafting a document with pen and paper does not preclude later AI additions. Indeed, the story I originally sketched out in the post, with some sections written by the Pope and senior Vatican officials not using AI, and other sections with other senior Vatican offiicals heavily involving AI use, is fully consistent with a first draft written by pen and paper.
Empirically, I don't think this has really been demonstrated.
The Atlantic and Snopes have both reached out to Christopher to substantiate his claims, and no evidence has been given so far.
Still, this has not stopped people from claiming that "an editor" of the encyclical has "confirmed" that I'm wrong, including in attempted community notes.
Ironically, the ppl screaming about how my careful multipronged analysis is "no evidence" consistently fell for obvious lies.
Note that cardinals are probably not the ones doing the actual writing of these things - they're too busy doing cardinal stuff. My guess is that it's staff at the Dicastery for the Doctrine of the Faith and other departments of the Vatican.
I recently saw two posts here saying the Pope's recent encyclical was largely written by AI. Two posts, on the front page at the same time.
Since I've sometimes crossposted blog posts here, I thought I'd write a post to at least make it clear that I have a different position.
priors
We should not expect the Vatican to want or need to use AI to write documents meant for wide public consumption. Yes, the liturgical writing style they tend to use can be awkward, but reading and writing such documents is kinda what Catholic priests do. Perhaps it's hard to believe that unassisted humans — in this age where people's attention spans have been fried by phones and social media — can write something 2% as long as Zvi's AI newsletter series, but an organization with the resources of the Catholic Church actually can. Also, the encyclical was presumably written and reviewed by several people working collaboratively; it's not like it was just one guy who can secretly use ChatGPT on his own.
Now, the US government has been using AI for important documents recently, like tariffs and health stuff and Epstein censorship, but that's a very different situation: the US government has plenty of workers, but the leadership didn't want to involve too many people because they didn't trust existing government employees to agree with their agenda. (Also, Trump's cabinet members are probably dumber than the average Catholic bishop.)
human judgement
Humans familiar with AI writing (and preferably with non-AI writing of the relevant type) are more reliable than AI-based AI detectors. This should be obvious to people with experience about this, but here's a citation I guess.
So, do we see people reading Magnifica Humanitas and saying "hey that's AI"? Not really. I read it, and I don't think it's AI. I don't think other forums are buying the "it's AI" argument either; here's r/ChatGPT being less credulous than LessWrong, and they seem to be familiar with what AI can do these days.
Yes, it has em-dashes, and some "not X but Y". But you can't analyze writing on such a simplistic level when it's a kind of thing that tends to use those. Is the quote below AI writing because it's using em-dashes and "not X but Y"?
Pangram reliability
"Pangram says previous encyclicals are 0% AI!"
Yeah, they're in the training set. Not only is that an obvious thing for Pangram to do, it's the only plausible way they could get 0% consistently. If we wait long enough, maybe Pangram will update their model's training and this new encyclical will magically become 0% AI too!
"this study said Pangram is reliable!"
Any study on AI detector reliability is using specific models, specific writing styles, and specific number thresholds. An encyclical is a different type of writing than such studies have used, so those results tend to not apply well. More generally, LLMs tend to copy some patterns used in formal writing, and most writing is not that. If you're checking some kind of writing that normally never uses em-dashes, you can just count em-dashes and get good accuracy results by such metrics.
a suggestion
If there are people around who paid for Pangram and just want to use it on something, and also have too much free time, I actually have a suggestion for something to do instead: a website that rates AI-ness of everything on the front page of Hacker News, and lets people filter links by AI-ness. A lot of the posts there are pretty obviously AI these days, and they're types of writing that automated AI detection should work better on.
conclusion
Blindly trusting software like Pangram is, uh, one of the things the encyclical was warning people about. So good job making its point, but also, maybe stop that.