This is a special post for quick takes by Marcello. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Marcello's Shortform
70Marcello
2ozziegooen
5DirectedEvolution
4[anonymous]
8Ben Wu
1[anonymous]
2ozziegooen
31Marcello
5Ben Pace
4Dagon
2DirectedEvolution
11Marcello
4Vladimir_Nesov
7Marcello
A Cautionary Tale about Trusting Numbers from Wikipedia
So this morning I woke up early and thought to myself: "You know what I haven't done in a while? Good old fashioned Wikipedia rabbit hole." So I started reading the article on rabbits. Things were relatively sane until I got to the section on rabbits as food.
Wild leporids comprise a small portion of global rabbit-meat consumption. Domesticated descendants of the European rabbit (Oryctolagus cuniculus) that are bred and kept as livestock (a practice called cuniculture) account for the estimated 200 million tons of rabbit meat produced annually.[161] Approximately 1.2 billion rabbits are slaughtered each year for meat worldwide.[162]
Something has gone very wrong here!
200 million tons is 400 billion pounds (add ~10% if they're metric tons, but we can ignore that.)
Divide that by 1.2 billion, and we can deduce that those rabbits weigh in at over 300 pounds each on average! Now I know we've bred some large animals for livestock, but I'm rolling to disbelieve when it comes to three hundred pound bunnies.
Of the two sources Wikipedia cites it looks like [161] is the less reliable looking one. It's a WSJ blog. But the biggest reason we shouldn't be trusting that article is that its numbers aren't even internally consistent!
From the article:
Globally, about some 200 million tons of rabbit meat are produced a year, says Luo Dong, director of the Chinese Rabbit Industry Association. China consumes about 30% of the whole production, he said, with 70% of such meat—or some 420,000 tons a year—going to Sichuan province as well as the neighboring municipality of Chongqing.
There's a basic arithmetic error here! `200 million * 70% * 30%` is 42 *million*, not 420,000.
If we assume this 200 million ton number was wrong and the 420,000 ton number for Sichuan was right, the global number should in fact be 2 million tons. This would make the rabbits weigh three pounds each on average, which is a much more reasonable weight for a rabbit!
If I had to take a guess as to how this mistake happened, putting on my linguist hat, Chinese has a single word for ten thousand, like the Greek-derived "myriad", (spelt either 万 or 萬). If you actually wanted to say 2*10^6 in Chinese, it would end up as something like "two hundred myriad". So I can see a fairly plausible way a translator could mess up and render it as "200 million".
Anyway, I've posted this essay to the talk page and submitted an edit request. We'll see how long it takes Wikipedia to fix this.
Links:
Original article: https://en.wikipedia.org/wiki/Rabbit#As_food_and_clothing
[161] https://web.archive.org/web/20170714001053/https://blogs.wsj.com/chinarealtime/2014/06/1 3/french-rabbit-heads-the-newest-delicacy-in-chinese-cuisine/
9That seems like a good example of a clear math error.
I'm kind of surprised that LLMs aren't catching things like that yet. I'm curious how far along such efforts are - it seems like an obvious thing to target.
By "aren't catching" do you mean "can't" or do you mean "wikipedia company/editors haven't deployed an LLM to crawl wikipedia, read sources and edit the article for errors"?
The 161 is paywall so I can't really test. My guess is Claude wouldn't find the math error off a "proofread this, here's its sources copy/pasted" type prompt but you can try.
My guess is Claude wouldn't find the math error off a "proofread this, here's its sources copy/pasted" type prompt but you can try.
I was curious about this so decided to check.
Both Claude 3.7 and GPT-4o were able to spot this error when I provided them just the Wikipedia page and instructed them to find any mistakes. They also spotted the arithmetic error when asked to proof-read the cited WSJ article. In all cases, their stated reasoning was that 200 million tons of rabbit meat was way too high, on the order of global meat production, so they didn't have to actually do any explicit arithmetic.[1]
Funnily enough, the LLMs found two other mistakes in the Rabbit Wikipedia page: the character Peter Warne was listed as Peter Wayne and doxycycline was misspelt as docycycline. So it does seem like, even without access to sources, current LLMs could do a good job at spotting typos and egregious errors in Wikipedia pages.
(caveat: both models also listed a bunch of other "mistakes" which I didn't check carefully but seemed like LLM hallucinations since the correction contradicted reputable sources)
- ^
GPT-4o stumbles slightly when trying to do the arithmetic on the WSJ article. It compares the article's 420,000 tons with 60 million (200 million x 0.3) rather than the correct calculation of 42 million (200 million x 0.3 x 0.7). However, I gave the same prompt to o1 and it did the maths correctly.
Neat. You can try to ask it for confidence interval and it'll probably correlate against the hallucinations. Another idea is run it against the top 1000 articles and see how accurate they are. I can't really guess back-of-envelope for if it's cost effective to run this over all of wiki per-article.
Also I kind of just want this on reddit and stuff. I'm more concerned about casually ingested fake news than errors in high quality articles when it comes to propaganda/disinfo.
By "aren't catching" do you mean "can't" or do you mean "wikipedia company/editors haven't deployed an LLM to crawl wikipedia, read sources and edit the article for errors"?
Yep.
My guess is that this would take some substantial prompt engineering, and potentially a fair bit of money.
I imagine they'll get to it eventually (as it becomes easier + cheaper), but it might be a while.
"Aspiring Rationalist" Considered Harmful
The "aspiring" in "aspiring rationalist" seems like superfluous humility at best. Calling yourself a "rationalist" never implied perfection in the first place. It's just like how calling yourself a "guitarist" doesn't mean you think you're Jimi Hendrix. I think this analogy is a good one, because rationality is a human art, just like playing the guitar.
I suppose one might object that the word "rational" denotes a perfect standard, unlike playing the guitar. However, we don't hesitate to call someone an "idealist" or a "perfectionist" when they're putting in a serious effort to conform to an ideal or strive towards perfection, so I think this objection is weak. The "-ist" suffix already means that you're a person trying to do the thing, with all the shortcomings that entails.
Furthermore, it appears harmful to add the "aspiring". It creates dilution. Think of what it would mean for a group of people to call themselves "aspiring guitarists". The trouble is, it also applies to the sort of person who daydreams about the adulation of playing for large audiences but never gets around to practicing. However, to honestly call yourself a "guitarist", you would have to actually, y'know, play the guitar once in a while.
While I acknowledge I'm writing this many years too late, please consider dropping the phrase "aspiring rationalist" from your lexicon.
Hm, I like this, I feel resolved against 'aspiring rationalist', which was always losing anyway because it's a longer and less catchy phrase.
I tend not to use "rationalist" for myself - the implication of identity and mix of description and value signaling rubs me the wrong way. For those who are describing actual group membership, part of the "rationalist community", I can see reasons to use "rationalist" and "aspiring rationalist" in different contexts, depending on what you're signaling and to whom.
Outside of community identification, "aspiring rationalist" implies a focus on application of rationality to one's personal life, where just "rationalist" is broader, and may only imply an interest in the topic.
Note: I should acknowledge that I don't think this is terribly important, and my standard advice for naming and jargon discussions remains "if it matters, use more words".
I get the point of view that we should be forthright about our goals, practices, and community affiliations. Nothing wrong with using a label to cultivate a sense of belonging. After all, Christians call themselves after their ideal of perfection, so why shouldn't we?
I think part of the reason is that just about everybody wants to be rational. Not everybody wants to be a guitarist, Christian, perfectionist, or idealist.
Also, most groups have some way of telling whether somebody's "doing the thing" or not. Catholics have the sacrament and you have to call him Jesus, not Frank. Guitarists practice or have chops. Just about everybody tries to think rationally from time to time, even if they fail, so what's the thing that somebody would have to do to not be a rationalist?
Why don't we call ourselves epistemologists. At least it's one syllable shorter than "aspiring rationalist." Plus, it comes with the implication that we're interested in rational thought, not experts at doing it.
Funnily enough, I feel more trepidation about referring to myself as an epistemologist than as a "rationalist." I think it sounds too much like a professional title. But heck, I'm an author even though I've never published a book. I'm a musician even though I don't play professionally. Why can't I be an epistemologist?
In Defense of the Shoggoth Analogy
In reply to: https://twitter.com/OwainEvans_UK/status/1636599127902662658
The explanations in the thread seem to me to be missing the middle or evading the heart of the problem. Zoomed out: an optimization target at level of personality. Zoomed in: a circuit diagram of layers. But those layers with billions of weights are pretty much Turing complete.
Unfortunately, I don't think anyone has much idea how all those little learned computations are make up said personality. My suspicion is there isn't going to be an *easy* way to explain what they're doing. Of course, I'd be relieved to be wrong here!
This matters because the analogy in the thread between averaged faces and LLM outputs is broken in an important way. (Nearly) every picture of a face in the training data has a nose. When you look at the nose of an averaged face, it's based very closely on the noses of all the faces that got averaged. However, despite the size of the training datasets for LLMs, the space of possible queries and topics of conversation is even vaster (it's exponential in the prompt-window size, unlike the query space for the average faces which are just the size of the image).
As such, LLMs are forced to extrapolate hard. So, I'd expect that which particular generalizations they learned, hiding in those weights, to start to matter once users start poking them in unanticipated ways.
In short, if LLMs are like averaged faces, I think they're faces that will readily fall apart into Shoggoths if someone looks at them from an unanticipated or uncommon angle.
Another disanalogy is in how GPT-4 writes novel quines without thinking out loud in the context window. It still needs to plan it, so the planning probably happens with layers updating the residual stream, the way it could've happened with thinking step by step, but using the inscrutable states of the network instead of tokens. Thinking step by step in tokens imitates humans from its training data, but who knows how the thinking step by step in the residual stream works.
Thus shoggoths might be the first to wake up, because models might already be training on this hypothetical alien deliberation in the residual stream, while human-imitating deliberation with generated tokens is still not being plugged back into the model as training data. This hypothesis also predicts future LLMs that are broadly trained the same as modern LLMs, still look non-agentic and situationally unaware like modern LLMs, but start succeeding in discussing advanced mathematics, because the necessary process of studying it (inventing and solving of exercises that are not already in the training set) might happen by alien deliberation within the residual stream during the training process, while SSL looks at episodes that involve related theory.
One of my pet journalism peeves is the "as" (or sometimes "while") construction, which I often see in titles or first sentences of articles. It looks like "<event A was happening> as <event B was happening>". You can fact check the events and it'll turn out they happened, but the phrasing comes with this super annoying nudge-nudge-wink-wink-implication that the two events totally have direct causal connection. Unfortunately, you can't pin this on the journalist because they didn't actually say it.
This sort of thing happens a lot. To give just a couple example templates, articles like "as <political thing happened>, markets rallied" or "<stock> falls as <CEO did something>" are often trying to pull this.