Brief notes on the Wikipedia game
7gjm
3Olli Järviniemi
5gjm
6Olli Järviniemi
3Morpheus
3Olli Järviniemi
2gjm
1Olli Järviniemi
2whestler
New Comment
I tried them all. My notes are full of spoilers.
Industrial Revolution (_2):
I didn't catch the deliberately-introduced error; instead I thought the bit about a recession in the 1830s was likely false. I took "the Industrial Revolution" to include the "Second Industrial Revolution" (as, indeed, the article seems to) and it seems to me that electricity was important for the progress of that, so I'm not entirely convinced that what was introduced to the article was exactly a falsehood.
Price gouging:
It seemed utterly unbelievable to me that a gathering of economists would disapprove wholeheartedly of "price gouging" and, lo, I was correct. (I feel like this one was pretty easy.)
World economy:
A few things seemed dubious and nothing seemed obviously wrong. There's a statement that economists usually use PPP to translate currencies and not official exchange rates, but then a table that (if I understand it right) uses both of those. (But this is kinda reconcilable: usually PPP is used for comparisons, and it's much better, but no harm in including nominal figures as well.) There's a claim that it's usual to consider only "human economic activity" and I don't know what that means -- what's the alternative? E.g., if it doesn't include making things with automatic machines then this is obviously false; if it's contrasting with aliens or something then it's so obviously true as not to be worth saying; probably it means something else but I don't know what, and looking it up feels a bit cheaty. This feels odd but my epistemic state is more "I don't understand something here" than "I know what this means and it's wrong". (Oh, I guess maybe it means "human economic activity as opposed to other human activities not classified as economic, like parents taking care of their own children", in which case it's not content-free and is probably correct.) There's a claim that before the Industrial Revolution "global output was dominated by China and India" and that India was the largest economy circa 1CE which seems iffy -- bigger than the Roman Empire? -- but, I dunno, those both had large populations and China at least was pretty advanced in some ways. There's a claim that the African Union has an economy bigger than $2T/year; I'm not sure just who's in the African Union but most of Africa is pretty poor and I'd have thought that if e.g. Nigeria and/or South Africa is at ~$1T then there ought to be a bigger list of $2T individual countries. The first of these feels like a nitpick; for the second, it feels like maybe I just don't understand a technical term; the third and fourth feel like memorization more than understanding. The thing that seems most likely wrong is the claim about China and India, though I wouldn't be astonished to find it's completely true. ... The actual fabrication turns out to be none of those things. I did wonder about errors introduced in the table that was falsified, but didn't consider the thing that was actually done. It's a fair cop.
Cell (biology):
Surely 1-5nm is waaaay too small for even a small prokaryotic cell. The 100nm given as biggest typical size for a eukaryotic cell feels way too small too. Are these really microns rather than nanometres, perhaps? Of course some eukaryotic cells get much larger than 100um -- consider a hen's egg, for instance. But 100um feels like a plausible "seldom much bigger than this" figure. (Yup, this is spot on, and indeed these were originally microns.)
Fundamental theorems of welfare economics:
This is not a field I'm aware of knowing anything about, but the two theorems sound familiar (though I hadn't associated them with the specific title here). The only thing that seems obviously fishy is that the article says the theorems don't ensure existence of the equilibria, but also says that the second theorem guarantees that "any Pareto optimum can be supported as a competitive equilibrium for some initial set of endowments", which sure sounds like ensuring the existence of equilibria to me. There are a couple of other things that look strange but I think they're more likely just badly-written-article strange: "The production economy is more general and entails additional assumptions" (surely to be more general is to require fewer assumptions, but maybe what's meant is that in the broader setting you need extra assumptions that automatically hold in the narrower setting or something), and "The assumptions are all based on the standard graduate microeconomics textbook" which is just a really weird sentence. There are lots of technicalities wherein errors could be hiding, but I'm no economist and wouldn't notice. So, my guess is that it's the "don't ensure existence" thing. -- Nope, all those are apparently correct, and actually there are some further technical conditions that I didn't know about. This seems very much in the "memorization" rather than "understanding" category to me; e.g., for all I know "perfect information" is implied by "rationality" as those terms are used in economics. Maybe given infinite intelligence I could have figured out that there needed to be some sort of condition about externalities, but: bah.
List of causes of death by rate:
This says that neurological disorders are the biggest single cause of death. Ahead of both cardiovascular and respiratory problems? Really? Can't possibly be right. And, indeed, it isn't. On the one hand, this is "memorization"; on the other, surely this is pretty well known.
Natural selection:
Surely that description of "microevolution" and "macroevolution" can't be right; the distinction is about the scale of the changes, not the nature of the environmental pressures that cause them. ... Huh, apparently that isn't the issue; the actual error is one I should have spotted if I'd been reading more carefully and hadn't seen something else I was sure was wrong. I have to say I still think those sentences about microevolution and macroevolution are wrong. Aren't they? (I tend to avoid those terms, which I mostly associate with creationists who argue that one is real and the other not, so maybe I'm misunderstanding them, but e.g. the Wikipedia pages for them don't seem to match what's claimed in this article.)
Working time:
Surely the decreases in working hours shown in that table are too dramatic. Unless, I guess, they're the result of e.g. increasing numbers of part-time workers, rather than changes to typical full-time hours or something? (If so, then right next to the table there should be a load of caveats about this.) But this claims that, for instance, in France in 1950 the average worker worked nearly 3000 hours per year, versus about half that now. And can it really be true that Greece has longer-working workers than any other EU country? It's not that any of the individual figures is completely unbelievable, but in the aggregate something seems really off. I'm sure there has been a decrease in working hours, but it surely can't be this big. (I may have been encouraged toward this conclusion by noticing that the directory containing this example had what looked like data-processing code, suggesting that something quantitative has been changed in a nontrivial way.) Anyway, cheaty or not, I was spot on with this.
So, I got exactly half of them right (and alternated between right and wrong!). For two of the wrong ones I'm prepared to argue that the thing I thought I'd spotted is wrong even though it wasn't deliberately introduced as a test; for another, I'm maybe prepared to argue that the deliberate error introduced isn't exactly an error.
Thanks for writing this, it was interesting to read a participant's thoughts!
Responses, spoilered:
Industrial revolution: I think if you re-read the article and look at all the modifications made, you will agree that there definitely are false claims. (The original answer sheet referred to a version that had fewer modifications than the final edited article; I have fixed this.)
Price gouging: I do think this is pretty clear if one understand economics, but indeed, the public has very different views from economists here, so I thought it makes for a good change. (This wasn't obvious to all of my friends, at least.)
World economy: I received some (fair) complaints about there being too much stuff in the article. In any case, it might be good for one to seriously think about what the world economy looks like.
Cell: Yep, one of the easier ones I'd say.
Fundamental theorems of welfare economics: Yeah, I don't think modification was successful. (But let me defend myself: I wanted to try some "lies of omission", and something where an omission is unarguably wrong is in the context of mathematical theorems with missing assumptions. Well, I thought for math it's difficult to find an example that is not too easy nor too hard, and decided to go for economics instead. And asymmetric information and externalities really are of practical importance.)
List of causes by death rate: Leans towards memorization, yes. I'd guess it's not obvious to everyone, though, and I think one can do non-zero inference here.
Natural selection: (I think this is one of the weaker modifications.)
Working time: Deleted the spoiler files, thanks.
Let's suppose it's true, as Olli seems to find, that most not-inconsequential things in Wikipedia are more "brute facts" than things one could reasonably deduce from other things. Does this tell us anything interesting about the world?
For instance: maybe it suggests that reasoning is less important than we might think, that in practice most things we care about we have to remember rather than working out. It certainly seems plausible that that's true, though "reasonining is less important than we might think" feels like a slightly tendentious way of putting it. (I suggest: Reasoning is very important on those occasions when you actually need to do it, but those occasions are rarer than those of us who are good at reasoning might like to think.)
I think there's more value to just remembering/knowing a lot of things than I have previously thought. One example is that one way LLMs are useful is by aggregating a lot of knowledge from basically anything even remotely common or popular. (At the same time this shifts the balance towards outsourcing, but that's beside the point.)
I still wouldn't update much on this. Wikipedia articles, and especially the articles you want to use for this exercise, are largely about established knowledge. But of course there are a lot of questions whose answers are not commonly agreed upon, or which we really don't have good answers to, and which we really want answers to. Think of e.g. basically all of the research humanity is doing.
The eleventh virtue is scholarship, but don't forget about the others.
1I used Alex Turners entire shortform for my prompt as context for gpt-4 which worked well enough to make the task difficult for me but maybe I just suck at this task.
1Thanks for the link, I wasn't aware of this.
I find your example to be better than my median modification, so that's great. My gut reaction was that the example statements are too isolated facts, but on reflection I think they are actually decent. Developmental psychology is not a bad article choice for the exercise.
(I also find the examples hard, so it's not just you. I also felt like I on average underestimated the difficulty of spotting the modifications I had made, in that my friends were less accurate than I unconsciously expected. Textbook example of hindsight bias.)
Ultimately, though, I would like this exercise to go beyond standard calibration training ("here's a binary statement, assign a probability from 0% to 100%"), since there are so many tools for that already and the exercise has potential for so much more. I'm just not yet sure how to unleash that potential.
There appear to be two edited versions of the Industrial Revolution article. Which one is recommended? (My guess: the _2 one because it's more recent.)
Thanks for spotting this; yes, _2 was the correct one. I removed the old one and renamed the new one.
This was interesting. I tried the Industrial Revolution one.
I initially thought it was strange that the textile industry was first (my history is patchy at best). I remembered that industrial looms were an important invention but it seemed to me that something earlier in the production chain should be bigger, like coal extraction or rail, steam engines, or agriculture. I noticed that electricity was not so significant until after the industrial revolution. I think my error sensors were over active though - I flagged a lot of stuff as false and only some of it was. Here's the summary:
Correctly spotted: Electricity too early, Telephone too early, lack of much mention of steam power + coal.
Incorrectly spotted: Textile industry seemed too dominant, Did not know about/expect there would be a recession in the 1830s
1Curated and popular this week