In the post 'Four layers of Intellectual Conversation', Eliezer says that both the writer of an idea, and the person writing a critique of that idea, need to expect to have to publicly defend what they say at least one time. Otherwise they can write something stupid and never lose status because they don't have to respond to the criticism.
I was wondering about where this sort of dialogue happens in academia. I have been told by many people that current journals are quite terrible, but I've also heard a romantic notion that science (especially physics and math) used to be more effectively pursued in the early 20th century (Einstein, Turing, Shannon, etc). So Oliver and I thought we'd look at the journals to see if they had real conversations.
We looked at two data points, and didn't find any.
First, Oliver looked through Einstein's publication history (Oli is German and could read it). Einstein has lots of 'reviews' of others' work in his list of publications, sometimes multiple of the same person, which seemed like a promising example of conversation. Alas, it turned out that Einstein had merely helped German journals write summaries of papers that had been written in English, and there was no real dialogue.
Second, I looked through a volume of the London Mathematical Society, in particular, the volume where Turing published his groundbreaking paper proving that not all mathematical propositions are decidable (thanks to sci-hub for making it possible for me to read the papers!). My eyes looked at about 60% of the pages in the journal (about 12 papers), and not one of them disagreed with any prior work. There was :
- A footnote that thanked an advisor for finding a flaw in a proof
- An addendum page (to the whole volume) that consisted of a single sentence thanking someone for showing one of their theorems was a special case of someone else's theorem
- One person who was skeptical of another person's theorem. But that theorem by Ramanujan (who was famous for stating theorems without proofs), and the whole paper primarily found proofs of his other theorems.
There were lots of discussions of people's work but always building, or extending, or finding a neater way of achieving the same results. Never disagreement, correction, or the finding of errors.
One thing that really confuses me about this is that it's really hard to get all the details right. Lots of great works are filled with tiny flaws (e.g. Donald Knuth reliably has people find errors in his texts). So I'd expect any discussion of old papers to bring up flaws, or that journals would require a section at the end for corrections of the previous volume. There were of course reviewers, but they can't be experts in all the areas.
But more importantly where did/does the dialogue happen if not in the journals?
If I try to be concrete about what I'm curious about:
As people go about the craft of doing science, they will make errors (conceptual mistakes, false proofs, and so on). One of the main pieces of infrastructure in academia are journals, where work gets published and can become common knowledge.
Two places to fix errors are pre-publication and post-publication. I don't know much about the pre-publication process, but if it is strong enough to ensure no errors got published, I'd like some insight into what that process was like. Alternatively, if course-correction happened post-publication, I'm interested to know how and where, because when I looked (see above) I couldn't find it.
There's also the third alternative, that no progress was made. And there's the fourth alternative, that most papers were bad and the thing scientists did was to just never read or build on them. I'm happy to get evidence for any of these, or a fifth alternative.
Added: Maybe this is a more crisp statement:
Why do (old) journals not claim to have errors in any of the papers? Is it because they're (implicitly) lying about the quality of the papers? Or if there's a reliable process that removed errors from 100% of papers, can someone tell me what that process was?
My expectation is that the fourth alternative, or some variation thereof, is the dominant answer. This is less a reflection of the quality of the papers, and more a reflection of the limited bandwidth of scientists for reading them.
This problem has been discussed in the modern context because of the explosion in the number of publications and the administrative responsibilities of scientists (for example, teaching and grant writing). But it is also noticed that deep reading of papers is both time consuming and cognitively intensive; troubling to write up a correction still more so. I argue there is still a fundamental bandwidth limit, and the early 20th century scientists still had to abide by it.
Following on the argument that reading papers deeply enough to correct errors and publish those corrections is difficult, I posit that the 'publish or perish' mechanism is responsible for corrections being published at all. I expect that even though there are errors, if the objective is to produce the best original work possible it is more efficient to correct them for yourself and then go on to use the corrected version for yourself; it could even be argued that leaving the errors publicly uncorrected is advantageous for being first. I also expect that if the objective shifts to total number of publications, it becomes more efficient to publish corrections because writing up a correction is less difficult than producing original work.
If my expectation is correct, then we should see very few corrections published leading up to World War II, and then an increasing number afterward as the professionalization of science progresses.
One good source for this kind of question would be histories of science and/or math. They do a pretty good job of disentangling what scientists thought and when, because they do the difficult work of going through notes, correspondence, and the published work. The downside is it will usually be from the subject's perspective, ie thermodynamics, instead of focusing on academia per se.