In the post 'Four layers of Intellectual Conversation', Eliezer says that both the writer of an idea, and the person writing a critique of that idea, need to expect to have to publicly defend what they say at least one time. Otherwise they can write something stupid and never lose status because they don't have to respond to the criticism.
I was wondering about where this sort of dialogue happens in academia. I have been told by many people that current journals are quite terrible, but I've also heard a romantic notion that science (especially physics and math) used to be more effectively pursued in the early 20th century (Einstein, Turing, Shannon, etc). So Oliver and I thought we'd look at the journals to see if they had real conversations.
We looked at two data points, and didn't find any.
First, Oliver looked through Einstein's publication history (Oli is German and could read it). Einstein has lots of 'reviews' of others' work in his list of publications, sometimes multiple of the same person, which seemed like a promising example of conversation. Alas, it turned out that Einstein had merely helped German journals write summaries of papers that had been written in English, and there was no real dialogue.
Second, I looked through a volume of the London Mathematical Society, in particular, the volume where Turing published his groundbreaking paper proving that not all mathematical propositions are decidable (thanks to sci-hub for making it possible for me to read the papers!). My eyes looked at about 60% of the pages in the journal (about 12 papers), and not one of them disagreed with any prior work. There was :
- A footnote that thanked an advisor for finding a flaw in a proof
- An addendum page (to the whole volume) that consisted of a single sentence thanking someone for showing one of their theorems was a special case of someone else's theorem
- One person who was skeptical of another person's theorem. But that theorem by Ramanujan (who was famous for stating theorems without proofs), and the whole paper primarily found proofs of his other theorems.
There were lots of discussions of people's work but always building, or extending, or finding a neater way of achieving the same results. Never disagreement, correction, or the finding of errors.
One thing that really confuses me about this is that it's really hard to get all the details right. Lots of great works are filled with tiny flaws (e.g. Donald Knuth reliably has people find errors in his texts). So I'd expect any discussion of old papers to bring up flaws, or that journals would require a section at the end for corrections of the previous volume. There were of course reviewers, but they can't be experts in all the areas.
But more importantly where did/does the dialogue happen if not in the journals?
If I try to be concrete about what I'm curious about:
As people go about the craft of doing science, they will make errors (conceptual mistakes, false proofs, and so on). One of the main pieces of infrastructure in academia are journals, where work gets published and can become common knowledge.
Two places to fix errors are pre-publication and post-publication. I don't know much about the pre-publication process, but if it is strong enough to ensure no errors got published, I'd like some insight into what that process was like. Alternatively, if course-correction happened post-publication, I'm interested to know how and where, because when I looked (see above) I couldn't find it.
There's also the third alternative, that no progress was made. And there's the fourth alternative, that most papers were bad and the thing scientists did was to just never read or build on them. I'm happy to get evidence for any of these, or a fifth alternative.
Added: Maybe this is a more crisp statement:
Why do (old) journals not claim to have errors in any of the papers? Is it because they're (implicitly) lying about the quality of the papers? Or if there's a reliable process that removed errors from 100% of papers, can someone tell me what that process was?
Hi
I am new to this website so please excuse me if I diverge from the rules of discourse.
I think there is an inaccurate assumption in this text. Even though scientific ideas become official through publishing in scientific journals it is not the only platform to discuss science. I will suggest here other mechanisms to root out errors and discuss scientific ideas based on my experience as a researcher in Earth Sciences. I know other fields use different methods though, to my understanding, these methods are fundamentally similar.
Since Earth Sciences suffer from chronicle lack of evidence Earth scientists are bound to come up with contradicting ideas. Most of these ideas are rooted out in the initial stage of fitting the model to existing evidence. The rest are then presented not in papers but as presentations in seminars, workshops and conferences. Since most of the problems are only handled by a small amount of researchers these meetings are very effective in highlighting all the existing models of the discussed issue. Naturally basic flaws will rise at that stage. Only afterwards ideas tend to be submitted to publication. Much can be written about the peer review system but if it is done well it will also point out weak claims and flaws in logic. Though, due to the aforementioned stages, it is usually focused on technical details or gaps in background and references rather than the baseline logic. Lastly, a published work is not the bottom line of an argument. Other papers may be published with other models or new evidence that will make older publications irrelevant. Since those papers will have to go through all the previous stages as well it is likely that they will result in either new agreed baseline or agreed disagreement that requires further evidence. In both cases it is unlikely that the argument will generate further publications (and thus may explain your findings). The exception is when new evidence emerge in a case of agreed disagreement. I find it likely that all sides of the argument will try to use their models to explain the new evidence with varying degrees of success. Normally when a model stop fitting the evidence it is simply ignored in future publications. So it may still appear to exist as a possible explanation but it is effectively not there. In most cases to find the agreed models for a problem researchers turn to review papers. These are meant to present the issue, all possible explanations and the strong and weak points of each model.
So, to sum things up, I think the process you are looking for is the one done under less official interactions. Theories are confronted in meetings and such. Less accurate theories are simply ignored in future discourse.
There are clear disadvantages to this system. The main one is that it is clearly sensitive to manipulation by strong people on key positions in their field. A theory can be ignored not because it is wrong but because its publisher is not popular. This may be the strongest argument to why publications that are agreed to be wrong should not be changed or commented after the fact.
I hope this makes sense and answers your question.
Lior
Could you elaborate? How one should know which publications are agreed to be wrong?