You submit a finished product, yes, and it can be accepted without revisions, but I have never heard of that happening actually and nobody I know has had that happen to them, I believe. Or, it might get rejected (but if so, no, you don't have to start over. If it was sent for review, you will receive feedback you can use to improve the study, and you may be invited to resubmit after making those changes, or you might submit the same paper to a different journal). Hopefully, it is accepted with major or minor revisions, so you go away and make the requested changes over a few more months, and then the reviewers take another look. And these changes can, but not always, be significant alterations to the study design.

Examples from my recent experience: I submitted a paper recently that developed a new data analysis method and then evaluated it on two different synthetic datasets. I was then asked by the editor for revisions: obtaining and using observational data as well as synthetic data. That's not changing the original study design, but it is a new chunk of research, a lot of work, and the results have to be interpreted differently. Another paper that I co-authored has been asked for major revisions which, if implemented, would be a massive change in the setup, data used, analysis methodology and narrative of the paper. The lead author is still deciding if they want to do that or instead to withdraw and resubmit somewhere else. On the other hand, often I have only been asked for minor text changes to explain things more clearly.

In Nature, the peer review files are openly available for each article, and they are pretty interesting to read, because papers there often go through quite significant changes before publication. That's a good way to get an idea of the ways papers and studies can evolve as they go through the peer review process. But, yeah, I assure you, in my experience as an author and reviewer, it is a collaborative process that can really reshape the study design in some cases.

Yes.

Generations of data, fewer differences in environmental factors between members (diets, lifestyle, diseases, etc) to obscure the effect. For long-term effects like this, 'modern science' hasn't really existed long enough to get much data in comparison to centuries of generational trial-and-error

Edit: also, long-term effects measured now have a bunch of confounders due to lifestyle change and rapid technological and medical development, while their conditions were basically stationary. Scientists would kill for that kind of data now!

Peer review usually results in papers being accepted with minor or major revisions, and very much can and does effect serious changes in the study design. You can read the peer review back-and-forth in many journals, they are often pretty interesting. Machine learning and computer science are different because they usually publish in conference proceedings. That means there are very tight deadlines, so it's more common to rebut the reviewer's comments outside of very minor changes. In my opinion it's why peer review is seen so poorly in ML, because there's not much paper-improvement going on as a result of the process.

The editor of an article is the person who decides whether to desk-reject or seek reviewers, find and coordinate the reviewers, communicate with the authors during the process and so on. That's standard at all journals afaik. The editor decides on publication according to the journal's criteria. PNAS does have this special track but one of the authors must be in NAS, and as that author you can't just submit a bunch of papers in that track, you can use it once a year or something. And most readers of PNAS know this and are suitably sceptical of those papers (and it's written on the paper if it used that track). The journal started out only accepting papers from NAS members and opened to everyone in the 90s so it's partly a historical quirk.

Chris Olah was absolutely not the first person to discuss the idea of using interpretability to better understand the underlying data-generating process. Most statistical modelling has been driven by that aim, and that way of thinking wasn't just abandoned as ML research progressed - Breiman (of random forest fame) discusses it in his 2001 paper on 'the two cultures', for example. While a lot of explainability/interpretability research has focussed on the goal of understanding the model, there has been plenty written and discussed about using those methods for scientific research too, and also plenty of research in various fields trying to do that.

The problem is that believing the results gained when using such methods in this way relies on two assumptions: the interpretations are an accurate reflection of the underlying model, and the model is an accurate reflection of the data-generating process for the phenomenon of interest. I would say that the first assumption is almost definitely invalid and the second is most probably invalid, given poor behaviour of models out-of-distribution (when you would expect a model which has captured causal behaviour to still be performant).

Perhaps you're already aware of all this, and apologies if so, but the fact that you write Olah was possibly the first to mention this suggests to me that you might not be aware of the existing literature on these topics: if you are interested in more of the issues involved in this avenue of research, look into causal discovery for more rigorous discussion of why learning causal relationships is usually impossible using real-world observational data, and explore more of the literature on the lack of robustness, fragility & challenges of explainable/interpretable ML methods. The literature discussing issues with Shapley/SHAP and feature importances is particularly helpful in terms of explicitly focussing on the connection to causal learning.

Finally, if you do research in this way, be very careful to design the methodology such that it includes ways to falsify your results. Check that the trained model responds in realistic ways when you ablate the input to extreme values, to ensure that the relationships captured are plausibly interesting at all. Include artificial predictors that are correlated with other variables and/or your target variable, but are generated after data collection so do not have a causal effect; if those predictors are identified as important in some way, that can act as a red flag that your results aren't capturing causal behaviour. Repeat your methodology on different subsets of data, with different models and parameter settings, and ensure the interpretations are robust. Use multiple interpretation methods and compare. Ideally, start with applying your methodology to synthetic data with similar characteristics and a known data-generating process and verify that your results can capture the right relationships in a simulated setting.

Explainable AI and interpretable ML research and methods, aside from the researchers affiliated with the rationalist scene, are for some reason excluded from the narrative. Is it really your view that 'mechanistic interpretability' is so different that it is an entirely different field? Doesn't it seem a bit questionable that the term 'mechanistic interpretability' was coined in order to distance Olah's research from other explanation approaches that had been found to have fundamental weaknesses - especially when mechanistic interpretability methods repeatedly fall prey to the exact same points of failure? The failure of SDL latents was unsurprising, the fact that it took such a long time for someone to call attention to it should have provoked much more of a discussion on how science is done in this community.

I agree with the similarities to neuroscience, and there is definitely much to learn from that field, but it would be an even easier step to just read a little more widely on interpretable/explainable machine learning and causal discovery, in which there is a wide body of literature discussing the very issues you mention and more. Why is research done outside of the self-labelled 'mechanistic interpretability' community mostly ignored? In neuroscience, if you prefer though, perhaps Jonas & Kording 2017 is relevant: Could a Neuroscientist Understand a Microprocessor? | PLOS Computational Biology https://share.google/WYGmCXAnX8FNbaRqi

I'd like to comment on your discussion of peer review.

'Tyler Cowen’s presentation of the criticism then compounds this, entitled ‘Modeling errors in AI doom circles’ (which is pejorative on multiple levels), calling the critique ‘excellent’ (the critique in its title calls the original ‘bad’), then presenting this as an argument for why this proves they should have… submitted AI 2027 to a journal? Huh?'

To me, this response in particular suggests you might misunderstand the point of submitting to journals and receiving peer review. The reason Tyler says they should have submitted it is not because the original model and publication being critiqued is good and especially worthy of publication, it is because it would have received this kind of careful review and feedback before publication, as solicited from an editor independent of the authors, and anonymously. The authors would then be able to improve their models accordingly and the reviewers and editor would decide if their changes were sufficient or request further revisions.

It is a lot of effort to engage with and critique this type of work, and it is unlikely titotal's review will be read as widely as the original piece, or the updated piece once these criticisms are taken into account. And I also found the responses to his critique slightly unsatisfying - only some of his points were taken on board by the authors, and I didn't see clear arguments why others were ignored.

Furthermore, it is not reasonable to expect most of the audience consuming AI 2027 and similar to have the necessary expertise and time to go through the methodology as carefully as titotal has done. Those readers are also particularly unlikely to read the critique and use it to shape their takeaways of the original article. However, they are likely to see that there are pages and pages of supplementary information and analysis that looks pretty serious and, based on that, assume the authors know what they are talking about.

You are right that AI research moves fast and tends to not bother waiting for the peer review process to finish, which can for sure be frustratingly time-consuming. However, realistically, a lot of ML research articles that are widely shared and hyped without going through peer review are really bad, don't replicate and don't even make an attempt to check the robustness of their findings. The incentive structure changes, leading to researchers overstating their findings on abstracts in order for articles to be picked up on social media, rather than expressing things more cautiously lest their statements be picked apart by the anonymous reviewers. Progress still gets made and very quickly, and the rapid sharing of preprints is definitely really helpful for disseminating ideas early and widely, but this aspect of the field does come with costs and we can't ignore that.

Finally, going through peer review doesn't prevent people from performing additional critique and review, like titotal has done, once an article has been published. It is not either-or. In many journals, peer review reports and responses are also published once the article is accepted, so this is also public.

Peer review is by no means a perfect system and I myself think it should be significantly reworked. However, I think the strengths and weaknesses of the existing structures are often not very well understood by the members of this community who argue for it to be gotten rid of wholesale.

Why not read and review the IPCC report? I am confused by why this seems to not be the most popular recommendation for people on this forum who want to understand the most up-to-date scientific consensus on climate change risk. It's written by an international community of scientists, it's very accessible with further higher-level overviews aimed at decision-makers and the broader public, all claims include an estimated level of uncertainty (and are very conservative) and you can follow the citations for any particular claim made. The website is great, the writing and figures are very clear.

I would not recommend Halstead's report to someone trying to learn more about this topic. His summary of the research is not great in my opinion. This is a huge topic that touches many fields and even an expert in one of these research areas would struggle to put together a good overview of all of the others. But as I skimmed through this I noticed a few takes that are quite...off, and papers that I know to be outdated or widely considered erroneous being used as evidence for claims.

Some more context for those interested: Climate impact research is almost never looking at 'worst case' scenarios, but plausible outcomes. Papers looking at projections with high warming levels are often older, when it was less certain that much action would be taken to curb emissions, and very high warming scenarios seemed more likely. Also, climate science has very different confidence levels than climate impact research. Climate science is heavily based on physics, and doesn't have to deal with the messiness of animal and human biology and behaviours and so on. Impact research is much murkier, as it depends on accurate understanding of the way animals, plants and humans respond to conditions that are outside of what has previously been observed, and often cannot be studied in RCTs. A lack of scientific consensus that extreme impacts are likely should not be mistaken for the presence of scientific consensus that extreme impacts are unlikely.

This weakness of SAEs is not surprising, as this is a general weakness of any interpretation method that is calculated based on model behaviours for a selected dataset. The same effect has been shown for permutation feature importances, partial dependence plots, Shapley values, integrated gradients and more. There is a reasonably large body of literature on the subject from the interpretable ML / explainable ML research communities in the last 5-10 years.