I think there should be much more seminars about philosophy of science, methodology of science, and strategy of research in the program. Perhaps, not at the expense of other seminar content, but via increasing the number of seminar hours. Talks about ethics, and in particular about ethical naturalism (such as Oliver Scott Curry’s “Morality as Cooperation”), or interaction of ethics and science more generally, also seem essential.
MATS is currently focused on developing scholars as per the "T-model of research" with three main levers:
The "T-model of research" is:
In the Winter 2022-23 Cohort, we ran several research strategy workshops (focusing on problem decomposition and strategy identification) and had dedicated scholar support staff who offered regular, airgapped 1-1 support for research strategy and researcher unblocking. We will publish some findings from our Scholar Support Post-Mortem soon. We plan to run further research strategy and “originality” workshops in the Summer 2023 Cohort and are currently reviewing our curriculum from the ground up.
Currently, we are not focused on “ethical naturalism” or similar as a curriculum element as it does not seem broadly useful for our cohort compared to project-specific, scholar-driven self-education. We are potentially open to hosting a seminar on this topic.
I'm somewhere in the middle of the cognitivist/enactivist spectrum. I think that e.g. relaxed adversarial training is motivated by trying to make an AI robust to arbitrary inputs it will receive in the world before it leaves the box. I'm sympathetic to the belief that this is computationally intractable; however, it feels more achievable than altering the world in the way I imagine would be necessary without it.
I'm not an idealist here: I think that some civilizational inadequacies should be addressed (e.g., better cooperation and commitment mechanisms) co... (read more)
(Independent) alignment researchers need to be exceptionally good at philosophy of science
In order to be effective, independent AI alignment/safety/x-risk researchers should be unusually competent in philosophy of science, epistemology, and methodology of research (the latter imports some strategic considerations, too, as I’ll illustrate shortly), relative to other fields of research.
The reasons for this are:
(1) The field is pre-paradigmatic.
(2) There is a number of epistemic complications with this type of research, many of which are detailed by Adam Shimi here (also see my comment to this post). I’d add, on top of that, that the very concept of risk is poorly understood and risk science is often conducted poorly (usually because one doesn’t realise they are doing risk science).
(3) The object of research relates to ethics (i.e., philosophy, unless one subscribes to ethical naturalism), so researchers should be able to properly synthesise science and philosophy.
(4) Alignment work within AGI labs tends to be more empirical and iterative, which partially ameliorates some of the epistemic challenges from (2), but independent researchers usually can’t afford this (also, this approach doesn’t make sense for them to pursue strategically), so these challenges remain unaddressed.
At the same time, I’m frustrated when I see a lot of alignment/safety research on LessWrong that resembles philosophy much more than science, towering reasoning on top of intuitions and unprincipled ontologies rather than more established scientific theories or mechanical models. This is a dead end and one of the important reasons why this work doesn’t stack. Contrasted with the particular demand for philosophy of science, this situation seems to harbour a huge opportunity for improvement.
Alignment researchers should think hard about their research methodology and strategy
Talking about research strategy, I also often see lapses, such as (but not limited to):
What to do about this?
At universities, students and young researchers are expected to learn good philosophy of science primarily “covertly”, by “osmosis” from their advisors, professors, and peers, plus the standard “Philosophy” or “Philosophy of Science” courses. This “covert” strategy of upskilling in philosophy of science is itself methodologically dubious and strikes me as ineffective, e.g., it hinges too much on how good at philosophy of science is the mentor/advisor themselves, which may vary greatly.
In programs like SERI MATS, the “osmosis” strategy of imparting good philosophy of science to people becomes even more problematic because the interaction and cooperation time is limited: in university, a student regularly meets with their advisor and other professors for several years, not several months.
Therefore, direct teaching via seminars seems necessary.
I surveyed the seminar curriculum of the last cohort and it barely touches on philosophy of science, epistemology, methodology and strategy of research. Apart from the debate that I mentioned above, which touches on the strategy, “How much should be invested in interpretability research?”, there is also a seminar about “Technical Writing” which is about methodology, and that’s it.
I think there should be much more seminars about philosophy of science, methodology of science, and strategy of research in the program. Perhaps, not at the expense of other seminar content, but via increasing the number of seminar hours. Talks about ethics, and in particular about ethical naturalism (such as Oliver Scott Curry’s “Morality as Cooperation”), or interaction of ethics and science more generally, also seem essential.
My own views on philosophy of science and methodology of research in application to AI Safety are expressed in the post "A multi-disciplinary view on AI safety research". It’s similar in spirit to Dan Hendryks’ “Interdisciplinary AI Safety”, based on the “Pragmatic AI Safety” agenda, but there are also some differences that I haven’t published yet and these differences are beyond the scope of this letter to detail.
It seems that Santa Fe Institute is organizationally interested in AI and is hosting a lot of relevant seminars themselves, so I imagine they would be interested to collaborate with SERI MATS on this. They are also good at philosophy of science.