The framing is based on contrasting Orthodox Judaism with Reform Judaism, only for AI Alignment.
If AI alignment is a religion, it’s now large and established enough to have a thriving “Reform” branch, in addition to the original “Orthodox” branch epitomized by Eliezer Yudkowsky and MIRI. As far as I can tell, this Reform branch now counts among its members a large fraction of the AI safety researchers now working in academia and industry. (I’ll leave the formation of a Conservative branch of AI alignment, which reacts against the Reform branch by moving slightly back in the direction of the Orthodox branch, as a problem for the future — to say nothing of Reconstructionist or Marxist branches.)
There are 8 points of comparison, here is just the first one:
(1) Orthodox AI-riskers tend to believe that humanity will survive or be destroyed based on the actions of a few elite engineers over the next decade or two. Everything else—climate change, droughts, the future of US democracy, war over Ukraine and maybe Taiwan—fades into insignificance except insofar as it affects those engineers.
We Reform AI-riskers, by contrast, believe that AI might well pose civilizational risks in the coming century, but so does all the other stuff, and it’s all tied together. An invasion of Taiwan might change which world power gets access to TSMC GPUs. Almost everything affects which entities pursue the AI scaling frontier and whether they’re cooperating or competing to be first.
This framing seems new and worthwhile, crystallizing some long-standing divisions in the AI community.
Instead of creating new ingroups and outgroups and tribal signifiers for enforcing such, we should focus on careful truth-seeking. Some mythologies and terms that engage our more primal instincts can be useful, like when Scott introduced "Moloch", but others are much more likely harmful. "Orthodox vs Reform" seems like a purely harmful one, that is only useful for enforcing further division and tribal warfare.
To summarize, in this post Aaronson,
Enforces the idea that AI safety is religion.
Creates new terminology to try and split the field into two ideological groups.
Chooses terminology that paints the other one as "old, conforming, traditional" and the other as, to quote wiktionary, "the change of something that is defective, broken, inefficient or otherwise negative".
Immediately adopts the tribal framing he has created, and already identifies as "We Reform AI-riskers", to quote him directly.
This seems like a powerful approach for waging ideological warfare. It is not constructive for truth-seeking.
This is wonderfully written, thank you.
I do worry that "further division and tribal warfare" seems the default, unless there's an active effort at reconciliation.
Sam Altman (CEO of OpenAI) tweets things like:
LessWrong compares OpenAI with Phillip Morris and, in general, seems very critical of OpenAI. "I've seen a lot more public criticism lately".
I doubt that it's actually good to have this strong division, and it might have positive EV to try to move into a more cooperative direction, and try to lower the temperature and divisiveness.
The framing is based on contrasting Orthodox Judaism with Reform Judaism, only for AI Alignment.
There are 8 points of comparison, here is just the first one:
This framing seems new and worthwhile, crystallizing some long-standing divisions in the AI community.