This is a special post for quick takes by Vladimir_Nesov. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
In a recent lecture at Oxford, Jack Clark (Anthropic co-founder) was somewhat clear about extinction risk and the prudence of a global slowdown in AI progress. Though he doesn't discuss credences around irreversible loss of control and is being fatalistic about the inability to coordinate a slowdown. This is in the context of expecting prosaic RSI and general automation [1] . At 10:42:
It's a technology that we do not fully understand because it's more grown than made. And it is a technology that you can concoct plausible scenarios where it could kill every single person on the planet. So to think building this technology is without risk would be an act of hubris or insanity.
The technology is in fact so powerful that I should clearly state that if it was possible to elegantly slow the development of this technology to give ourselves more time as a species to deal with it, that would likely be a good thing. ... But in the absence of a coordinated global slowdown, we are left with the current situation, which is a powerful technology being developed at breakneck speed by a variety of actors and a variety of countries locked in a competition with one another where commercial and geopolitical rivalries are often drowning out the larger existential-to-the-species aspects of the technology being built. This isn't an ideal situation, but it's the one we find ourselves in.
This is happening without superintelligence in a sense I find meaningful, being qualitatively more capable than humanity at everything relevant to strategic planning and technological development. The way Clark uses "superintelligence" seems to be more about scalability and the speed of execution, as he says at 43:22:
↩︎We will have superintelligences show up in the world sooner than we think that grant the power of synthetic workforces and nation-state security skills to individuals. We'll also have individuals like me and many others in this room who are able to take work that previously took them weeks and now do it in minutes.
5
1