From Making Modern Science:
To those who (like Whewell) retained the hope that science and religion could work in harmony, the materialist program of the Enlightenment was a positive danger to science. It encouraged scientists to abandon their objectivity in favor of the arrogant claim that the laws of nature could explain everything. Alfred North Whitehead's Science and the Modern World (1926) urged the scientific community to turn its back on this materialist program and return to an earlier vision in which nature was studied on the assumption that it would reveal evidence of divine purpose. This model of science's history dismisses episodes such as the trial of Galileo as aberrations and portrays the Scientific Revolution as founded on the hope that nature could be seen as the handiwork of a rational and benevolent Creator. For Whitehead and others of his generation, evolution itself could be seen as the unfolding of a divine purpose. This debate between two rival views of science-and hence of its history-is still active today.
In the early twentieth century, the legacy of the rationalist program was transformed in the work of Marxists such J. D. Bernal. Bernal, an eminent crystallographer, berated the scientific community for selling out to the industrialists. In his Social Function of Science (1939) he called for a renewed commitment to use science for the good of all. His 1954 Science in History was a monumental attempt to depict science as a potential force for good (as in the Enlightenment program) that had been perverted by its absorption into the military-industrial complex. In one important respect, then, the Marxists challenged the assumption that the rise of science represented the progress of human rationality. For them, science had emerged as a byproduct of the search for technical mastery over nature, not a disinterested search for knowledge, and the information it accumulated tended to reflect the interests of the society within which the scientist functioned. The aim of the Marxists was not to create a purely objective science but to reshape society so that the science that was done would benefit everyone, not just the capitalists. They dismissed the program advocated by Whitehead as a smokescreen for covering up science's involvement in the rise of capitalism. Similarly, many intellectual historians reacted furiously to what they regarded as the denigration of science implicit in works such as the Soviet historian Boris Hessen's "The Social and Economic Roots of Newton's `Principia"' from 1931. The outbreak of World War II highlighted two conflicting visions of science's history, both of which linked it to the dangers revealed in Nazi Germany. The optimistic vision of the Enlightenment had vanished along with the idea of inevitable progress in the calamities that the Western world had now experienced. Science must either turn its back on materialism and renew its links with religion or turn its back on capitalism and begin fighting for the common good.
More (#1) from Making Modern Science:
...To some extent, the [history of science] continued and extended the Whiggish approach favored by the scientists themselves, because progress was defined in terms of steps toward what were perceived to be the main components of our modern worldview. In another respect, however, the new historiography of science did go beyond Whiggism: it was willing to admit that scientists were deeply involved with philosophical and religious concerns and often shaped their theories in accordance with their views on these wider questi
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: