Note: I'm not a WWII history buff, so if there's any significant misinterpretation of relevant history here, I welcome corrections in the comments.
In December 1944, Joseph Rotblat, a physicist working on the Manhattan project, resigned. He had reasoned that Germany had most likely abandoned its bomb project, and so his initial reason for working on the bomb was no longer valid.[1] He would later ask himself:
Why did other scientists not make the same decision? [...]there were many scientists for whom the German factor was the main motivation.[2] Why did they not quit when this factor ceased to be?
According to him, conversations with the other scientists suggested the following reasons (emphasis mine):
The most frequent reason given was pure and simple scientific curiosity—the strong urge to find out whether the theoretical calculations and predictions would come true. [...]
Others were [...] persuaded by the argument that many American lives would be saved if the bomb brought a rapid end to the war with Japan. Only when peace was restored would they take a hand in efforts to ensure that the bomb would not be used again.[...][3]
Still others, while agreeing that the project should have been stopped when the German factor ceased to operate, were not willing to take an individual stand because they feared it would adversely affect their future career.[...]
The majority were not bothered by moral scruples; they were quite content to leave it to others to decide how their work would be used.
Here's some corroborations of this perspective from other scientists:
Richard Feynman:
The original reason to start the project, which was that the Germans were a danger, started me off on a process of action which was to try to develop this first system at Princeton and then at Los Alamos, to try to make the bomb work. All kinds of attempts were made to redesign it to make it a worse bomb and so on. But what I did, - immorally I would say – was to not remember the reason that I said I was doing it, so that when the reason changed, because Germany was defeated, not the singlest thought came to my mind at all about that, that that, meant now I would have to reconsider why I am continuing to do this. I simply didn’t think, okay?
Victor Weisskopf:
One of the reasons was for us to work in Los Alamos was to prevent Hitler from getting the bomb. But obviously he didn't have it. Should we go on working? Now I don't think that anybody ever thought about this at that time, and I would like to explain why. Why? This was two months before the bomb was finished. We were in the midst of a tremendous thing where we worked almost literally day and night. How could you at this moment stop? [...] I would almost say, practically nobody thought of stopping at that time.
Now I can rest. We have done our job. And ah, what happens now of application and so is not our business... [4]
Robert Oppenheimer:
When you see something that is technically sweet, you go ahead and do it and you argue about what to do about it only after you have had your technical success. That is the way it was with the atomic bomb.
Freeman Dyson:
they actually enjoyed doing what they were doing. It was not just that they were soldiers fighting in a good cause, but they were also having a great time, having the best time of their lives, and that was in some sense sinful.
Roy Glauber:
We had joined this project fearing that the Germans were working on trying to produce a bomb. If they succeeded in reaching it before we did, they wouldn’t be very sentimental about using it. [...] We never heard any suggestion from Oppenheimer that there was any course other than continuing. There was a kind of momentum involved in our efforts in this direction. It was an enormous project, we were all deeply involved in finding out whether the darn thing would work.
Adverse epistemic environments
Another important detail is that after deciding to resign, Rotblat was not permitted by the intelligence chief to tell anyone about his reasons for doing so, because of concerns about how it would affect the motivation of the other scientists. (Instead, he had to make up a fake personal reason for quitting.) Even worse, he was informed by the chief of a bunch of evidence suggesting that he was a Soviet spy (the evidence was proven to be fabricated).[5]
The immediate takeaway from this is that once you are placed in an environment where others with power benefit from your work, not only do you yourself have incentives to think one way or another; not only do others have incentives to make you think one way or another; there can be asymmetric pressures (deliberately, adversarially) placed on the very ideas that make it to your awareness.[6]
Beating a dead horse
Okay, so things being what they are lately, obviously this is actually about AI, with many salient comparisons of varying aptness.
(Consider: with Fable 5.1 competitive with GPT 6 Astra and Opus 5 competitive with GPT 6 Sol, did Anthropic really need to release Opus 5.5? When shortly thereafter OpenAI announced a pause on all training, did Anthropic then need to release Sonnet 5.5?)
Appendix: Some nitpicking about whether it was in fact clear that Germany didn't have a bomb
Rotblat left the project in late 1944, possibly based on evidence that was not fully public about the state of the German bomb project. However, he reasoned substantially from the fact that America, with its tremendous effort and resource investment, still struggled to put one together, and that Germany lacked resources badly enough that it was losing at conventional war in the meantime. No matter what, it'd be possible to argue that it was necessary to work towards the bomb as long as there was any uncertainty about the matter, but surely this argument fell apart after Germany fully surrendered in April 1945, and no one quit then, either.[7]
Appendix: Some other provocative quotes from the Franck report
The scientists on this project do not presume to speak authoritatively on problems of national and international policy. However, we found ourselves, by the force of events, during the last five years, in the position of a small group of citizens cognizant of a grave danger for the safety of this country as well as for the future of all the other nations, of which the rest of mankind is unaware. We therefore feel it is our duty to urge that the political problems, arising from the mastering of nuclear power, be recognized in all their gravity, and that appropriate steps be taken for their study and the preparation of necessary decisions.
If an international agreement is not concluded immediately after the first demonstration, this will mean a flying start toward an unlimited armaments race. If this race is inevitable, we have every reason to delay its beginning as long as possible in order to increase our head start still further.
Another argument which could be quoted in favor of using atomic bombs as soon as they are available is that so much taxpayers' money has been invested in these Projects that the Congress and the American public will demand a return for their money. The attitude of American public opinion, mentioned earlier, in the matter of the use of poison gas against Japan, shows that one can expect the American public to understand that it is sometimes desirable to keep a weapon in readiness for use only in extreme emergency; and as soon as the potentialities of nuclear weapons are revealed to the American people, one can be sure that they will support all attempts to make the use of such weapons impossible.
Nevertheless, they felt, having gone so far, they would like to see whether all these theoretical calculations, whether they really come out in practice. I mean, in other words, they wanted to see until the bomb was tested, which came later, in July '45, and then perhaps they would leave the project. [...] By that time, when Japan entered the war and the cruelty of Japan to the war prisoners became well known, many people by then... at that time changed their views about the bomb; they felt we may have to use it in order to bring the war in the Far East to a rapid end. And I believe that this is the psychology, the war psychology: once we enter war, we almost lose... you know, our moral values are thrown overboard, and we are encouraged, you know, to kill people[...]
during the Manhattan Project, the ideas gradually changed from... from needing the bomb as a deterrent, to the idea that we need to use it, because we needed to finish the war very quickly. But this, of course, exists not only during the war years. Science as such, pure science, does not involve making weapons of mass destruction. This if ever it needs to be done, it should be done by some military engineers and so on. Nevertheless, scientists did take an active part later on, not only during the Manhattan Project. But during the war, the time when we had the arms race going on, scientists played a very important role in keeping up the momentum of the arms race. In other words, they really became almost addicted to making weapons of mass destruction.
Relevant quote: When it became evident, toward the end of 1944, that the Germans had abandoned their bomb project, the whole purpose of my being in Los Alamos ceased to be... ↩︎
Independent evidence of this statement can be found in the Franck report, made to the Secretary of War in June 1945; relevant quote: "One may point out that scientists themselves have initiated the development of this "secret weapon" and it is therefore strange that they should be reluctant to try it out on the enemy as soon as it is available. The answer to this question was given above -- the compelling reason for creating this weapon with such speed was our fear that Germany had the technical skill necessary to develop such a weapon and that the German government had no moral restraints regarding its use." ↩︎
This is the most reasonable-seeming/least selfish argument and with the knowledge of hindsight it seems more reasonable still. To the present day the validity of this argument is legitimately debated. Nevertheless, consider the discussion to follow about motivated reasoning. ↩︎
He of course didn't feel this way after the fact: "Now perhaps one could defend in some ways the bombing of Hiroshima, although I would have preferred a demonstration at least to try it out. But anyway, it did have the effect and it end...it ended the war. But why did we really bomb Nagasaki three days after?" ↩︎
To his credit, it seems like the intelligence chief wasn't outright threatening him with a false accusation, since in his words he "was rather embarrassed by all this and conceded that the dossier was worthless". ↩︎
I think I originally saw this point made on LW somewhere, but I can't find where.
It should also be noted that a bunch of other scientists did seem to express hesitation and regret well ahead of the dropping of the bomb, most notably Leo Szilard and Robert R. Wilson, with the latter having explicitly considered quitting, and regretting "not having considered it". A bunch of Los Alamos scientists also made an effort to prevent the bomb being dropped on Japan, but ultimately were not listened to. ↩︎
Note: I'm not a WWII history buff, so if there's any significant misinterpretation of relevant history here, I welcome corrections in the comments.
In December 1944, Joseph Rotblat, a physicist working on the Manhattan project, resigned. He had reasoned that Germany had most likely abandoned its bomb project, and so his initial reason for working on the bomb was no longer valid.[1] He would later ask himself:
According to him, conversations with the other scientists suggested the following reasons (emphasis mine):
Here's some corroborations of this perspective from other scientists:
Richard Feynman:
Victor Weisskopf:
Robert Oppenheimer:
Freeman Dyson:
Roy Glauber:
Adverse epistemic environments
Another important detail is that after deciding to resign, Rotblat was not permitted by the intelligence chief to tell anyone about his reasons for doing so, because of concerns about how it would affect the motivation of the other scientists. (Instead, he had to make up a fake personal reason for quitting.) Even worse, he was informed by the chief of a bunch of evidence suggesting that he was a Soviet spy (the evidence was proven to be fabricated).[5]
The immediate takeaway from this is that once you are placed in an environment where others with power benefit from your work, not only do you yourself have incentives to think one way or another; not only do others have incentives to make you think one way or another; there can be asymmetric pressures (deliberately, adversarially) placed on the very ideas that make it to your awareness.[6]
Beating a dead horse
Okay, so things being what they are lately, obviously this is actually about AI, with many salient comparisons of varying aptness.
(Consider: with Fable 5.1 competitive with GPT 6 Astra and Opus 5 competitive with GPT 6 Sol, did Anthropic really need to release Opus 5.5? When shortly thereafter OpenAI announced a pause on all training, did Anthropic then need to release Sonnet 5.5?)
Appendix: Some nitpicking about whether it was in fact clear that Germany didn't have a bomb
Rotblat left the project in late 1944, possibly based on evidence that was not fully public about the state of the German bomb project. However, he reasoned substantially from the fact that America, with its tremendous effort and resource investment, still struggled to put one together, and that Germany lacked resources badly enough that it was losing at conventional war in the meantime. No matter what, it'd be possible to argue that it was necessary to work towards the bomb as long as there was any uncertainty about the matter, but surely this argument fell apart after Germany fully surrendered in April 1945, and no one quit then, either.[7]
Appendix: Some other provocative quotes from the Franck report
Appendix: More quotes
From another interview of Rotblat:
Past discussions of the matter on LW
A brief history of ethically concerned scientists
Michael Nielsen remarks on 'Oppenheimer'
Principles for the AGI Race
Relevant quote: When it became evident, toward the end of 1944, that the Germans had abandoned their bomb project, the whole purpose of my being in Los Alamos ceased to be... ↩︎
Independent evidence of this statement can be found in the Franck report, made to the Secretary of War in June 1945; relevant quote: "One may point out that scientists themselves have initiated the development of this "secret weapon" and it is therefore strange that they should be reluctant to try it out on the enemy as soon as it is available. The answer to this question was given above -- the compelling reason for creating this weapon with such speed was our fear that Germany had the technical skill necessary to develop such a weapon and that the German government had no moral restraints regarding its use." ↩︎
This is the most reasonable-seeming/least selfish argument and with the knowledge of hindsight it seems more reasonable still. To the present day the validity of this argument is legitimately debated. Nevertheless, consider the discussion to follow about motivated reasoning. ↩︎
He of course didn't feel this way after the fact: "Now perhaps one could defend in some ways the bombing of Hiroshima, although I would have preferred a demonstration at least to try it out. But anyway, it did have the effect and it end...it ended the war. But why did we really bomb Nagasaki three days after?" ↩︎
To his credit, it seems like the intelligence chief wasn't outright threatening him with a false accusation, since in his words he "was rather embarrassed by all this and conceded that the dossier was worthless". ↩︎
I think I originally saw this point made on LW somewhere, but I can't find where.
It should also be noted that a bunch of other scientists did seem to express hesitation and regret well ahead of the dropping of the bomb, most notably Leo Szilard and Robert R. Wilson, with the latter having explicitly considered quitting, and regretting "not having considered it". A bunch of Los Alamos scientists also made an effort to prevent the bomb being dropped on Japan, but ultimately were not listened to. ↩︎