This is a special post for quick takes by 307th. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Some people think it is bad to work at a frontier lab because it makes p(doom) higher. This is a strong reason to not work at a frontier lab if true, but is debatable. So they often add: yes, it's bad for consequentialist reasons, but also it is just obviously bad for deontological reasons to do something that might destroy the world - even if it appears to you to make p(doom) lower, you still shouldn't do it. I don't think they actually believe this - their real reason is the consequentialist one.
To some extent, this is already obvious because in order to pass the intellectual turing test I had to say "even if it appears to you to make p(doom) lower", rather than "even if it makes p(doom) lower". But to illustrate this point further:
Suppose there is a bomb with two wires. One, when cut, defuses the bomb - the other instantly detonates, destroying the world. If left alone, the bomb will detonate anyway in 10 minutes. You have no idea which wire is which, and no one else does either. Do you cut a wire?
Obviously you should. But, if you do, you are "taking an action which might destroy the world" in a much stronger sense than a lab employee - you are personally and solely responsible for a 50% chance of destroying the world.
You can also tweak the example so that it is at least possible that by doing nothing, the world doesn't get destroyed - maybe there is only a 99% chance it detonates in 10 minutes instead of 100%. Even still, you should cut a wire.
Someone who argued "you cut the wire, you admit this had a 50% chance of killing everyone, that is worse than the holocaust and you should be held accountable" would be being disingenuous, because they would be omitting the fact that if no one had cut the wire, there would be a 99% chance of everyone being killed.
In other words, the entire issue hinges on the consequentialist cost-benefit evaluation, rather than some deontological bar.
Don't deontologists believe you're deontologically barred from destroying the world? You seem to have given an example where deontologists and consequentialists would disagree about the moral course of action, and then stated that obviously you should do the thing dictated by consequentialism because the consequences will be better. Look, I'm inclined to agree with consequentialism, but there are deontologists. They believe you shouldn't pull the lever in the trolley problem even though if you do pull it fewer people will die. Immanuel Kant really believed... (read more)