I think the current situation is/was greatly distorted by signalling games that people play. Once everyone realises that this is an actual choice, there is a chance they change their opinions to reflect the true tradeoff. (This depends a lot on network effects, shifting Overton window etc., I'm not claiming that 100% of the effect would be rational consideration. But I think rational consideration biases the process to in a non-negligible way.). But yes, one of the pieces of evidence is how old people don't seem to particularly care about the future of civilisation.
Statement: I want to deliberately balance the caution and the recklessness in developing AGI, such that it gets created in the last possible moment so that I and my close ones do not die.
This Statement confuses me. There are several observations I can make about it. There are also many questions I want to ask but have no idea how to answer. The goal of this post is to deconfuse myself, and to get feedback on the points that I raised (or failed to raise) below.
First observation: The Statement is directly relevant to LW interests.
It ties together the issues of immortality and AI risk, both of which are topics people here are interested in.... (read 1295 more words →)
You are right, there are three possible avenues of approaching this: (1) people have certain goals and lie about them to advance their interests, (2) people have certain goals, and they self-delude about their true content so that they advance their interests, (3) people don't have any goals, they are simply executing certain heuristics that proved to be useful in-distribution (Reward is not an optimisation target approach), I omitted the last one from the post. But think that my observation about (2) having non-zero chance of explaining variance in opinions still stands true. And this is even more true for people engaged in AI safety, such as members of Pause AI, e/acc and (to a lesser extent) academics doing research on AI.
Even if (3) has more explanatory power, it doesn't really defat the central point of the post, which is the ought question (which is a bit of a evasive answer, I admit).