Have you incorporated Government Notkilleveryoneism into your model? Rational people interested in not dying ought to invest in AI safety in proportion to the likelihood that they expect to be killed by AI. Rational governments ought to invest in AI safety in proportion to the likelihood they expect to be killed by AI. But, as we see from this article, what kills governments is not what kills people. 

The government of Acheampong is in more danger from a miscalculated letter than from an AI-caused catastrophe that kills 10% of people or more, so long as important people like Acheampong are not among the 10%. The government of Acheampong, as a mimetic agent, should not care about alignment as much as it should care about having a military; governments maintain standing armies for protection against threats from within and without its borders.

You do not have to expect a government to irrationally invest in AI alignment anyway. Private Investment in AI 2024 was $130 billion. There is plenty of private money which could dramatically increase funding for AI safety. Yet it has not happened. Here are some potential explanations.

1) People do not think there is AI risk, because other people do not think there is AI risk, the issue you mentioned.

2) When people must put their money where their mouth is, their revealed preference is that they see no AI risk.

3) People think there is AI risk, but see no way to invest in mitigating AI risk. They would rather invest in flood insurance.

If option 3 is the case, Mr. Lee, you should create AI catastrophe insurance. You can charge a premium akin to high-risk life insurance. You can invest some of your revenue in assets you think will survive the AI catastrophe and then distribute these to policyholders or policyholders' next of kin in case of catastrophe, and you can invest the rest in AI safety. If there is an AI catastrophe, you will be physically prepared. If there is not an AI catastrophe, you will profit handsomely from the success of your AI safety investments and the service you would have done in a counterfactual world. You said yourself "nobody is doing anything about it." This is your chance to do something. Good luck. I'm excited to hear how it goes.

Agreed. Unfortunately, I do not have the intellectual acumen to translate this into a playful rant, so I am afraid I must leave this as is.

A spicy hypothesis raised by this is that socializing too much with children is simply not good for your intellectual development. (I’m not going to test that hypothesis!)

Or, more likely, these children were unusual to begin with. Foucault, for all the weakness in his thinking, understood this pretty well. Strange people can help society to change. But remember, for every odd fellow ten standard deviations up there is another odd fellow ten standard deviations down. 

^ Growing pains.

You are both more correct than me, my wording was quite strong because I just felt like a discussion of the importance of a non-sendentary lifestyle for improvements in thinking was completely unadressed on the site. Hopefully someone else can read more carefully and help us to optimize based on our many varied characteristics. Thanks for the emotional dampening!

Playful thinking sets up questions in your brain that you wouldn't have known how to set up just by explicitly trying to set up questions. A question is a record of a struggle to deal with some context. These questions, like the dozen problems Feynman carries around with him, set you up to connect with new ideas more thoroughly.

Reminds me of Feynman's story about how a plate got him out of a depressive funk. https://kongar-olondar.bandcamp.com/track/at-cornell-part-2 

Non-athletic thinkers are myopic. I see at least three very important reasons any rationalist must value exercise.

As someone who has trained extremely hard in distance running, sports in general and athletic conditioning in particular create an intuitive understanding of the fallaciousness of Cartesian Dualism and the accuracy of materialism in the sense that the mind is merely a part of the body. 

Physical challenges also force one to understand the limitations of Kahneman's "system two." For example, one may know what it means to not start a race too quickly and then "die" (running jargon for running out of steam prematurely), but repeated failures in actual races teach one to realize the limitations of one's rationality, especially under stress (even if distance running is classified as eustress rather than distress; remember, cortisol levels & arousal are the same in either case).

Conditioning has been shown to increase cognitive performance. This study notes a difference better reaction time in fit adolescents (no causal link, n = 30). This review shows decreased "system 2" aging in physically fit elderly individuals (causality likely, preponderance of evidence). Acute exercise has also been shown to cause immediate improvements in decision making.

In your attempt to sound Rational, you have merely restated the platitude, "would you jump off a bridge if your friends did it?" You claim that you are a follower of efficient markets, but then you say that this was an acceptable exception. Great news, you got lucky this once. Don't defect again, you overestimate yourself.

If the false positive rate is 5 percent, 50 out of every one thousand will test false positive. An additional person will test true positive. Therefore 51 will test positive. The probability a positive tester being positive is 1 out of 51. Multiply to 2 out of 102, and it becomes clear .02 overestimates the true value of those who would test positive as 2 hundredths is greater than 2 hundred-and-secondths.