> I think it's cultural and goes back to the rise of Christianity.
This seems testable with a cross-cultural analysis. Not just the pre-Christian Greek stories that Garrett mentioned, but Chinese, Japanese, Indian, and Middle Eastern cultures should have plenty of non-Christian stories.
This is pretty cool. As for Opus, could you just use it for "free" by running it in Claude Code and use your account's built-in usage limits.
Edit: That might also work for gemini-cli and 2.5 Pro.
This is a great story and the animation is also great. Good work everyone!
Oh yeah, I also find that annoying.
> If we were to put a number on how likely extinction is in the absence of an aggressive near-term policy response, MIRI’s research leadership would give one upward of 90%.
This is what I interpreted as implying p(doom) > 90%, but it's clearly a misreading to assume that someone advocating for "an aggressive near-term policy response" believes that it has a ~0% chance of happening.
I am in camp 2, but will try to refine my argument more before writing it down.
This is great. I recognize that this is almost certainly related to the book "If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All", which I have preordered, but as a standalone piece I feel estimating p(doom) > 90 and dismissing alignment-by-default without an argument is too aggressive.
Yeah, I don't think I read this when it came out, but I'm happy to read it now.
https://www.lesswrong.com/posts/yew6zFWAKG4AGs3Wk/foom-and-doom-1-brain-in-a-box-in-a-basement (and the sequel) seem highly related.
If you had a solution to alignment, building a Night-Watchman ASI would be decent, but that is a massive thing to assume. At the point where you could build this, it might be better to just build an ASI with the goal of maximizing flourishing.
Avoiding what you suggested is why private conversations are an advantage. I think you misunderstood the essay, unless I'm misunderstanding your response.