This is a special post for quick takes by romeostevensit. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I'm worried about notkilleveryonism as a meme. Years ago, Tyler Cowen wrote a post about why more econ professors didn't blog, and his conclusion was that it's too easy to make yourself look like an idiot relative to the payoffs. And that he had observed this actually play out in a bunch of cases where econ professors started blogs, put their foot in their mouth, and quietly stopped. Since earnest discussion of notkilleveryonism tends to make everyone, including the high status, look dumb within ten minutes of starting, it seems like there will be a strong inclination towards attribute substitution. People will tend towards 'nuanced' takes that give them more opportunity to signal with less chance of looking stupid.
I dunno, the problem with "alignment" is that it doesn't unambiguously refer to the urgent problem, but "notkilleveryoneism" does. Alignment used to mean same-values, but then got both relaxed into compatible-values (that boundary-respecting norms allow to notkilleveryone) and strengthened with various AI safety features like corrigibility and soft optimization. Then there is prosaic alignment, which redefines it into bad-word-censure and reliable compliance with requests, neither being about values. Also, "existential catastrophe" inconveniently includes ... (read more)