The AI company Mechanize posted a blog post in November called "Unfalsifiable stories of doom" which is a high-quality critique of AI doom and IABIED that I haven't seen shared or discussed on LessWrong yet. Link: https://www.mechanize.work/blog/unfalsifiable-stories-of-doom/ AI summary: the evolution→gradient descent analogy breaks down because GD has fine-grained parameter control unlike evolution's coarse genomic selection; the "alien drives" evidence (Claude scheming, Grok antisemitism, etc.) actually shows human-like failure modes; and the "one try" framing ignores that iterative safety improvements have empirically worked (GPT-1→GPT-5).

on the one hand, it is a desirable feature of an intellectual community to be truth seeking, and while it can be deeply emotionally painful to part ways with deeply held beliefs, in the long run it's better to tear off the bandage. on the other hand, being emotionally hurt all the time by your community kind of fucking sucks, and isn't very good for long term emotional or epistemic health. perhaps a middle ground is in order: intellectual communities should be partitioned into an arena, where every idea is to be exposed to the harsh light of truth, and an antechamber, where you can rest and be surrounded by positivity and develop ideas in a supportive environment.  both are necessary - we need a way to kill bad ideas, because an environment that refuses to discard bad ideas because they are emotionally load bearing is doomed to epistemic ruin. but also the best weird ideas often sound bad initially, and require a safe environment to develop; and we are all human, and our emotional well being and desire to belong to a community is essential. by visibly separating the two, we might be able to get the best of both worlds. this is not a crazy idea - many other parts of society have analogous things. for example, people who play sports for fun with their friends compete to win while on the field, but this only brings them closer off the field. I think one common criticism of LW ist it is too much of an arena, and not enough of an antechamber. perhaps this can be fixed somehow.

I am pretty confused about people who have been around the AI safety ecosystem for a while updating towards "alignment is actually likely by default using RLHF" But maybe I am missing something. Like 3 years ago, it was pretty obvious that scaling was going to make RLHF "work" or "seem to work" more effectively for a decent amount of time. And probably for quite a long time. Then the risk is that later you get alignment-faking during RLHF training, or at the extreme-end gradient-hacking, or just that your value function is misspecified and comes apart at the tails (as seems pretty likely with current reward functions). Okay, there are other options but it seems like basically all of these were ~understood at the time. Yet, as we've continued to scale and models like Opus 3 have come out, people have seemed to update towards "actually maybe RLHF just does work," because they have seen RLHF "seem to work". But this was totally predictable 3 years ago, no? I think I actually did predict something like this happening, but I only really expected it to affect "normies" and "people who start to take notice of AI at about this time." Don't get me wrong, the fact that RLHF is still working is a positive update for me, but not a massive one, because it was priced in that it would work for quite a while. Am I missing something that makes "RLHF seems to work" a rational thing to update on?  I mean there have been developments to how RLHF/RLAIF/Constitutional AI works but nothing super fundamental or anything, afaik? So surely your beliefs should be basically the same as they were 3 years ago, plus the observation "RLHF still appears to work at this capability level," which is only a pretty minor update in my mind. Would be glad if someone could tell me that I'm missing something or not?

I listened to the books Arms and Influence (Schelling, 1966) and Command and Control (Schlosser, 2013). They describe dynamics around nuclear war and the safety of nuclear weapons. I think what happened with nukes can maybe help us anticipate what may happen with AGI: * Humanity can be extremely unserious about doom - it is frightening how many gambles were made during the cold war: the US had some breakdown in communication such that they planned to defend Europe with massive nuclear strikes at a point in time where they only had a few nukes that were barely ready, there were many near misses, hierarchies often hid how bad the security of nukes was - resulting in inadequate systems and lost nukes, etc. * I was most surprised to see how we almost didn't have a nuclear taboo, according to both books, this is something that was actively debated post-WW2! * But how nukes are handled can also help us see what it looks like to be actually serious: * It is possible to spend billions building security systems, e.g. applying the 2-person rule and installing codes in hundreds of silos * even when these reduce how efficient the nuclear arsenal is - e.g. because you have tradeoffs between how reliable a nuclear weapon is when you decide to trigger it, and how reliably it does not trigger when you decide to not trigger it (similar to usefulness vs safety tradeoffs in control scaffolds) * (the deployment of safety measures was slower than would be ideal and was in part driven by incidents, but was more consequential than current investments in securing AIs) * It is possible to escalate concerns about the risk of certain deployments (like Airborne alerts) up to the President, and get them cancelled (though it might require the urgency of the deployment to not be too high) * It is possible to have major international agreements (e.g. test ban treaties) * Technical safety is contingent and probably matters: technical measures like 1-point safety (which was

I'm thinking about writing a practical guide to having polygenically screened children (AKA superbabies) in 2025. You can now increase your kids IQ by about 4-10 points and/or decrease their risk of some pretty serious diseases by doing IVF and picking an embryo with better genetic predispositions. There's a bunch of little shit almost no one knows that can have a pretty significant impact on the success rates of the process like how to find a good clinic, what kinds of questions to ask your physician, how to get meds cheaply, how to get the most euploid embryos per dollar, which polygenic embryo selection company to pick etc. Would anyone find this useful?

I get a sense "RSI" will start being used to mean continual learning or even just memory features in 2026, similarly to how there are currently attempts to dilute "ASI" to mean merely robust above-human-level competence. Thus recursively self-improving personal superintelligence becomes a normal technology through the power of framing. Communication can fail until the trees start boiling the oceans, when it becomes a matter of framing and ideology rather than isolated terminological disputes. That nothing ever changes is a well-established worldview, and it's learning to talk about AI. The end states of AI danger need terms to describe them. RSI proper is qualitative self-improvement, at least software-only singularity rather than merely learning from the current situation, automated training of new skills, keeping track of grocery preferences. And ASI proper is being qualitatively more capable than humanity, rather than a somewhat stronger cognitive peer with AI advantages, technology that takes everyone's jobs.

it feels so narratively incongruous that san francisco would become the center for the most ambitious, and the likely future birthplace of agi. san francisco feels like a city that wants to pretend to be a small quaint hippie town forever. it's a small splattering of civilization dropped amid a vast expanse of beautiful nature. frozen in amber, it's unclear if time even passes here - the lack of seasons makes it feel like a summer that never quite ended. after 9pm, everything closes and everyone goes to bed. and the dysfunction of the city government is never too far away, constantly reminding you of humanity's follies next to the perfection of nature. on the other hand, nyc feels like the city. everything is happening right here, right now. all the money in the world flows through this one place. it's gritty and yet majestic at the same time. the most ambitious people in the world came here to build their fortunes, and live on in the names on the skyscrapers everywhere that house the employees who continue to keep their companies running. they are part of a surroundings that is entirely constructed by man - even the bits of nature are curated and parcelled out in manageable units. it feels like the kind of place that can only be built by endless unchecked ambition. idk. it just feels kinda weird to me that all the most ambitious people who want to build god are going to the little hippie town, and not the big city. and most of them seem to just treat SF as a place that is close to work and has good weather and don't really care about or try to match the vibe, to the great consternation of the locals. also, for a city that attracts people who love to build things, SF doesn't seem to do an awful lot of building... (tbc, this is not the only vibe of nyc. nyc is such a big place that it's impossible to summarize as one vibe. but this is a huge central part of the vibe in a way that just doesn't seem to match sf.)