I often see people advocate others sacrifice their souls. People often justify lying, political violence, coverups of “your side’s” crimes and misdeeds, or professional misconduct of government officials and journalists, because their cause is sufficiently True and Just. I’m overall skeptical of this entire class of arguments. This is not because I intrinsically value “clean hands” or seeming good over actual good outcomes. Nor is it because I have a sort of magical thinking common in movies, where things miraculously work out well if you just ignore tradeoffs. Rather, it’s because I think the empirical consequences of deception, violence, criminal activity, and other norm violations are often (not always) quite bad, and people aren’t smart or wise enough to tell the exceptions apart from the general case, especially when they’re ideologically and emotionally compromised, as is often the case. Instead, I think it often helps to be interpersonally nice, conduct yourself with honor, and overall be true to your internal and/or society-wide notions of ethics and integrity. I’m especially skeptical of galaxy-brained positions where to be a hard-nosed consequentialist or whatever, you are supposed to do a specific and concrete Hard Thing (usually involving harming innocents) to achieve some large, underspecified, and far-off positive outcome. I think it's like those thought experiments about torturing a terrorist (or a terrorist's child) to find the location of the a ticking nuclear bomb under Manhattan where somehow you know the torture would do it. I mean, sure, if presented that way I'd think it's a good idea but has anybody here checked the literature on the reliability of evidence extracted under torture? Is that really the most effective interrogation technique? So many people seem eager to rush to sell their souls, without first checking to see if the Devil’s willing to fulfill his end of the bargain. (x-posted from Substack)

It has been less than one year since I posted my model of what is going on with LLMs: https://www.lesswrong.com/posts/vvgND6aLjuDR6QzDF/my-model-of-what-is-going-on-with-llms It already does not seem to hold up very well. I underestimated how far reasoning models based on CoT could scale. Generally, I’m now more convinced that labs can consistently convert money into capabilities by training transformers. In particular, I was previously very skeptical that RL would “just happen” to be solved with the same techniques as sequence prediction / text generation. It does seem that sequential decision making (eg Pokémon) continues to be the biggest weakness (by far relative to other capabilities), and agents are not very useful outside of coding, but this may be a transient issue with perception or just training focus (though I still think it’s not clear, I would bet that way).  I think progress is very hard to predict and trying is not very constructive (at least for me). In particular, I’ve updated that my area of expertise (eg AIXI) may not be very helpful for making such predictions, as compared to more direct experience with models. Also, I think perhaps my reasoning has been corrupted by optimism which prevented me from viewing the labs as competent adversaries (equipped to overcome many of the obstacles I hoped would stop them). There still have not been very convincing original insights from LLMs, and it’s possible that I’m updating too far and early-2025 me will turn out to be more right than late-2025 me. One problem is that there really is a great deal of hype / advertising around LLMs, and incentives to blow up the significance of their accomplishments so it’s hard to say exactly how good they are at things like math except through first-hand experience (until they start one-shotting seriously hard problems). But even from first-hand experience alone, IF I HAD TO BET, I’d say the writing is on the wall.

I am pretty confused about people who have been around the AI safety ecosystem for a while updating towards "alignment is actually likely by default using RLHF" But maybe I am missing something. Like 3 years ago, it was pretty obvious that scaling was going to make RLHF "work" or "seem to work" more effectively for a decent amount of time. And probably for quite a long time. Then the risk is that later you get alignment-faking during RLHF training, or at the extreme-end gradient-hacking, or just that your value function is misspecified and comes apart at the tails (as seems pretty likely with current reward functions). Okay, there are other options but it seems like basically all of these were ~understood at the time. Yet, as we've continued to scale and models like Opus 3 have come out, people have seemed to update towards "actually maybe RLHF just does work," because they have seen RLHF "seem to work". But this was totally predictable 3 years ago, no? I think I actually did predict something like this happening, but I only really expected it to affect "normies" and "people who start to take notice of AI at about this time." Don't get me wrong, the fact that RLHF is still working is a positive update for me, but not a massive one, because it was priced in that it would work for quite a while. Am I missing something that makes "RLHF seems to work" a rational thing to update on?  I mean there have been developments to how RLHF/RLAIF/Constitutional AI works but nothing super fundamental or anything, afaik? So surely your beliefs should be basically the same as they were 3 years ago, plus the observation "RLHF still appears to work at this capability level," which is only a pretty minor update in my mind. Would be glad if someone could tell me that I'm missing something or not?

on the one hand, it is a desirable feature of an intellectual community to be truth seeking, and while it can be deeply emotionally painful to part ways with deeply held beliefs, in the long run it's better to tear off the bandage. on the other hand, being emotionally hurt all the time by your community kind of fucking sucks, and isn't very good for long term emotional or epistemic health. perhaps a middle ground is in order: intellectual communities should be partitioned into an arena, where every idea is to be exposed to the harsh light of truth, and an antechamber, where you can rest and be surrounded by positivity and develop ideas in a supportive environment.  both are necessary - we need a way to kill bad ideas, because an environment that refuses to discard bad ideas because they are emotionally load bearing is doomed to epistemic ruin. but also the best weird ideas often sound bad initially, and require a safe environment to develop; and we are all human, and our emotional well being and desire to belong to a community is essential. by visibly separating the two, we might be able to get the best of both worlds. this is not a crazy idea - many other parts of society have analogous things. for example, people who play sports for fun with their friends compete to win while on the field, but this only brings them closer off the field. I think one common criticism of LW ist it is too much of an arena, and not enough of an antechamber. perhaps this can be fixed somehow.

I'm thinking about writing a practical guide to having polygenically screened children (AKA superbabies) in 2025. You can now increase your kids IQ by about 4-10 points and/or decrease their risk of some pretty serious diseases by doing IVF and picking an embryo with better genetic predispositions. There's a bunch of little shit almost no one knows that can have a pretty significant impact on the success rates of the process like how to find a good clinic, what kinds of questions to ask your physician, how to get meds cheaply, how to get the most euploid embryos per dollar, which polygenic embryo selection company to pick etc. Would anyone find this useful?

The AI company Mechanize posted a blog post in November called "Unfalsifiable stories of doom" which is a high-quality critique of AI doom and IABIED that I haven't seen shared or discussed on LessWrong yet. Link: https://www.mechanize.work/blog/unfalsifiable-stories-of-doom/ AI summary: the evolution→gradient descent analogy breaks down because GD has fine-grained parameter control unlike evolution's coarse genomic selection; the "alien drives" evidence (Claude scheming, Grok antisemitism, etc.) actually shows human-like failure modes; and the "one try" framing ignores that iterative safety improvements have empirically worked (GPT-1→GPT-5).

I listened to the books Arms and Influence (Schelling, 1966) and Command and Control (Schlosser, 2013). They describe dynamics around nuclear war and the safety of nuclear weapons. I think what happened with nukes can maybe help us anticipate what may happen with AGI: * Humanity can be extremely unserious about doom - it is frightening how many gambles were made during the cold war: the US had some breakdown in communication such that they planned to defend Europe with massive nuclear strikes at a point in time where they only had a few nukes that were barely ready, there were many near misses, hierarchies often hid how bad the security of nukes was - resulting in inadequate systems and lost nukes, etc. * I was most surprised to see how we almost didn't have a nuclear taboo, according to both books, this is something that was actively debated post-WW2! * But how nukes are handled can also help us see what it looks like to be actually serious: * It is possible to spend billions building security systems, e.g. applying the 2-person rule and installing codes in hundreds of silos * even when these reduce how efficient the nuclear arsenal is - e.g. because you have tradeoffs between how reliable a nuclear weapon is when you decide to trigger it, and how reliably it does not trigger when you decide to not trigger it (similar to usefulness vs safety tradeoffs in control scaffolds) * (the deployment of safety measures was slower than would be ideal and was in part driven by incidents, but was more consequential than current investments in securing AIs) * It is possible to escalate concerns about the risk of certain deployments (like Airborne alerts) up to the President, and get them cancelled (though it might require the urgency of the deployment to not be too high) * It is possible to have major international agreements (e.g. test ban treaties) * Technical safety is contingent and probably matters: technical measures like 1-point safety (which was