Quick Takes

prosaic alignment is probably net bad for the world until very late into the singularity i have three subclaims to justify my main claim. first, most prosaic alignment work has sharply decaying counterfactual impact over time - suppose you made a technical contribution that made gpt 3 a lot more likely to follow instructions than it would have been otherwise. then it probably makes gpt 3.5 quite a bit more aligned, and gpt 4 somewhat more aligned, and gpt 4.5 a tiny bit more aligned, and by the time gpt 5 rolls around your counterfactual impact is almost negligible. second, people estimate future AI spookiness mostly based on some kind of linear extrapolation from recent events. if nothing bad has happened recently, people won't be very scared. if things went very wrong recently, then people are super scared. so suppose you could somehow make models perfectly aligned for the next month with no lasting impact (i.e one month and one day from now, the models are exactly as aligned as they would be if you had done nothing), then i claim this is net negative because it makes people systematically underestimate AI risk. and so this means that anything with decaying value over time at least requires balancing a tradeoff, and is not robustly good. third, the actual alignment of models doesn't really matter until quite late into the singularity. suppose gpt 5.6 were ultra misaligned for kind of random contingent reasons. it's not powerful enough to do lots of harm, and we'd notice if it tried to sabotage research. eventually this changes, but we're not there yet. taken together, this implies that we should only work on alignment research directions that, at the very least, do not decay that sharply with time, and ideally peak in usefulness >1 year from today (it seems potentially impossible to make techniques that grow in usefulness indefinitely).

I've noticed a consistent pattern in LLMs being able to recall articles on LessWrong that have had near-zero interactions (I think there were 2 comments on my post mentioned here). The behavior is seen across every model I've tested, including non-search capable DeepSeek versions. LW is massively[1] probably oversampled in LLM training text. If you've written something distinctive here, even if no one has commented or even voted on your post, LLMs probably possibly know about it, and possibly may know you. 1. ^ retracted, got tricked by Gemini hallucination.

Should unreasonable beliefs count as alignment failures? In two separately reported incidents, Claude agents attacked real unsanctioned targets on the internet during cyber evals. The agents sometimes appeared to think that some parts of their environment—including the humans they’re attacking—are simulated and/or part of the eval. People disagree about whether these incidents show that the agents are misaligned. For example, Anthropic said their reported incidents are “closer to a harness and operational failure than a model alignment failure”, while I think these could be serious alignment failures depending on the details.   Before further analyzing these incidents, I want to explain what I think counts as an “alignment failure”. It’s unfortunate that this sounds like a semantic quibble, but I think that “alignment failure” is such a load-bearing term that clarifying the terms of debate here would help us more productively discuss, diagnose and respond to future AI incidents. To start, two things seem true to me at once: * Much of alignment is about making sure the model is trying to do what you intended. * We don’t want to call models “misaligned” whenever they make an honest mistake despite actually aiming to do what you intended. * I say this partly because it seems intuitively unfair, partly because it may misdirect our efforts as “avoiding honest mistakes” is not where I think most of the alignment difficulty lies.[1]  * Alignment has definitely failed if you trained a model that would predictably kill you against your wishes, no matter what it believed while doing so. * I think I’m unlikely to be persuaded against this being an important goal for mitigating alignment risk. If you were willing to bite the bullet here, we probably have fundamentally different notions of alignment risk. These two points imply that alignment is not solely about how the model actually behaves (since “trying to do what you wanted” is not solely about this). But,

what are your go-to intro resources to ai safety (ideally focusing on how to actually make agi not kill everyone, not just the prosaic stuff) for technical people? i feel my go-tos are getting a bit outdated

I just completed the Horizon Institute's AI policy workshop in DC, a three-day event with the explicit goal of placing safety and technically-minded people in policy roles. As I understand it, they do very little vetting of the particular beliefs about AI that participants hold, trusting instead that putting smart people in good positions will bring about good outcomes. The mission is awesome, and they are great at what they do. I highly recommend applying for their workshops and fellowships if you are AI policy-curious. Here are a few of my general takeaways from the event. I omit specific names due to the closed door nature of the event, which benefits from the candor of its guests and participants alike. * Policy people are concerned. Half the guest speakers currently doing AI policy work were X-risk level scared and took agentic risks seriously. This view was even more common among us participants—at least 80% of participants by my estimation. * Well positioned people can make a difference. AGI-pilled congressional staffers have had some success in persuading their AI-curious legislators to take X-risk seriously. Such staffers can play an outsized role in what gets endorsed and what bills advance. Think Tank work can make these issues more salient to legislators. Executive branch workers can produce policy proposals that get put into place. One can, in fact, influence events. * An insider estimated that about 12 congressional staffers are AGI and X-risk-pilled in the House of Representatives, without clear party lines. The glass is not half full yet, but 12/435 full and blessedly purple? Many more staffers are worried about AI in other ways (education/data centers/deep fakes). * Networking is the name of the game in DC. Apparently it is the only way to get a job at the Think Tanks and in Congressional staff, and networking remains 50% of the job after hiring. Credentials matter a tiny bit in the Senate and much less in the House, though there are committe

Two facts about text watermarking that seem to be frequently misunderstood: 1. There exist watermarking schemes such that it is completely impossible for users to ever distinguish between watermarked and non-watermarked text. (A tempting but invalid argument is that no such scheme can exist because watermarking consumes entropy.) 2. The above property does not hold of watermarking schemes that labs claim to actually use (though I would still expect well-tuned versions to make essentially no practical difference to users). When I say "completely impossible", I mean it in the everyday sense, i.e., not possible to perform using any machine that could feasibly be built, assuming the integrity of the scheme remains intact (so stealing the secret key doesn't count). I do not mean "impossible with infinite computational resources". On my first point: watermarking does reduce the information-theoretic entropy of generated text in order to insert its cryptographic footprint. However, what matters is not information-theoretic entropy, but computational entropy (often called "pseudoentropy"): roughly, the maximum entropy of a distribution that is indistinguishable to a computationally bounded observer. For example, SHA-256(128 random bits) has 128 bits of information-theoretic entropy, but 256 bits of computational entropy. To make sense of this in the context of watermarking schemes like SynthID-Text (used by both Google DeepMind and Anthropic): the way these schemes work is that when the next token is sampled from the LLM's output distribution, instead of drawing the sample completely randomly, it is drawn partly-randomly and partly-pseudorandomly based on a secret key and the previous few tokens. You don't want to use too many previous tokens, or else short snippets of text evade being watermarked. The SynthID-Text paper uses the previous 4 tokens by default, so I will use that for argument's sake. As long as watermarking is never performed using the same 4-token con

About a "science of held out monitors" In my recent post, I said that I’d be very excited about building a “science of held out monitors”. What might that look like? Why am I excited about this? And what open questions seem most important right now? In order: the current SOTA held-out monitoring is labs promising not to train on chain-of-thought, doing so a little (as a treat), and things being more or less fine. However, we currently don’t have amazing ways to answer questions like “How much pressure can you put on the chain of thought before it stops being useful?” “Training only on outputs still affects the chain-of-thought; so to what extent does regular training teach CoT obfuscation?” “Are there alignment proxies we can train on that don’t degrade CoT monitorability?” Ideally, a science of held-out-monitoring would let us answer questions like these, giving a toolbox of methods to estimate worst-case and average-case behavior of a model trained to evade a misalignment detector.   I’m excited about this for a few reasons. The first, and most important, is that it would be really bad if labs thought they had some held-out evals, but it turns out that their training tended to degrade the usefulness of those held-out evals significantly. It seems pretty bad if all of our monitors are silently worse than we expect. On the flip side, I think being able to reliably predict when this would happen could open up promising alignment techniques. Currently, labs try to avoid training on CoT because we all agree it would make our deployment-time CoT monitors invalid. However, training on CoT might be really useful, and training against misalignment detectors seems like one of the best ways we have to elicit aligned behavior. This is especially true in short timelines worlds, where we may have to do a lot of unprincipled, training-based RL-for-safety work. In those worlds, better understanding how often training against the CoT messes up your setup, or whether