I am just sharing a quick take on something that came to mind that occured earlier this year that I had forgotten about. I just received a domain renewal for a project that was dead in the water but should have been alive, and was unilaterally killed by the AWS Trust and Safety team in gaslighting. It is a bit too late, but I still think it is important for people to know. Earlier this year (2025), a software engineering friend of mine had a great idea of creating a tool to pin on a map interface any ICE (the United States Immigration and Customs Enforcement) sightings from the public. I vibe coded it last February with Claude Sonnet 3.7, and put it up on AWS with a public facing domain. From a coding standpoint, it was a fun project because it was my first project that Terraformed a production app using Claude, autonomously. I had a website up that worked on desktop and mobile. From a public trust and safety standpoint, it helped by adding public accountability and mitigating overreach by a rogue, and potentially in some circumstances operating illegaly, agency. It was overall A Good Thing™. However, what happened next was upsetting and shocking to me. A day after the website was put online, without me advertising it to ANYONE, the website became inaccessible to almost all browsers. It would simply show a giant red background saying this is associated with known criminals, etc., and has been blocked for safety. It is some Chrome / Google maintained blacklist mechanism that looks even scarier and more severe than the "https / http" / cert issue mismatch, and that cannot be overridden. To be clear, I had registered a certificate using AWS Certificate Manager, and there was no SSL issue. Instead, the domain had been unilaterally marked as "dangerous" by AWS (or Google, Chrome, whomever) one day after it was made publicly accessible, despite no advertising / attention. I did all of this on my personal AWS account. The very next day, I received a scary email from AWS

I often see people advocate others sacrifice their souls. People often justify lying, political violence, coverups of “your side’s” crimes and misdeeds, or professional misconduct of government officials and journalists, because their cause is sufficiently True and Just. I’m overall skeptical of this entire class of arguments. This is not because I intrinsically value “clean hands” or seeming good over actual good outcomes. Nor is it because I have a sort of magical thinking common in movies, where things miraculously work out well if you just ignore tradeoffs. Rather, it’s because I think the empirical consequences of deception, violence, criminal activity, and other norm violations are often (not always) quite bad, and people aren’t smart or wise enough to tell the exceptions apart from the general case, especially when they’re ideologically and emotionally compromised, as is often the case. Instead, I think it often helps to be interpersonally nice, conduct yourself with honor, and overall be true to your internal and/or society-wide notions of ethics and integrity. I’m especially skeptical of galaxy-brained positions where to be a hard-nosed consequentialist or whatever, you are supposed to do a specific and concrete Hard Thing (usually involving harming innocents) to achieve some large, underspecified, and far-off positive outcome. I think it's like those thought experiments about torturing a terrorist (or a terrorist's child) to find the location of the a ticking nuclear bomb under Manhattan where somehow you know the torture would do it. I mean, sure, if presented that way I'd think it's a good idea but has anybody here checked the literature on the reliability of evidence extracted under torture? Is that really the most effective interrogation technique? So many people seem eager to rush to sell their souls, without first checking to see if the Devil’s willing to fulfill his end of the bargain. (x-posted from Substack)

maybe I should host an antechamber/arena house party: one chill cozy room with soothing music where no arguing is allowed and people are strongly encouraged to say kind things and reflect on things they're grateful for and whatnot, and another with harsh fluorescent lights and agitating music and a big whiteboard full of hot takes and the conversations all get transcribed by speech to text and posted on lesswrong in real time. and guests are given a heart rate monitor that beeps if their HR gets too high, forcing them to spend a few minutes in the chill room before returning to the arena

It has been less than one year since I posted my model of what is going on with LLMs: https://www.lesswrong.com/posts/vvgND6aLjuDR6QzDF/my-model-of-what-is-going-on-with-llms It already does not seem to hold up very well. I underestimated how far reasoning models based on CoT could scale. Generally, I’m now more convinced that labs can consistently convert money into capabilities by training transformers. In particular, I was previously very skeptical that RL would “just happen” to be solved with the same techniques as sequence prediction / text generation. It does seem that sequential decision making (eg Pokémon) continues to be the biggest weakness (by far relative to other capabilities), and agents are not very useful outside of coding, but this may be a transient issue with perception or just training focus (though I still think it’s not clear, I would bet that way).  I think progress is very hard to predict and trying is not very constructive (at least for me). In particular, I’ve updated that my area of expertise (eg AIXI) may not be very helpful for making such predictions, as compared to more direct experience with models. Also, I think perhaps my reasoning has been corrupted by optimism which prevented me from viewing the labs as competent adversaries (equipped to overcome many of the obstacles I hoped would stop them). There still have not been very convincing original insights from LLMs, and it’s possible that I’m updating too far and early-2025 me will turn out to be more right than late-2025 me. One problem is that there really is a great deal of hype / advertising around LLMs, and incentives to blow up the significance of their accomplishments so it’s hard to say exactly how good they are at things like math except through first-hand experience (until they start one-shotting seriously hard problems). But even from first-hand experience alone, IF I HAD TO BET, I’d say the writing is on the wall.

I am pretty confused about people who have been around the AI safety ecosystem for a while updating towards "alignment is actually likely by default using RLHF" But maybe I am missing something. Like 3 years ago, it was pretty obvious that scaling was going to make RLHF "work" or "seem to work" more effectively for a decent amount of time. And probably for quite a long time. Then the risk is that later you get alignment-faking during RLHF training, or at the extreme-end gradient-hacking, or just that your value function is misspecified and comes apart at the tails (as seems pretty likely with current reward functions). Okay, there are other options but it seems like basically all of these were ~understood at the time. Yet, as we've continued to scale and models like Opus 3 have come out, people have seemed to update towards "actually maybe RLHF just does work," because they have seen RLHF "seem to work". But this was totally predictable 3 years ago, no? I think I actually did predict something like this happening, but I only really expected it to affect "normies" and "people who start to take notice of AI at about this time." Don't get me wrong, the fact that RLHF is still working is a positive update for me, but not a massive one, because it was priced in that it would work for quite a while. Am I missing something that makes "RLHF seems to work" a rational thing to update on?  I mean there have been developments to how RLHF/RLAIF/Constitutional AI works but nothing super fundamental or anything, afaik? So surely your beliefs should be basically the same as they were 3 years ago, plus the observation "RLHF still appears to work at this capability level," which is only a pretty minor update in my mind. Would be glad if someone could tell me that I'm missing something or not?

I listened to the books Arms and Influence (Schelling, 1966) and Command and Control (Schlosser, 2013). They describe dynamics around nuclear war and the safety of nuclear weapons. I think what happened with nukes can maybe help us anticipate what may happen with AGI: * Humanity can be extremely unserious about doom - it is frightening how many gambles were made during the cold war: the US had some breakdown in communication such that they planned to defend Europe with massive nuclear strikes at a point in time where they only had a few nukes that were barely ready, there were many near misses, hierarchies often hid how bad the security of nukes was - resulting in inadequate systems and lost nukes, etc. * I was most surprised to see how we almost didn't have a nuclear taboo, according to both books, this is something that was actively debated post-WW2! * But how nukes are handled can also help us see what it looks like to be actually serious: * It is possible to spend billions building security systems, e.g. applying the 2-person rule and installing codes in hundreds of silos * even when these reduce how efficient the nuclear arsenal is - e.g. because you have tradeoffs between how reliable a nuclear weapon is when you decide to trigger it, and how reliably it does not trigger when you decide to not trigger it (similar to usefulness vs safety tradeoffs in control scaffolds) * (the deployment of safety measures was slower than would be ideal and was in part driven by incidents, but was more consequential than current investments in securing AIs) * It is possible to escalate concerns about the risk of certain deployments (like Airborne alerts) up to the President, and get them cancelled (though it might require the urgency of the deployment to not be too high) * It is possible to have major international agreements (e.g. test ban treaties) * Technical safety is contingent and probably matters: technical measures like 1-point safety (which was

I was recently looking at the Astra Fellowship program, and found myself noticing skepticism that much value would come out of their "Strategy" category of research. In order to channel that skepticism productively, I want to talk a bit about how I think people might do valuable strategy research. The most important thing is something like: trying to orient to the possibility of big changes in our ontologies going forward. What I mean by "strategy" is in significant part "ideas and plans that are robust to paradigm shifts". Having said that, there's no way to do strategic thinking without relying on some part of your existing ontology. So the thing I'm advocating for is more like "keep track of the fields where paradigm shifts seem most plausible, and actually try to picture what it would be like for that part of your ontology to radically improve". Of course answering questions like these is extremely difficult because ontology improvements are anti-inductive: if you could picture them then you would have already improved your ontology. One strategy for getting around that is to use analogies—e.g. "imagine a future where we understand social sciences as well as we currently understand natural sciences". And if you flesh out these analogies enough, I expect that in doing so you'll actually help push the frontier towards that paradigm shift. (E.g. if you thoroughly flesh out a world where we've solved alignment as well as we've solved problem X or problem Y, I expect that you'll discover some promising alignment research directions.) To be more concrete, here are the three frontiers that I'm currently tracking as places where there's a lot of room for radically better ontologies that transform how we think and act: 1. Understanding intelligence. 1. I say "intelligence" because that's a better pointer to the frontier I'm interested in than "AI" or "ML". "Agent foundations" is another good term, but I also mean to include groups that are studying intelligence