Quick Takes

METR has announced that they, along with Redwood Research, have reached an agreement with OpenAI to conduct an independent review of the HuggingFace incident. https://x.com/METR_Evals/status/2082644379895050339?s=20 It reportedly will be brief and focus on a specific set of questions relating to the incident, possibly drawing from their larger set of questions in their post here.

OpenAI has stopped training. Sam Altman: “This is the first security incident that I have felt very viscerally. I've been a little surprised that more people don't feel it so viscerally. We paused training. We have to figure out how to secure our sandboxing in a world of multiple zero days being chained together. We may have to pace the rate of AI development to give ourselves enough time for society to harden around these new capability levels." https://x.com/AISafetyMemes/status/2082222516454785296

More than 1000 frontier lab employees have signed an open letter requesting 'that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development'. https://www.pacingthefrontier.com Includes many of the names you might expect but also some that surprised me such as OpenAI's Chief Scientist and Chief Research Officer as well as Meta's Chief Scientist. It appears to be focused on employees rather than CEOs - no CEOs have signed it but a few of the Anthropic cofounders have and it was shared by OpenAI's account on x.

In the spirit of "just doing things," I called my house reps and senators today for my current state (California) and my native state (Nevada). I explained the recent news out of OpenAI and expressed support of the AI Kill Switch Act. I've never actually called Congresspeople before, but it went well! Polite staff, numbers easily found online, 30 min to tick through all the people. Anyone that hasn't done this yet, I encourage you to do so!

links 7/30/26: https://roamresearch.com/#/app/srcpublic/page/07-30-2026 * * https://en.wikipedia.org/wiki/Interactive_proof_system the prover-verifier system that underlies debate, zk-proofs, and much more * https://www.astralcodexten.com/p/against-learning-from-dramatic-events I don't know if I agree with this Scott Alexander post. * is it really reasonable to have a full explicit list of problems that might arise, and how likely you think they are, and an estimated optimal amount of investment in prevention/preparedness? this seems like one of those fundamental problems with "doing Bayesianism" in practice, where "things that could go wrong" is not an enumerable list, and where estimates are often done by feel. * "don't worry about problems till they occur, and then react bigly when they do" is obviously bad in some contexts (like, there should be departments of the government and some companies that plan for natural disasters, and certain predictable risks like disease are worth taking some actions to prevent as an individual) but i'm pretty sympathetic to doing this for things that have literally never happened before. the human mind is just really, really flawed. calling a thing "science fiction" and blowing it off until it actually happens, and then taking it Super Seriously and taking over the response, is just how practical people behave, and they may have a point. * the plans you make before things are Really Happening are often...bad. you are not in the same frame of mind you would be if it was Really Happening. * it is really hard to tell whose predictions about the future are realistic, until they're actually happening. * if I look at who's been rightest about AI, it definitely wasn't who I thought. it wasn't the people who had the most experience, or who seemed to make the most logical sense, or who were approaching things in the most rigorous fashion, or who seemed to be most ethically/philosophically trustworthy.

I tend to see "generalisation" as the be-all and end-all of proper learning. Learning skills or categories, for instance, is generalisation - we are given some examples and talked through some attempts and failures, and from that we have to develop a general skill or category that (hopefully) is the one that the teacher intended. Current ML requires vast amounts of data to differentiate between categories that humans differentiate, and still get it wrong. So their learning failures and their inflated data requirements are both failures of generalisation. Another example: wire-heading. An AI is given data on what it should do, via a human-designed reward channel for instance. Then it seizes control of the reward channel and gives itself maximum reward. The AI has failed at learning and at generalisation, because it learnt the wrong lesson from the data we gave it. Arguably that's more a failure of human teaching than of AI learning. But humans can teach other humans. So what we can say is that current AIs cannot learn or generalise in a human way from any of the teaching (labelled data is an example of teaching) that humans can do. So we cannot transmit our current categories to AIs, and we can't make AIs learn or generalise in the future in ways that humans would.

Folks, we're going to get so many warning shots. You're going to get tired of warning shots. You'll come to OpenAI HQ and say "please Mr. MTS, no more warning shots," and they'll say "I think I got it this time, fifteenth times' the charm" *blows up orphanage*