Quick Takes

Given the recent news events (like Pax Silica pushing for exclusivity and barring countries from joining WAICO), I've grown increasingly nervous about previous AI comms strategies. Everybody knows that technical ai safety is a very fraught field and a lot of empirical AI safety research carries great uncertainty over whether you are helping or harming, but I suspect the same is true of AI Governance. Looking at previous successful statements, AI 2027 was highly impactful but it also strongly framed AI as a national security issue with a race against China and I suspect this was partially deliberate to get the government to take AI seriously. But given recent news, I'm not actually sure framing it as a national security issue was correct. On the other hand, AI 2040 got much less attention, so maybe the theory that you have to frame it that way to make an impact is correct actually? It's hard to say. Thinking about writing a full post with my rhoughts on this topic.

the no magic ood principle a lot of alignment proposals have a step where you have some kind of magic ood generalization. depending on the shape of the proposal this could be obvious or subtle. i think a desirable property for alignment proposals is to avoid having a magic ood step, or to make a strong case why the amount of magic is smaller than competing proposals, or to empirically test the magic ood and understand it deeply.

why does the arc white box challenge use random networks? it seems likely that real trained networks have structure that makes it possible to estimate that random networks don’t

Effectively multi-tasking with AIs seems to require a state of mind that is inimical to deep work / thinking. When I'm switching between AI panes I'm constantly distracted, disoriented, frazzled. I get tired out having to pick up and put down large amounts of context each time, etc. etc. I've observed that doing large volumes of work with AI coding agents mostly destroys my ability to do deep thinking for the rest of the day. I don't know what the right way is to resolve this. What's worked reasonably well for me so far is to do most of my deep thinking in the morning(s) when my mind is pretty fresh. And then do AI coding in the afternoon once I'm reasonably sure I've hit the limit on how much deep thinking I want to do that day. But this isn't really perfect either since it doesn't address the root cause. Tentatively I feel like the problem is that "deep work" and "AI work" exercise two very different modes of thinking, which are a bit hard to switch between. 1. The deep work exercises a low-bandwidth, single-threaded, highly sequential mode of thinking. Deep work requires emptiness and focus so I can get into flow state. Deep thinking in particular requires deliberately taking time and space for listening to my own curiosity, so I can hear the inner thoughts which are easily drowned out by background noise. 2. The AI work part exercises a high-bandwidth, multi-threaded, parallel mode of thinking. This requires loading and unloading context, making quick 80/20 decisions, communication, and constantly monitoring (and remembering to monitor!) the situation for updates. Though it seems very different, it somehow also drains the mental energy needed for doing deep work. Other observations. * "Deep work" vs "AI work" is a bit like "focusing on the big picture" vs "focusing on the day to day". In many successful companies, these roles are split across different people. A similar specialization of roles might be important for properly getting the benefits of both.

One metric I've been using for AI impressiveness is how much I could do if I was the only one in the world with them. Like, if you gave me Sonnet 5, and nobody else in the world has LLMs or know they exist, what's that look like? I'd look like an insanely good engineer, right? What kind of engineer knows every coding language on github, runs through features the sprint planning meeting thought would take a few days in half an hour, and can read thousands of lines of code in a minute? We agree in 2016 if you 'saturated' leetcode you were great, right? So. Imagine the date is August 15th, 2016. You, and only you, have access to Mythos. Everyone else in the world has only the machine learning tools of 2016. That's a crazy advantage, right? That is the kind of supertech that comic books are made of. You'd look like Dr Doom. The least ambitious, most ethical thing I can imagine doing in that situation is turning in a couple dozen bug bounties and taking a month off, then doing it again whenever rent was due. I can get a lot more ethical if I'm willing to be a little more ambitious. This insight brought to you by me spending two hours vibecoding a videogame this weekend, getting it to the point where it's fun, and thinking 'okay, how long would it have taken me to do this if I was actually writing all the code?'

https://www.astralcodexten.com/p/your-book-review-the-escape-artist Reading about the unwillingness of Jews and Allies to believe that the holocaust was real, while it was happening, and even in some cases while being herded into the camps, was interesting and depressing.

Sam Altman and Dario Amodei are the public faces of AI safety and that's poisoning the well Normies just smell a rat whenever those two bring up AI safety. I think when the AI safety community gets a word in edge-wise (like you're arguing with your friends or maybe a tweet goes viral), we usually say something like "it could kill everyone. but China could also kill everyone, so now more than ever we have to supercharge OpenAI and Anthropic." and we just sound like we're "one of them." I feel like I should be hearing "shut it down" about as often as I hear "broke containment" and that's not really happening, instead it's more like "let OpenAI cook." So a number of people are pretty soured on AI safety because of this.