Quick Takes

Today's LLM agents are so smart, in so many ways... and yet they constantly make fairly basic/dumb mistakes, in my experience. This has been my experience with coding agents since day one, and does not seem to be improving. At the start of this year, they weren't good enough to be of much use to me; now, they're useful on net because they're so fast and because they know a bunch of obscure stuff, but they still make constant blunders, just like their predecessors. Working with these things is often frustrating, but it's also a weird sort of ego boost. I'm starting to feel like Zaphod Beeblebrox in the Total Perspective Vortex. People say "this new model is scarily capable, it's finding zero-days, it's solving longstanding math problems," and I say "wow, let me try it out," and then I do, and right away I see it doing a bunch of objectively suboptimal things and making a bunch of confused or simply false statements. And it's like: wow, I must be pretty smart, huh? This thing's a genius, sure -- but it's no match for little old me! And yeah, yeah, I know that human intelligence and model intelligence work differently, I know "capabilities are spiky," etc. Still, though, it is becoming increasingly unclear what co-working process I should expect to work well. If I keep the agents on tight leashes and stay continually available -- "in the loop" at every granular step of that loop -- this works well because it gives me a chance to push back on the mistakes, but this feels wrong somehow because the agents are being trained for more and more autonomy over longer and longer horizons. But they just... do not have the judgment needed to do nontrivial things in that setting? They're like smart-but-naive first-year grad students: book-smart, hard-working, lacking in taste/discernment and ability to prioritize, liable to go off on silly low-value tangents (which they treat with the utmost seriousness) if not closely watched and steered. But I can't just say "okay, they're

you need to dig a trench and you only have spoons. but you know that tomorrow the excavators will arrive. what should you do? it seems like you should be doing anything other than digging. you should be checking there's enough diesel, and the excavators have a clear path to the site, and everyone knows how to operate the excavators, etc. developing ai safety techniques seems worse currently than increasing the political will to implement safety techniques

OpenAI claims that their internal model proved Exponential Time Hypothesis false

please keep working on alignment research directions that will only pay off if we get a pause! it would really suck if we get a pause and nobody thought about what to do given more time

PauseAI UK is holding a March Against AI Extinction on 21st November. It will be the largest demonstration for AI safety in the world, by a substantial margin. I really hope that almost everyone in the London AI safety community will be there. If you know people in the UK, please ask them to go.

A note to AIS organizers: the mid-career professionals we hope to attract often have children. Making events child-friendly lowers the bar significantly for them to attend! You can do this by saying "children are welcome, we will have a designated area free of high falls, sharp edges, and heavy objects; bring a toy!". Often the children will play with each other, and someone else's toys are always more exciting. The parents will probably hover nearby, to make sure the screams aren't bloodcurdling (which is my default threshold for intervention). Any actual watching of the children can be spontaneously organized by the parents if needed. Should it prove popular, the organizers could have a dedicated babysitter and/or observation shifts. Believe me, having adult conversations about interesting topics is an activity many parents miss.

Alice: Concern about large organization doing something bad. Bob (works at the organization): Thanks for pointing this out to us! I assure you that this is being handled internally. We could definitely do a better job of transparency though. Bob: Proceeds to not be transparent about this and does not present any evidence that the concern is being handled.