Quick Takes

I've noticed that I really find it helpful when debates occur in the form of people arguing with each other in full blogposts. This sounds slightly hyperbolic, but I genuinely feel like that might be the best debate format ever invented, and I wish there was a lot more of it. (Comments are OK, but I think they have severe limitations for complex issues, because if you're wrong about something, it's often still quite easy to write a superficially-convincing comment without leaving much "attack surface".) Currently, I'd estimate that <5% of popular blogposts are direct replies to another post. I wish this number was, I don't know, 40%?

Some things I'm doing now to reduce the risk of software supply chain attacks: * I run all package manager installs through Socket Firewall (sfw npm install ...), which blocks known malicious packages and warns on various other problems. * Claude Code sessions run as a separate user[1], and I plan to move them all the way to a VM[2] in the near future. * Unfortunately, this user does need access to a lot of API keys including a GitHub token, but I limit their scope to the extent I can. * The risk of prompt injection seems to be relatively low in modern Claudes, so I'm not that worried about it. I do use agent-sanitizer in my custom harness though. * Claude has instructions to immediately notify me if malicious packages get installed anyway. * Claude does various supply chain checks on every dependency update PR from Dependabot. * I sometimes have Claude rewrite overly complex dependencies (too many transitive deps, written in a scary language like C). * I also use sfw on my own user account, and have removed passwordless docker and libvirt access, although I think if a supply chain attack hits my user account I'm probably doomed anyway. 1. ^ I use a custom web interface to make this convenient. 2. ^ I should have done this from day one, but agents need GPU access which is tricky to sandbox. Now that I have a custom GPU queue, my plan is to run normal agent sessions in a VM and expose a container with GPU access through gVisor.

On December 25, 2026, the movie Artificial about Sam Altman’s firing is set to come out. Can it somehow raise public awareness of AI risks? Is it possible to reach out to someone in the film industry who could help? Some possible ideas: * Preparing high-quality explainers like agy.fyi/agi.wtf for people who google “Artificial movie true story” after watching. Would likely also make sense to purchase some good domain names for this purpose (like openai.wtf or smth like that) * Timing op-eds, interviews and podcast appearances to the release * Reaching journalists who will review the film * Getting something into the film itself (e.g. a QR code in the credits linking to good resources). Probably too late, and no longer feasible. Or perhaps I’m mistaken?

I frequently hear on Reddit that model companies are dumbing models down while they are in use and new models are more capable on release day then later. To what extend is that just a conspiracy theory and to what extend does it match reality?

One common source of Illusion of Transparency problems is different senses of what's a normal range. The countercharm for this is using actual numbers and not relative comparisons. For instance, I talk to a lot of people who run conferences and meetups. Most of them anchor the size of an event off what they've seen or run. So someone might say "yeah, that was a really big convention last week!" to mean an EAG with almost two thousand people. Then someone else might say "yeah, the convention next week is looking like it's going to be a small one" and they're talking about DEF CON where twenty thousand would be a low turnout. For another example, I like hiking. Sometimes I say I'm going on a short hiking trip, and someone reasonably assumes I'll be back in a few hours, when in fact I'm bringing a tent and three or four days of food. But I'm not that into hiking; on the trail I regularly meet people who enjoy hiking the Long Trail (~270 miles) as a short backpacking trip because it's not the Appalachian Trail! (~2,200 miles.)

I'm worried that people won't reflect well and so (1) they'll decide to use the cosmic endowment for something mediocre and/or (2) they'll have a catastrophic failure in some of {decision theory, bargaining, dealing with threats, reasoning about simulators, acausal trade (including ECL etc.)}. Some angles: * Current AI systems are way better at science/engineering than philosophy. Maybe even ASI won't solve philosophy (in particular, maybe there's not a sufficiently good training signal to train super-philosophers via ML). * By default almost everyone is and will be overconfident in their intuitions on philosophy/metaethics/axiology. By default almost nobody will "suspect that they are confused about morality, and [be] eager or willing to apply reason and deliberation to find out what their real values are, or to correct their moral beliefs." * Long horizon agency / strategic competence approximately does not exist among humans, even the smartest ones. And maybe it's necessary. * It may be natural for powerful people to use AIs for instrumental stuff (acquiring power, inventing technology, building industrial capacity) but not terminal stuff (axiology, etc.). Some angles: * If they asked an enlightened philosophy ASI about its views and it replied straightforwardly, they would think it sounded crazy. * In general the ASIs will say crazy-sounding things about cosmology, religion, etc. * They will have seen other people "reflect" with "help" from ASIs and they think those people now have crazy views, talk about living in a simulation, etc. and they don't want to end up like that. * @habryka probably disagrees. Idk, it's so easy for me to imagine that Elon (or almost anyone else) would listen/defer to AI on the instrumental stuff but not reflect on philosophy/axiology. * Maybe there will have been a period where superpersuaders went around getting people to adopt new ideologies—maybe even by saying true but filtered things—and so people develop stro

So, the Hackening. I have yet to hear any sensible-sounding takes on what to do about it, at a personal level. Mostly people repeat the usual shibboleths about refreshing your passwords, which doesn't seem to engage at all with any model of the new threat regime. Anybody have actual models about what the new threats will look like, how well various actors will keep up with them, and what to do defensively at a personal level?