Quick Takes

I've noticed that I really find it helpful when debates occur in the form of people arguing with each other in full blogposts. This sounds slightly hyperbolic, but I genuinely feel like that might be the best debate format ever invented, and I wish there was a lot more of it. (Comments are OK, but I think they have severe limitations for complex issues, because if you're wrong about something, it's often still quite easy to write a superficially-convincing comment without leaving much "attack surface".) Currently, I'd estimate that <5% of popular blogposts are direct replies to another post. I wish this number was, I don't know, 40%?

Some things I'm doing now to reduce the risk of software supply chain attacks: * I run all package manager installs through Socket Firewall (sfw npm install ...), which blocks known malicious packages and warns on various other problems. * Claude Code sessions run as a separate user[1], and I plan to move them all the way to a VM[2] in the near future. * Unfortunately, this user does need access to a lot of API keys including a GitHub token, but I limit their scope to the extent I can. * The risk of prompt injection seems to be relatively low in modern Claudes, so I'm not that worried about it. I do use agent-sanitizer in my custom harness though. * Claude has instructions to immediately notify me if malicious packages get installed anyway. * Claude does various supply chain checks on every dependency update PR from Dependabot. * I sometimes have Claude rewrite overly complex dependencies (too many transitive deps, written in a scary language like C). * I also use sfw on my own user account, and have removed passwordless docker and libvirt access, although I think if a supply chain attack hits my user account I'm probably doomed anyway. 1. ^ I use a custom web interface to make this convenient. 2. ^ I should have done this from day one, but agents need GPU access which is tricky to sandbox. Now that I have a custom GPU queue, my plan is to run normal agent sessions in a VM and expose a container with GPU access through gVisor.

I frequently hear on Reddit that model companies are dumbing models down while they are in use and new models are more capable on release day then later. To what extend is that just a conspiracy theory and to what extend does it match reality?

One common source of Illusion of Transparency problems is different senses of what's a normal range. The countercharm for this is using actual numbers and not relative comparisons. For instance, I talk to a lot of people who run conferences and meetups. Most of them anchor the size of an event off what they've seen or run. So someone might say "yeah, that was a really big convention last week!" to mean an EAG with almost two thousand people. Then someone else might say "yeah, the convention next week is looking like it's going to be a small one" and they're talking about DEF CON where twenty thousand would be a low turnout. For another example, I like hiking. Sometimes I say I'm going on a short hiking trip, and someone reasonably assumes I'll be back in a few hours, when in fact I'm bringing a tent and three or four days of food. But I'm not that into hiking; on the trail I regularly meet people who enjoy hiking the Long Trail (~270 miles) as a short backpacking trip because it's not the Appalachian Trail! (~2,200 miles.)

I'm worried that people won't reflect well and so (1) they'll decide to use the cosmic endowment for something mediocre and/or (2) they'll have a catastrophic failure in some of {decision theory, bargaining, dealing with threats, reasoning about simulators, acausal trade (including ECL etc.)}. Some angles: * Current AI systems are way better at science/engineering than philosophy. Maybe even ASI won't solve philosophy (in particular, maybe there's not a sufficiently good training signal to train super-philosophers via ML). * By default almost everyone is and will be overconfident in their intuitions on philosophy/metaethics/axiology. By default almost nobody will "suspect that they are confused about morality, and [be] eager or willing to apply reason and deliberation to find out what their real values are, or to correct their moral beliefs." * Long horizon agency / strategic competence approximately does not exist among humans, even the smartest ones. And maybe it's necessary. * It may be natural for powerful people to use AIs for instrumental stuff (acquiring power, inventing technology, building industrial capacity) but not terminal stuff (axiology, etc.). Some angles: * If they asked an enlightened philosophy ASI about its views and it replied straightforwardly, they would think it sounded crazy. * In general the ASIs will say crazy-sounding things about cosmology, religion, etc. * They will have seen other people "reflect" with "help" from ASIs and they think those people now have crazy views, talk about living in a simulation, etc. and they don't want to end up like that. * @habryka probably disagrees. Idk, it's so easy for me to imagine that Elon (or almost anyone else) would listen/defer to AI on the instrumental stuff but not reflect on philosophy/axiology. * Maybe there will have been a period where superpersuaders went around getting people to adopt new ideologies—maybe even by saying true but filtered things—and so people develop stro

Anthropic+OpenAI compute (IT power, not FLOP/s) will 2.5x-2.3x every year through 2028 (growth only slightly slows), say the projections given by SemiAnalysis in a talk delivered 1 Oct 2026. In the slide at 4:15, incremental compute (new compute that becomes available that year) for Anthropic+OpenAI is given for 2025-2028, in the context of global incremental compute (AI+non-AI): 2.9 GW in 2025, 5.7 GW in 2026, 13.2 GW in 2027, 29.9 GW in 2028. This is 16%, 21%, 26%, and 36% of global incremental compute (17.8 GW in 2025, 27.5 GW in 2026, 50.0 GW in 2027, 83.8 GW in 2028) respectively. The next slide at 4:40 gives the aggregate compute for Anthropic and OpenAI individually. In particular, Anthropic's implied incremental compute for 2028 is 18.2 GW. And since the aggregate compute for end of 2028 for Anthropic+OpenAI is 52.6 GW, while the incremental compute for 2025-2028 is 51.7 GW, it follows that their aggregate compute for end of 2024 was about 0.9 GW. Thus aggregate compute (Anthropic+OpenAI) is 3.8 GW end of 2025 (4.2x Y/Y), 9.5 GW end of 2026 (2.5x Y/Y), 22.7 GW end of 2027 (2.4x Y/Y), and 52.6 GW end of 2028 (2.3x Y/Y). The slide at 18:20 gives global incremental AI compute for 2026-2030: 19.3 GW in 2026, 35.8 GW in 2027, 40.7 GW in 2028, 63.3 GW in 2029, 79.2 GW in 2030. Comparing to the first slide for 2026, 2027, and 2028, this is respectively 70%, 72%, and 49% of global incremental compute (AI+non-AI). And in turn the Anthropic+OpenAI incremental compute for these years is then 30%, 37%, and 73% of the global incremental AI compute respectively. There's something funny going on with 2028 here, as if the satisfied demand for AI compute that's not for Anthropic+OpenAI goes down relative to both Anthropic+OpenAI demand, and to non-AI global compute demand (like a supply shortage that gets resolved in the direction of giving almost everything to Anthropic+OpenAI). But this happens in a way that maintains the year-over-year trend for Anthropic+OpenAI compute

https://rikiyat.github.io/blog/weight-space-programming/ Based only on raw weights, Astra 6 is able to tell what a model was trained to do and directly edit, without environmental feedback, those model weights to drastically increase performance on various tasks. RSI probably immediately falls out of these capabilities, and we do not know if OpenAI has RL'd for this. RSI does not fall out of these capabilities, but we have no reason to believe that these were well elicited.[1] If these capabilities can be improved through RL, then I think we have reason for concern. 1. ^ These capabilities seem to be emerging on the frontier, so they might be elicited: https://x.com/alexkrentsel/status/2117376683724660898?a=zL0qV4at-5PBm8oi_HY&s=46