Quick Takes

[Take that's been bumping around in AI governance circles, not original to me] The middle powers have strong incentives to stop/slow AI development (even if we ignore misalignment risk, which we shouldn't), and can plausibly do something about it. Even if we ignore misalignment risk, countries that are not the US and China by default will become totally geopolitically irrelevant if ASI is built. One or both of the US and China will have overwhelming economic and military power. The middle powers should therefore want to slow/stop AI development, until they can work out how this can be done without them becoming irrelevant. Middle powers have some leverage here. States with critical parts of the AI chip supply chain (Netherlands, Korea, Japan, maybe Taiwan) can refuse to sell critical components without certain demands. The EU market is a fairly big deal. States with nuclear weapons can threaten to use them, if they risk becoming totally disempowered, and can defend the chip supply chain states. My favorite ask right now is that chip supply chain states (Netherlands, Korea, Japan) should refuse to help manufacture chips (both selling components, and keeping SME machines running), unless chips are manufactured with secure verification mechanisms. These verification mechanisms can help give these states assurances that the chips aren't being used to disempower them, and can be used to verify a halt to AI development in the future. This move might be met by strong retaliation by states that want to keep buying chips, and so middle powers (including nuclear states) should coordinate and back each other up. This includes defense pacts. I therefore think it would probably be good to wake up governments in these countries in key middle powers about ASI. If ASI is real, then this is fairly straightforwardly in their interests, even ignoring misalignment risk.

LLMs are current level are already phenomenal. Enough to usher in a new industrial revolution even without further progress. Also still remarkable how untethered or nonsensical their reasoning can be, even with Opus 4.6 or similar. Ex1. I was working on parking brake issue with my car, comparing the clamping force I was getting the wheel with the observation that it had wanted to roll down the hill. I told it I was getting enough clamping to be unable to turn the wheel by hand. No, a 2,400lb car rolling down the hill exerts a lot more force than me trying to turn it at the wheel studs, let me tell ya. Ex2. I was setting of a long-running gene analysis job. A while after it had started, I asked if actually we could parallelize it. Claudes says yes, absolutely, there's a parameter already for that. I ask it to estimate whether it'd make sense to stop and restart the job. Yes, it says, would take half the time – but we've already started it so might as well let it finish. I feel like I get some many of these bonkers inferences, that there's something interesting here to reconcile with the brilliance they have in other moments.

Something I was wrong about: credit assignment. I used to think: I‘m an altruist; it doesn’t matter whether I get credit for my contributions. Now I think getting credit is often important. In some contexts, when you do a good thing, a lot of the value comes indirectly via you getting empowered. And if others systematically steal your credit or block you from taking credit, you should be scared of the prospect that (1) you’re giving them power which they will use poorly or (2) you become dependent on them — if you’d gotten credit, then you’d be empowered and people would listen to you, but since they took credit, you’re stuck only able to exert influence by advising them, and they can stop listening to you. There’s often reasonable-sounding arguments that it’s better if someone else takes credit, but you should (1) be suspicious and (2) just pay some costs to preserve proper credit assignment.

As far as I can tell, we are currently living in a glorious space opera future... for eukaryotes. From their early origins in a hydrothermal vent the eukaryotes have spread across the galaxy (earth), forming civilisations of unbelievable complexity and titanic power (animals and humans). Megastructures (houses) and gigastructures (cities) now exist where entire orders of eukaryotes live out their lives in peace, never feeling the threat of drought or lack of ATP. Generation ships called "planes", "trains", and "boats" carry generations from gigastructure to gigastructure at timescales that dwarf eukaryote understanding. Massive galactic empires make peace and war, destroying trillions with weapons of unfathomable power, while science gets ever closer to understanding the fundamental origins of life and the means by which it operates. Recently the empires have even launched expeditions beyond the known universe, seeking to harvest ever more gigantic sources of power to fuel glorious eukaryote replication throughout the multiverse. To an average eukaryote these wonders are beyond their comprehension, but in a very real way they are still making it happen.

How valuable are prediction markets for wake up / transparency? I think it's likely that prediction markets will be really helpful for keeping the public / civil society / policymakers aware of what's going on inside the labs. This is because prediction markets are a great way of aggregating public information, and also eliciting private information from insiders. And we've seen a few high-profile cases of big political decisions based on movements in prediction markets (e.g. the replacement of Joe Biden in the 2024 POTUS campaign). If this is true -- what should we do? Maybe we can subsidise prediction markets that we think are really important? Note that there are some downside risks — prediction markets cause the same problems that are caused by any push for transparency, i.e. sometimes we don't want dangerous information to be leaked, and sometimes people will change their behaviour in undesirable ways if they know the information would be leaked. But in general, I think more transparency about the capabilities and risks within the lab would be helpful. There are also other wrinkles like: * How do prediction markets work if people put significant weight on extinction, the expropriation of their resources, crazy interest rates stuff, and unknown unknowns. * How can we ensure that markets enjoy sufficient liquidity? * Will prediction markets cause further wealth concentration within the labs, because they have access to more information and better trading-enabiling AI capabilities?

I am starting to viscerally feel the possibility that I might be dead in <5 years because of AI. This is pretty new: I had a brief phase like this a few years ago when I first encountered the arguments for AI x-risk. Since then, I've developed some barely-above-subconscious coping mechanisms for not being too emotionally disturbed, most of which amounted to reasons why LLMs would not scale to ASI. But these have just stopped being convincing to my emotional brain the past few weeks. I've been having nightmares and ruminating a lot. I wonder if this is in the region of how a terminally ill person feels. I am also pretty scared of the fact that this anxiety is likely to become a lot worse as capabilities increase and more of my emotional brain's cope melts away. I'm not really sure what I'm trying to accomplish with this post. Perhaps others have been feeling this way and might want to share some tips for not going insane?

My dad is 100, and worked on nuclear energy policy for the US government for 50+ years. He was one of the attendees at the co nference [1] that led to the formation of the IAEA. He is also somewhat AGI pilled. I f anyone has questions they'd like me to ask him, let me know. He'd be happy to help in any way he can. 1. ^ He attended the second of two conferences that lead to the formation of the IAEA.