Interesting thought, Thomas. Although I agree with RussellThor that it seems like doing something along the lines of "just jitter the position of the RV using little retrofitted fins / airbrakes" might be enough to defeat your essentially "pre-positioned / stationary interceptors". (Not literally stationary, but it is as if they are stationary given that they aren't very maneuverable relative to the speed of the incoming RV, and targeted only based on projected RV trajectories calculated several minutes earlier.)
(Is the already-existing atmospheric turbulence already enough to make this plan problematic, even with zero retrofitting? The circular-error-probable of the most accurate ICBMs is around 100 meters; presumably the vast majority of this uncertainty is locked in during the initial launch into space. But if atmospheric drag during reentry is contributing even a couple of those meters of error, that could be a problem for "stationary interceptors".)
Failing all else, I suppose an attacker could also go with Russell's hilarious "nuke your way through the atmosphere" concept, although this does at least start to favor the defender (if you call it favorable to have hundreds of nukes go off in the air above your country, lol) insofar as the attacker is forced to expend some warheads just punching a hole through the missile defense -- a kind of "reverse MIRV" effect.
Regardless, you still face the geography problem, where you have to cover the entire USA with Patriot missile batteries just to defend against a single ICBM (which can choose to aim anywhere).
I would also worry that "in the limit of perfect sensing" elides the fact that you don't JUST have to worry about getting such good sensing that you can pin down an RV's trajectory to within, like, less than a meter? (In order to place a completely dumb interceptor EXACTLY in the RV's path. Or maybe a few tens of meters, if you're able to put some sensors onto your cheap interceptor without raising the price too much, and make use of what little maneuverability you have versus the RV?) You ALSO have to worry about distinguishing real warheads from fake decoys, right? Sorting out the decoys from the warheads might be even harder than exactly pinning down an RV's trajectory.
According to a random redditor, apparently today's decoys are "inflatable balloons for exoatmospheric use and small darts for in-atmosphere", plus "radio jammers, chafe, and other things designed to confuse enemy detection. With better and better sensing, maybe you could force an attacker to up their decoy game, retrofitting their missiles to use fewer, more lifelike decoys, maybe even to such an extreme extent that it's no longer really worth using decoys at all, compared to just putting more MIRV warheads on each missile? But if decoys still work, then you need that many more interceptors.
"In the limit of perfect sensors" (and also perfectly affordable sensors), with perfect interception (against non-retrofitted, non-HGV missiles) and perfect decoy discrimination, I suppose it becomes a defense-economics balance where you are hoping that the cost of lots of small rockets is cheaper than the cost of the attacking ICBM system. These small rockets don't have to be super fast, don't have to go super high, and don't have to be super maneuverable. But they do have to be precise enough to maneuver to an exact precalculated location, and you need enough of them to blanket essentially your entire country (or at least all the important cities). You are basically relying on the fact that the ICBM has to be super big and heavy to launch a significant payload all the way around the earth, while the numerous small missiles only have to fly a couple of kilometers into the sky.
Final, dumbest thought saved for last:
Aside from developing HGVs, couldn't the ICBMs in theory overcome this defense with brute scale, by MIRV-ing to an an absurd degree? How many warheads can dance on the head of a Starship? Could you just put the entire US nuclear arsenal on a single launcher? The cost of your ICBM would essentially be zero when amortized over all those warheads, so the defense economics battle just becomes the cost of warheads vs patriots, instead of entire ICBMs vs patriots. Obviously there are many reasons why this idea is nuts:
But I thought it was kind of funny to think about, and this absurd thought experiment maybe sheds some light on the underlying dynamics of the situation.
IUCN numbers are a decent starting point, although as Shankar notes, the IUCN is too conservative in the sense that they wait a long time before finally conceding a species has gone fully extinct.
For megafauna extinctions during the Ice Age, wikipedia tallies up 168 lost species. (On the one hand, maybe not literally all of these were due to humans. But on the other hand, our record of creatures living 12,000 years ago is probably pretty spotty compared to today, so we might be missing a bunch!) https://en.m.wikipedia.org/wiki/Late_Pleistocene_extinctions
Another difficult aspect of trying to estimate "how many species have gone extinct" is that we have lots of detailed information about mammals and birds, okay info about reptiles, fish, etc, but then MUCH spottier information about insects, stuff like plankton or bacteria, etc. And while there are about 6000 mammal and 11,000 bird species worldwide, there are maybe somewhere around a million species of insect, and who knows how many types of bacteria / plankton / whatever.
https://ourworldindata.org/how-many-species-are-there
https://ourworldindata.org/grapher/share-of-species-evaluated-iucn
(For instance, the IUCN data on actual confirmed extinctions has a pretty different ranking than their list of at-risk endangered species: )
So for mammals + birds and a few other groups, you can get a pretty definitive estimate of "what percentage of species have gone extinct in the last few hundred years". But if you are asking about ALL species, then your answer almost entirely depends on how you choose to extrapolate the well-documented mammal/bird rates to much larger groups like insects, plants, fungi, and bacteria (and indeed, weird activists will extrapolate in stupid, biased ways all the time). It's not straightforward to figure out how to do this right, since different kinds of life have different rates of extinction -- plants, for instance, seem to almost never go extinct compared to animals. Birds seem more robust than mammals to the effects of human civilization (since they can fly, they're less immediately-screwed than land-dwelling creatures, when humans break up previously homogenous environments into isolated patches of habitat separated by roads, fences, farmland, etc). Meanwhile, amphibians' absorbent skin and pretty specialized habitat needs make them especially vulnerable to extinction via pollution and habitat disruption. Then there are creatures like hard corals where it's like "if the ocean becomes X amount more acidic then they basically all die at once". So... are insects resilient like plants, or extra vulnerable like amphibians or corals? Personally I have no idea. Are thousands of bacteria species going extinct all the time for weird chemical or micro-ecological reasons that we barely understand? Or would they just cruise right through even a worst-case meteor strike while hardly batting an eyelid? Hard to say; too bad they make up maybe 90% of all species!
You could try to construct some weighting scheme to reflect the intuition that obviously losing a species of tiger or elephant is worse than losing one of 10,000 small indistinguishable beetles. Aside from the difficulty of operationalizing "how much do i care about each species" (physical size? squared neuron count? upweight mammals vs otherwise-equal birds because they're more closely related to ourselves? or should birds get extra points for being colorful and pretty?), such a project also runs into a lot of interesting questions about phylogenetics and evolutionary distinctiveness. (Two "different" "species" that actually only differ by a few mutations, IMO is unfair double-counting; they're basically just one species and shouldn't be entitled to double the conservation effort. Meanwhile, very unique species that have been evolving along their own unusual track for millions of years seem like they should get credit for punching above their weight in terms of maintaining earth's diversity of life. Some species also contain within themselves lots of interesting variation and distinct sub-populations, while other species are more of a homogenous monoculture. et cetera.)
The above info is all a rough and probably misunderstood paraphrase of stuff my wife has told me over the years; she runs the "Ecoresilience Initiative", a nascent sort of EA-for-biodiversity research group. Email her there if you want to learn more!
https://ecoresilience.weebly.com/
My thoughts about the story, perhaps interesting to any future reader trying to decipher the "mysterianism" here, a la these two analyses of sci-fi short stories, or my own attempts at exegeses of videogames like Braid or The Witness. Consider the following also as a token of thanks for all the enjoyment I've recieved from reading Gwern's various analyses over the years.
I think that jumping straight to big-picture ideological questions is a mistake. But for what it's worth, I tried to tally up some pros and cons of "ideologically who is more likely to implement a post-scarcity socialist utopia" here; I think it's more of a mixed bag than many assume.
I think when it comes to the question of "who's more likely to use AGI build fully-automated luxury communism", there are actually a lot of competing considerations on both sides, and it's not nearly as clear as you make it out.
Xi Jinping, the leader of the CCP, seems like kind of a mixed bag:
Meanwhile on the American side, I'd probably agree with you that the morality of America's current national leaders strikes me as... leaving much to be desired, to put it lightly. Personally, I would give Trump maybe only 1 or 1.5 points out of three on my earlier criteria of "fundamentally humanist outlook + not a psychopath + not a destructive idiot".
Finally, I would note that you are basically raising concerns about humanity's "gradual disempowerment" through misaligned economic and political processes, AI concentration-of-power risks where a small cadre of capricious national leaders and insiders gets to decide the fate of humanity, etc. Per my other comment in this thread, these types of AI safety concerns seem like right now they are being discussed almost exclusively in the West, and not in China. (This particular gradual-disempowerment stuff seems even MORE lopsided in favor of the West, even compared to superintelligence / existential risk concerns in general, which are already more lopsided in favor of the West than the entire category of AI safety overall.) So... maybe give some weight to the idea that if you are worried about a big problem, the problem might be more likely to get solved in the country where people are talking about the problem!
@Tomás B. There is also vastly less of an "AI safety community" in China -- probably much less AI safety research in general, and much less of it, in percentage terms, is aimed at thinking ahead about superintelligent AI. (ie, more of China's "AI safety research" is probably focused on things like reducing LLM hallucinations, making sure it doesn't make politically incorrect statements, etc.)
When people ask this question about the relative value of "US" vs "Chinese" AI, they often go straight for big-picture political questions about whether the leadership of China or the US is more morally righteous, less likely to abuse human rights, et cetera. Personally, in these debates, I do tend to favor the USA, although certainly both the US and China have many deep and extremely troubling flaws -- both seem very far from the kind of responsible, competent, benevolent entity to whom I would like to entrust humanity's future.
But before we even get to that question of "What would national leaders do with an aligned superintelligence, if they had one," we must answer the question "Do this nation's AI labs seem likely to produce an aligned superintelligence?" Again, the USA leaves a lot to be desired here. But oftentimes China seems to not even be thinking about the problem. This is a huge issue from both a technical perspective (if you don't have any kind of plan for how you're going to align superintelligence, perhaps you are less likely to align superintelligence), AND from a governance perspective (if policymakers just think of AI as a tool for boosting economic / military progress and haven't thought about the many unique implications of superintelligence, then they will probably make worse decisions during an extremely important period in history).
Now, indeed -- has Trump thought about superintelligence? Obviously not -- just trying to understand intelligent humans must be difficult for him. But the USA in general seems much more full of people who "take AI seriously" in one way or another -- sillicon-valley CEOs, pentagon advisers, billionare philanthropists, et cetera. Even in today's embarassing administration, there are very high-ranking people (like Elon Musk and J. D. Vance) who seem at least aware of the transformative potential of AI. China's government is more opaque, so maybe they're thinking about this stuff too. But all public evidence suggests to me that they're kinda just blindly racing forward, trying to match and surpass the West on capabilities, without giving much thought as to where this technology might ultimately go.
Thanks for this informative review! (May I suggest that The Witness is a much better candidate for "this generation's Myst"!)
That's a good point -- the kind of idealized personal life coach / advisor Dario describes in his post "Machines of Loving Grace" is definitely in a sense a personality upgrade over Claude 3.7. But I feel like when you think about it more closely, most of the improvements from Claude to ideal-AI-life-coach are coming from non-personality improvements, like:
Semi-related: if I'm reading OpenAI's recent post "How we think about safety and alignment" correctly, they seem to announce that they're planning on implementing some kind of AI Control agenda. Under the heading "iterative development" in the section "Our Core Principles" they say:
In the future, we may see scenarios where the model risks become unacceptable even relative to benefits. We’ll work hard to figure out how to mitigate those risks so that the benefits of the model can be realized. Along the way, we’ll likely test them in secure, controlled settings. We may deploy into constrained environments, limit to trusted users, or release tools, systems, or technologies developed by the AI rather than the AI itself.
Given the surrounding context in the original post, I think most people would read those sentences as saying something like: "In the future, we might develop AI with a lot of misuse risk, ie AI that can generate compelling propaganda or create cyberattacks. So we reserve the right to restrict how we deploy our models (ie giving the biology tool only to cancer researchers, not to everyone on the internet)."
But as written, I think OpenAI intends the sentences above to ALSO cover AI control scenarios like: "In the future, we might develop misaligned AIs that are actively scheming against us. If that happens, we reserve the right to continue to use those models internally, even though we know they're misaligned, while using AI control techniques ('deploy into constrained environments, limit to trusted users', etc) to try and get useful superalignment work out of them anyways."
I don't have a take on the pros/cons of a control agenda, but I haven't seen anyone else note this seeming policy statement of OpenAI's, so I figured I'd write it up.
I think you are missing out on a key second half to this story, which would make your motivational take at the end ("uh.. feel good about yourself for trying or something?") a lot stronger:
When you go to a ski resort, or a gym, or etc, it's not JUST that you only see the people who ski, work out, etc, while not seeing the 90% who don't do that activity. You see people WEIGHTED by the AMOUNT OF TIME they spend doing that activity, which skews heavily towards the most intense practitioners.
For example, suppose your local gym has 21 patrons:
- 7 have lapsed in their actual workout habit; they never show up to the gym even though they keep getting auto-charged the monthly fee because they've forgotten to cancel their membership.
- 7 manage to keep up a healthy but not outstanding workout habit -- they each manage to do a one-hour workout once a week
- 7 are total gym bros who get in a one-hour workout every single day, stacking those gainz
On a typical day, who visits the gym?
- zero of the lapsed members
- on average, just one of the once-a-week members (7 * 1/7 = 1)
- all seven of the hardcore gym rats
So, it's not just that you never see the lapsed members (or the people who never signed up in the first place). It's also that you get an extremely skewed view of who "goes to the gym" -- visiting the gym and looking around makes it seem like the clientele is 87.5% hardcore gym rats, when the true proportion is actually just 50%. (Albeit that 87.5% of the "total time spent in the gym" is spent by gym rats.)
You even mention this in some of your anecdotes, like "most of the other riders have been out 90 days just this season".
For me, this fact is heartening. For something like a gym or a ski resort (or the blog posts in your feed), comparing yourself to the people you see around you is actually setting a really high bar, since the people you see around you are weighted by the time they spend doing the activity (and/or by the number of posts they write). A gentler, more intermediate basis of comparison is to all the people who do "go skiing", but don't go every day -- the huge shadow mass of people who ski a couple times a year, whose population is probably way higher than the "I have a cabin next to the resort and buy the season pass every winter" contingent, but who are in the minority every day on the mountain.