MIRI is potentially interested in supporting reading groups for If Anyone Builds It, Everyone Dies by offering study questions, facilitation, and / or copies of the book, at our discretion. If you lead a pre-existing reading group of some kind (or meetup group that occasionally reads things together), please fill out this form.

The deadline to submit is September 22, but sooner is much better.

Reading so many reviews/responses to IABIED, I wish more people had registered how they expected to feel about the book, or how they think a book on x-risk ought to look, prior to the book's release. 

Finalizing any Real Actual Object requires making tradeoffs. I think it's pretty easy to critique the book on a level of abstraction that respects what it is Trying To Be in only the broadest possible terms, rather than acknowledging various sub-goals (e.g. providing an updated version of Nate + Eliezer's now very old 'canonical' arguments), modulations of the broader goal (e.g. avoiding making strong claims about timelines, knowing this might hamstring the urgency of the message), and constraints (e.g. going through an accelerated version of the traditional publishing timeline, which means the text predates Anthropic's Agentic Misalignment and, I'm sure, various other important recent findings).

A lot of the takes I see seem to come from a place of defending the ideal version of such a text by the lights of the reviewer, but it's actually unclear to me whether many of these reviewers would have made the opposite critiques if the book had made the opposite call on the various tradeoffs. I don't mean to say I think these reviewers are acting in bad faith; I just think it's easy to avoid confronting how your ideal version couldn't possibly be realized, and make post-hoc adjustments to that ideal thing in service of some (genuine, worthwhile) critique of the Real Thing.

Previously, it annoyed me that people had pre-judged the book's contents. Now, I'm grateful to folks who wrote about it, or talked to me about it, before they read it (Buck Shlegeris, Nina Panickserry, a few others), because I can judge the consistency of their rubric myself, rather than just feeling:

Yes, this came up during drafting, but there was a reasonable tradeoff. We won't know if that was a good call until later. If I had more energy I'd go 20 comments deep with you, and you'd probably agree it was a reasonable call by the end, but still think it was incorrect, and we'd agree to let time tell.

Which is the feeling that's overtaken me as I read the various reviews from folks throughout the community.

I should say: I'm grateful for all the conversation, including the dissent, because it's all data, but it is worse data than it would have been if you'd taken it upon yourself to cause a fuss in one of the many LW posts made in the lead-up to the book (and in the future I will be less rude to people who do this, because actually, it's a kindness!).

Sometimes people give a short description of their work. Sometimes they give a long one.

I have an imaginary friend whose work I’m excited about. I recently overheard them introduce and motivate their work to a crowd of young safety researchers, and I took notes. Here’s my best reconstruction of what he’s up to:

"I work on median-case out-with-a-whimper scenarios and automation forecasting, with special attention to the possibility of mass-disempowerment due to wealth disparity and/or centralization of labor power. I identify existing legal and technological bottlenecks to this hypothetical automation wave, including a list of relevant laws in industries likely to be affected and a suite of evals designed to detect exactly which kinds of tasks are likely to be automated and when. 

"My guess is that there are economically valuable AI systems between us and AGI/TAI/ASI, and that executing on safety and alignment plans in the midst of a rapid automation wave is dizzyingly challenging. Thinking through those waves in advance feels like a natural extension of placing any weight at all on the structure of the organization that happens to develop the first Real Scary AI. If we think that the organizational structure and local incentives of a scaling lab matter, shouldn’t we also think that the societal conditions and broader class of incentives matter? Might they matter more? The state of the world just before The Thing comes on line, or as The Team that makes The Thing is considering making The Thing, has consequences for the nature of the socio-technical solutions that work in context.

"At minimum, my work aims to buy us some time and orienting-power as the stakes raise. I’d imagine my maximal impact is something like “develop automation timelines and rollout plans that you can peg AI development to, such that the state of the world and the state of the art AI technology advance in step, minimizing the collateral damage and chaos of any great economic shift.”

"When I’ve brought these concerns up to folks at labs, they’ve said that these matters get discussed internally, and that there’s at least some agreement that my direction is important, but that they can’t possibly be expected to do everything to make the world ready for their tech. I, perhaps somewhat cynically, think they’re doing narrow work here on the most economically valuable parts, but that they’re disinterested in broader coordination with public and private entities, since it would be economically disadvantageous to them.

"When I’ve brought these concerns up to folks in policy, they’ve said that some work like this is happening, but that it’s typically done in secret, to avoid amorphous negative externalities. Indeed, the more important this work is, the less likely someone is to publish it. There’s some concern that a robust and publicly available framework of this type could become a roadmap for scaling labs that helps them focus their efforts for near-term investor returns, possibly creating more fluid investment feedback loops and lowering the odds that disappointed investors back out, indirectly accelerating progress.

"Publicly available work on the topic is ~abysmal, painting the best case scenario as the most economically explosive one (most work of this type is written for investors and other powerful people), rather than pricing in the heightened x-risk embedded in this kind of destabilization. There's actually an IMF paper modeling automation from AI systems using math from the industrial revolution. Surely there's a better way here, and I hope to find it."

[errant thought pointing a direction, low-confidence musing, likely retreading old ground]

There’s a disagreement that crops up in conversations about changing people’s minds. Sides are roughly:

1. You should explain things by walking someone through your entire thought process, as it actually unfolded. Changing minds is best done by offering an account of how your own mind was changed.
2. You should explain things by back-chaining the most viable (valid) argument, from your conclusions, with respect to your specific audience.

This first strategy invites framing your argument around the question “How did I come to change my mind?”, and this second invites framing your argument around the question “How might I change my audience’s mind?”. I am sometimes characterized as advocating for approach 2, and have never actually taken that to be my position.I think there’s a third approach here, which will look to advocates of approach 1 as if it were approach 2, and look to advocates of approach 2 as if it were approach 1. That is, you should frame the strategy around the question “How might my audience come to change their mind?”, and then not even try to change it yourself.

This third strategy is about giving people handles and mechanisms that empower them to update based on evidence they will encounter in the natural course of their lives, rather than trying to do all of the work upfront. Don’t frame your own position as some competing argument in the market place of ideas; hand your interlocutor a tool, tell them what they might expect, and let their experience confirm your predictions.I think this approach has a few major differences over the other two approaches, from the perspective of its impact:

1. It requires much less authority. (Strength!)
2. It can be executed in a targeted, light-weight fashion. (Strength!)
3. It’s less likely to slip into deception than option 2, and less confrontational than option 1. (Strength!)
4. Even if it works, they won’t end up thinking exactly what you think (Weakness?)
5. ….but they’ll be better equipped to make sense of new evidence (Strength!)
6. Plausibly more mimetically fit than option 1 or 2 (a failure mode of 1 is that your interlocutor won’t be empowered to stand up to criticism while spreading the ideas, even to people very much like themselves, and for option 2 it’s that they will ONLY be successful in spreading the idea to people who are like themselves, since they only know the argument that works on them).

I think Eliezer has talked about some version of this in the past, and this is part of why people like predictions in general, but I think pasting a prediction at the end of an argument built around strategy 1 or 2 isn't actually Doing The Thing I mean here.

Friends report Logan's writing strongly has this property.

Do you think of rationality as a similar sort of 'object' or 'discipline' to philosophy? If not, what kind of object do you think of it as being?

(I am no great advocate for academic philosophy; I left that shit way behind ~a decade ago after going quite a ways down the path. I just want to better understand whether folks consider Rationality as a replacement for philosophy, a replacement for some of philosophy, a subset of philosophical commitments, a series of cognitive practices, or something else entirely. I can model it, internally, as aiming to be any of these things, without other parts of my understanding changing very much, but they all have 'gaps', where there are things that I associate with Rationality that don't actually naturally fall out of the core concepts as construed as any of these types of category [I suppose this is the 'being a subculture' x-factor]).

Sometimes people express concern that AIs may replace them in the workplace. This is (mostly) silly. Not that it won't happen, but you've gotta break some eggs to make an industrial revolution. This is just 'how economies work' (whether or not they can / should work this way is a different question altogether). 

The intrinsic fear of joblessness-resulting-from-automation is tantamount to worrying that curing infectious diseases would put gravediggers out of business.

There is a special case here, though: double digit unemployment (and youth unemployment, in particular) is a major destabilizing force in politics. You definitely don't want an automation wave so rapid that the jobless and nihilistic youth mount a civil war, sharply curtailing your ability to govern the dangerous technologies which took everyone's jobs in the first place.

As AI systems become more expensive, and more powerful, and pressure to deploy them profitably increases, I'm fairly concerned that we'll see a massive hollowing out of many white collar professions, resulting in substantial civil unrest, violence, chaos. I'm not confident that we'll get (i.e.) a UBI (or that it would meaningfully change the situation even if we did), and I'm not confident that there's enough inertia in existing economic structures to soften the blow.

The IMF estimates that current tech (~GPT 4 at launch) can automate ~30% of human labor performed in the US. That's a big, scary number. About half of these, they imagine, are the kinds of things you always want more of anyway, and that this complementarity will just drive production in that 15% of cases. The other 15%, though, probably just stop existing as jobs altogether (for various reasons, I think a 9:1 replacement rate is more likely than full automation, with current tech).

This mostly isn't happening yet because you need an ML engineer to commit Serious Time to automating away That Job In Particular. ML engineers are expensive, and usually not specced for the kind of client-facing work that this would require (i.e. breaking down tasks that are part of a job, knowing what parts can be automated, and via what mechanisms, be that purpose-built models, fine-tuning, a prompt library for a human operator, some specialized scaffolding...). There's just a lot of friction and lay-speak necessary to accomplish this, and it's probably not economically worth it for some subset of necessary parties (ML engineers can make more elsewhere than small business owners can afford to pay them to automate things away, for instance).

So we've got a bottleneck, and on the other side of it, this speculative 15% leap in unemployment. That 15% potential leap, though, is climbing as capabilities increase (this is tautologically true; "drop in replacement for a remote worker" is one major criteria used in discussions about AI progress).

I don't expect 15% unemployment to destabilize the government (Great Depression peak was 25%, which is a decent lower bound on 'potentially dangerous' levels of unemployment in the US). But I do expect that 15% powder keg to grow in size, and potentially cross into dangerous territory before it's lit.

Previously, I'd actually arrived at that 30% number myself (almost exactly one year ago), but I had initially expected:

1. Labs would devote substantial resources to this automation, and it would happen more quickly than it has so far.
2. All of these jobs were just on the chopping block (frankly, I'm not sure how much I buy the complementarity argument, but I am An Internet Crank, and they are the International Monetary Fund, so I'll defer to them).

These two beliefs made the situation look much more dire than I now believe it to be, but it's still, I claim, worth entertaining as A Way This Whole Thing Could Go, especially if we're hitting a capabilities plateau, and especially if we're doubling down on government intervention as our key lever in obviating x-risk.

[I'm not advocating for a centrally planned automation schema, to be clear; I think these things have basically never worked, but would like to hear counterexamples. Maybe just like... a tax on automation to help staunch the flow of resources into the labs and their surrogates, a restructuring of unemployment benefits and retraining programs and, before any of that, a more robust effort to model the economic consequences of current and future systems than the IMF report that just duplicates the findings of some idiot (me) frantically reviewing BLS statistics in the winter of 2023.]

I (and maybe you) have historically underrated the density of people with religious backgrounds in secular hubs. Most of these people don't 'think differently', in a structural sense, from their forebears; they just don't believe in that God anymore. 

The hallmark here is a kind of naive enlightenment approach that ignores ~200 years of intellectual history (and a great many thinkers from before that period, including canonical philosophers they might claim to love/respect/understand). This type of thing.

They're no less tribal or dogmatic, or more critical, than the place they came from. They just vote the other way and can maybe talk about one or two levels of abstraction beyond the stereotype they identify against (although they can't really think about those levels).

You should still be nice to them, and honest with them, but you should understand what you're getting into.

The mere biographical detail of having a religious background or being religious isn't a strong mark against someone's thinking on other topics, but it is a sign you may be talking to a member of a certain meta-intellectual culture, and need to modulate your style. I have definitely had valuable conversations with people that firmly belong in this category, and would not categorically discourage engagement. Just don't be so surprised when the usual jutsu falls flat!

I don't think I really understood what it meant for establishment politics to be divisive until this past election.

As good as it feels to sit on the left and say "they want you to hate immigrants" or "they want you to hate queer people", it seems similarly (although probably not equally?) true that the center left also has people they want you to hate (the religious, the rich, the slightly-more-successful-than-you, the ideologically-impure-who-once-said-a-bad-thing-on-the-internet).

But there's also a deeper, structural sense in which it's true.

Working on AIS, I've long hoped that we could form a coalition with all of the other people worried about AI, because a good deal of them just.. share (some version of) our concerns, and our most ambitious policy solutions (e.g. stopping development, mandating more robust interpretability and evals) could also solve a bunch of problems highlighted by the FATE community, the automation-concerned, etc etc. 

Their positions also have the benefit of conforming to widely-held anxieties ('I am worried AI will just be another tool of empire', 'I am worried I will lose my job for banal normie reasons that have nothing to do with civilizational robustness', 'I am worried AI's will cheaply replace human labor and do a worse job, enshittifying everything in the developed world'). We could generally curry popular support and favor, without being dishonest, by looking at the Venn diagram of things we want and things they want (which would also help keep AI policy from sliding into partisanship, if such a thing is still possible, given the largely right-leaning associations of the AIS community*).

For the next four years, at the very least, I am forced to lay this hope aside. That the EO contained language in service of the FATE community was, in hindsight, very bad, and probably foreseeably so, given that even moderate Republicans like to score easy points on culture war bullshit. Probably it will be revoked, because language about bias made it an easy thing for Vance to call "far left".

"This is ok because it will just be replaced."

Given the current state of the game board, I don't want to be losing any turns. We've already lost too many turns; setbacks are unacceptable.

"What if it gets replaced by something better?"

I envy your optimism. I'm also concerned about the same dynamic playing out in reverse; what if the new EO (or piece of legislation via whatever mechanism), like the old EO, contains some language that is (to us) beside the point, but nonetheless signals partisanship, and is retributively revoked or repealed by the next administration? This is why you don't want AIS to be partisan; partisanship is dialectics without teleology.

Ok, so structurally divisive: establishment politics has made it ~impossible to form meaningful coalitions around issues other than absolute lightning rods (e.g. abortion, immigration; the 'levers' available to partisan hacks looking to gin up donations). It's not just that they make you hate your neighbors, it's that they make you behave as though you hate your neighbors, lest your policy proposals get painted with the broad red brush and summarily dismissed.

I think this is the kind of observation that leads many experienced people interested in AIS to work on things outside of AIS, but with an eye toward implications for AI (e.g. Critch, A Ray). You just have these lucid flashes of how stacked the deck really is, and set about digging the channel that is, compared to the existing channels, marginally more robust to reactionary dynamics ('aligning the current of history with your aims' is maybe a good image).

Hopefully undemocratic regulatory processes serve their function as a backdoor for the sensible, but it's unclear how penetrating the partisanship will be over the next four years (and, of course, those at the top are promising that it will be Very Penetrating).

*I am somewhat ambivalent about how right-leaning AIS really is. Right-leaning compared to middle class Americans living in major metros? Probably. Tolerant of people with pretty far-right views? Sure, to a point. Right of the American center as defined in electoral politics (e.g. 'Republican-voting')? Usually not.