Other MIRI staff report that the book helped them fit the whole argument in their head better, or that it made sharp some intuitions they had that were previously vague. So you might get something out of it even if you’ve been around a while.
Can confirm! I've followed this stuff for forever, but always felt at the edge of my technical depth when it came to alignment. It wasn’t until I read an early draft of this book a year ago that I felt like I could trace a continuous, solid line from “superintelligence grown by a blind process...” to “...develops weird internal drives we could not have anticipated”. Before, I was like, "We don't have justifiable confidence that we can make something that reflects our values, especially over the long haul," and now I'm like, "Oh, you can't get there from here. Clear as day."
As for why this spells disaster if anyone builds it, I didn't need any new lessons, but they are here, and they are chilling--even for someone who was already convinced we were in trouble.
Having played some small part in helping this book come together, I would like to attest to the sheer amount of iteration it has gone through over the last year. Nate and co. have been relentlessly paring and grinding this text ever closer to the kind of accessibility that won't just help individuals understand why we must act, but will make them feel like their neighbors and political leaders can understand it, too. I think that last part counts for a lot.
The book is also pretty engaging.
The pitch I suggested we share with our friends and allies is this:
If you’ve been waiting for a book that can explain the technical roots of the problem in terms your representative and your mother can both understand, this is the one. This is the grounded, no-nonsense primer on why superintelligence built blindly via gradient descent will predictably develop human-incompatible drives; on why humanity cannot hope to endure if the fatal, invisible threshold is crossed; and on what it will take to survive the coming years and decades.
Bestseller algorithms are secret and shifty, but hardcover is generally believed to count a little more. And as for overall impact, if either format is good for you, a hardcover preorder helps more because it encourages the publisher to print a bigger initial run of physical copies, which can get pumped into stores and onto shelves where people will see them.
¿Por qué no los dos? Different problems, both worth solving, and not especially connected, in my model.
"Start with an easier problem with similar dynamics to build up expertise" sounds to me like a fully general argument for working on anything other than the actual thing one cares about; it's a procrastinator's mantra, in most cases. There are plenty of exceptions, but I don't think this is one of them.
If I thought there could only ever be one concerted attempt at an AGI ban, and that this was much less likely to fail if the promoters had previously solved some other kind of ban, I could sort of see the logic. But that doesn't match my model. I'm not even convinced that banning bio gain-of-function is harder, having not tried it, and noticing that I'm confused about the entrenched interests that must exist for it to have continued as long as it has.
I also could see this approach backfiring, as the AGI ban promoters would be seen as the "guys who like to ban stuff" people, and as people who were more concerned about gain-of-function than AI catastrophe.
Glad you got something out of the post! I recognize and appreciate your generous action, and will DM you with regards to your request.
MIRI has its monthly newsletters, though I can tell that's not quite what you want. I predict (medium confidence) that we will be upping our level of active coordination with allied orgs and individuals on upcoming projects once we ship some of our current ones. I believe you already have channels to people at MIRI, but feel free to DM me if you want to chat.
Me too! I put the AI problem in the broader class of topics where apathy serves as a dual defense mechanism — not just against needing to expend effort and resources, but against emotional discomfort. You can see the same dual barrier when promoting charitable causes aimed at reducing human misery, or when teaching a subject to students who have really struggled with it in the past.
As a teacher, I attacked both of those roots more deliberately as I grew, trying hard to not make my class feel like work most days while building an atmosphere of low-stakes experimentation where failure could be fun rather than painful. (An example of what success looked like: students taking the risk of trying the more advanced writing approaches I modeled instead of endlessly rewriting the same basic meta essay they had learned in middle school.)
One tactic for eroding defensive apathy is therapeutic empathy. You see this both in many good teachers and (I imagine) relationship counselors. It’s much harder in writing, though I suppose I did a little bit of that in this post when I talked about how the reader and I have probably both felt the pull of Apathy with regards to the AI problem. I think empathy works partly because it builds a human connection, and partly because it brings the feared pain to the surface, where we find (with the help of that human connection) that it can be endured, freeing us to work on the problem that accompanies it.
Whether and how to use authentic human connections in our communications is a topic of ongoing research and debate at MIRI. It has obvious problems with regards to scientific respectability, as there’s this sense in intellectual culture that it’s impossible to be impartial about anything one has feelings about.
And sure, the science itself should be dispassionate. The universe doesn’t care how we feel about it, and our emotions will try to keep us from learning things we don’t want to be true.
But when communicating our findings? To the extent that our task is two-pronged: (1) communicating the truth as we understand it and (2) eliciting a global response to address it, I suspect we will need some human warmth and connection in the second prong even as we continue to avoid it in the first. Apathy loves the cold.
From chatting with those peak students during the experiment, I think their experience is more like being in a cafeteria abuzz with the voices of friends and acquaintances. At some point, you're not even trying to follow every conversation, but are just maintaining some vague awareness of the conversations that are taking place and jumping in when you feel like it. People can and do think about other things in a noisy cafeteria. Some even read books! The brain can filter out a constant buzz. It's just wind blowing through the trees.
The upper middle zone where it's still possible to try to follow everything (and maybe even reply) looked like more of an attention trap, and was where I was more likely to find that handful of students I already knew had a problem. The FOMO is probably more distracting than the notifications themselves.
They should not have been counting pull notifications, as they were instructed to not engage with their phones during the experiment except to maybe see what caused a vibration or ding. I don't think students think of pull notifications as real notifications the way we were using the word. They were logging the notifications they could notice while their phone flat was flat on their desk not being touched.
No. Everyone seemed to know what they were, because they all claimed to know someone who uses them. But I don't recall anyone ever admitting to being such a someone. I sense there's a bit of a stigma around them.
Looking back, I was surprised by the (unflattering, in my opinion) degree to which LWers saw this data as strong confirmation of their hypotheses about phones being the source of the ills they see in our schools and our young.
I thought it was much more of a mixed picture despite -- or perhaps because -- the numbers were significantly higher than I, a veteran teacher, had expected: If I had been told the previous summer that "next year, we're giving you a cohort of students with this phone usage profile", I might have braced for a crop of students that came off as especially scatterbrained. But overall, these students -- even most of the high-receivers on the left of the chart -- seemed to me about as well adjusted as teenagers of yesteryear and, I would say, better adjusted than the students I remember from the early days of pre-smartphone texting going mainstream. (There's probably a thread to pick there about messages being more alive among teens with vibrant in-person social lives.)
The update for me from this experient was more like, "Huh, I guess notifications are just sort of a background hum of modern existence now. If adults can largely manage to be productive in this environment, maybe I shouldn't be so surprised that young people who grew up in it can, too." This it's-just-a-different-world-now feeling was accentuated by the fact that the school itself was now behind a high fraction of notifications for the median student.
I'm wary of anecdotal clickbait-ish tales of "This school locked away phones, and something miraculous happened" because I would expect a honeymoon phase where an engaging teacher can step in and fill any real attention void before the more traditonal expressions of mental exhaustion and apathy can assert themselves.
To change my mind about this, I would need to linger in the counterfactual world where the students I had known from the start to be actually, problematically distracted by their devices were permanently deprived of them. My mental simulation of that world sees about half of these students gradually becoming traditional excessively-chatty-in-class types, and most of the rest daydreaming, sleeping, or checking out in some other way. To the extent that a busy life on a phone keeps a teen's mind active, it might actually be an improvement?