The Best of LessWrong

I think the high-level point in this post is correct and important, though not super action-guiding. I find it interesting to notice in 2025 that this post failed to mention the "obvious" social failure mode of extreme power concentration/power grabs/coups. Seems like a missed opportunity for this particular reflection.

I'm giving this -4 points because it seems anti-helpful on net, but not all bad. * The title is both clickbait and irritating to read for anyone who disagrees. It's like the author is trying to gaslight us into agreeing that alignment is dead, despite the field growing every year (including many agent foundations approaches). The linked resources do not sufficiently explain why the author thinks this way. My guess is it only got 328 karma because it was so provocative. * The author is wrong about several verifiable things eg the technical background of MATS scholars * OP is a big fan of ILIAD. ILIAD people are split-- Vanessa likes it but see Alexander Oldenziel's somewhat passive-aggressive comment claiming OP doesn't put in effort to understand what he's critiquing. * As Leo Gao points out in the comments, it's true in many fields that "most work will be crap (often predictably so ex ante), due in part to bad incentives, and then there will be a few people who still do good work anyways." So to expect otherwise for alignment is actually a very high standard. * As I wrote last year, much of what OP calls streetlighting is actually correct prioritization by tractability. Ok, streetlighting is also a problem sometimes, but OP's proposed solutions also seem fairly misguided. I basically still stand by that comment. * Overall I don't disagree with everything in the post (incentives to think alignment is easy at labs are certainly real), but 80% of it just reads as OP, a person who likes physics-type approaches to alignment rather than prosaic ones, complaining about how the taste of the average alignment researcher differs from theirs.

"Armchair psychologizing about which of my rhetorical opponents' cognitive deficits cause them to fail to agree with me" is by far my least favorite kind of LessWrong post, and the proposed solution to the "problem" ("recruit smarter people to the field") is not interesting or insightful.

To my mind, what this post did was clarify a kind of subtle, implicit blind spot in a lot of AI risk thinking. I think this was inextricably linked to the writing itself leaning into a form of beauty that doesn't tend to crop up much around these parts. And though the piece draws a lot of it back to Yudkowsky, I think the absence of green much wider than him and in many ways he's not the worst offender. It's hard to accurately compress the insights: the piece itself draws a lot on soft metaphor and on explaining what green is not. But personally it made me realise that the posture I and others tend to adopt when thinking about superintelligence and the arc of civilisation has a tendency to shut out some pretty deep intuitions that are particularly hard to translate into forceful argument. Even if I can't easily say what those are, I can now at least point to it in conversation by saying there's some kind of green thing missing.

"On Green" is one of the LessWrong essays which I most often refer to in my own thoughts (along with "Deep Atheism", which I think of as a partner to "On Green"). Many essays I absorb into my thinking, metabolize the contents, but don't often think of the essay itself. But this one is different. The essay itself has stuck in my head as the canonical pointer to... well, Green, or whatever one wants to call the thing Joe is gesturing at. When I first read the essay, my main thought was "Wow, Joe did a really good job pointing to a thing that I do not like. Screw Green. And thank you Joe for pointing so well at this thing I do not like.". And when the essay has come up in my thoughts, it's mostly been along similar lines: I encounter something in the wild, I'm like "ugh screw this thing... wait, why am I all 'ugh' at this, and why do other people like it?... oh it's Carlsmith!Green, I see". But rereading the essay today, I'm struck by different thoughts than a year ago. Green = Oxytocin Feeling? Let's start with the headline gesturing: [...] Today, reading that, I think "oxytocin?". That association probably isn't obvious to most of you; oxytocin is mostly known as the hormone associated with companionate love, which looks sort of like wholesomeness and happy villages but not so much like nature. So let me fill in the gap here. Most people feel companionate love toward their family and (often metaphorical) village. Some people - usually more hippie-ish types - report feeling the same thing towards all of humanity. Even more hippie-ish types go further, feeling the same thing toward animals, or all living things, or the universe as a whole. Generally, the phrase which cues me in is usually "deep connection" - when people report feeling a deep connection to something, that usually seems to cash out to the oxytocin-feeling. So when I read the post, I think "hmm, maybe Joe is basically gesturing at the oxytocin-feeling, i.e. companionate love". That sure does at l

I appreciate this post (still, two years later). It draws into plain view the argument: "If extreme optimization for anything except one's own exact values causes a very bad world, humans other than oneself getting power should be scary in roughly the same way as a papperclipper getting power should be scary." I find it helpful to have this argument in plainer view, and to contemplate together whether the reply is something like: 1. Yes 2. Yes, but much less so because value isn't that fragile 3. No, because human values aren't made of "take some utility function and subject it to extreme optimization," but of something else, e.g. looking for places where many different thingies converge, as with convergent instrumental utility (my own guess is something in this vague vicinity, which also gives me somewhat more hope that I might like some things about what autonomous AIs build if they go Foom) 4. ...?