This post inspired a pretty longrunning trail of thought for me I am still chewing on. I have considered pivoting my life to pursue the sort of vision this post articulates. I haven't actually done it because other things so far have seemed more urgent/tractable but I still think it's pretty important.

I've had this on my to-review log all review season, and I guess I'm getting this in mere hours before it closes.

What does this post add to the conversation?

The most important piece I think this adds is that the problem is not simple.

I keep seeing people run into one of these community conflict issues, and propose simple and reasonable sounding ways to solve it. I do not currently believe the issue is impossible, but it's not as simple as some folks think when they stare at it the first time.

How did this post affect you, your thinking, and your actions?

Con... (read more)

There's lots of low hanging fruit on Wikipedia. Now when I think "ugh, come on" or "that looks wrong" or "why doesn't this post have X??" etc. I either edit it immediately or write it down to edit (or procrastinate on editing) later. The most rewarding part is when I'm trying to recall something in a conversation, pull up a relevant Wikipedia page, scroll through to find the info... and then realize I'm the one that put it there.

This post and others like it was what got me to start editing. I imagine that somewhere out there is someone like my teenage self, reading Wikipedia and having a slightly easier time learning a bit more than I did.

"Armchair psychologizing about which of my rhetorical opponents' cognitive deficits cause them to fail to agree with me" is by far my least favorite kind of LessWrong post, and the proposed solution to the "problem" ("recruit smarter people to the field") is not interesting or insightful.

The median researcher hypothesis seems false. Something like an 80/20 distribution seems much more plausible, and is presumably more like what you'd find for measurable proxies of 'influence on a field' like number of publications in "top tier" journals, or number of researchers in the field who were your grad student. Voting "no".

I gave this a +4 because it feels pretty important for the "how to develop a good intellectual" question. I'd give it a +9 if it was better argued.

Here's a visual description: Imagine all worlds, before you see evidence cut into two: YEP and NOPE. The ratio of how many are in each (aka probability mass or size) represents the prior odds. Now, you see some evidence E (e.g. a metal detector beeping), so we want to know the ratio after seeing it.

Each part of the prior cut produces worlds with E (e.g. produces beeps). A YEP produces (Chance of E if YEP) amount of E worlds while a NOPE produces (Chance of E with NOPE).

And thus the new ratio is the product.

In case you don't know what odds are, they express... (read more)

This post is a good overview of near-term prospects for human biological enhancement, arguing that selection is more tractable than editing, and further that iterated selection has the potential to be exceptionally powerful. The biology is largely accurate as of the time of writing, although the claim "we can now do iterated meiosis" was premature. Since 2024 I've been working on refining my meiosis induction to improve epigenetics and also allow for iterated meiosis. As of early 2026, I've made significant progress but this still isn't ready for use in hu... (read more)

I think this is a great attempt at trying to create a causal model of enlightenment and I think it falls flat due to some very specific reasons.

I have a hard time putting my disagreements with this post into words but the disagreement is there and I think it is relatively substantial. The problem as I see it comes from the perspectives that are drawn upon for the explanation of meditative experiences.

It is a bit how we have that WEIRD https://www.theguardian.com/books/2020/nov/20/the-weirdest-people-in-the-world-review-a-theory-of-everything-study people s... (read more)

Due to Raemon's call for more reviews, "especially critical ones", I decided to read and review this one, so yes, I am interested in the topic. The post lists many interesting aspects of the discussed regulation. 

Nonetheless, I think the post would not be a good choice for the 2024 Review.

• Firstly, the post is a post about American politics and American policy, and I hope the perspective or target audience of LW is not such that is assumed that readers of LW posts naturally understand "we", "our failures", "we have required" etc as "American". 
• Sec

I found this to be the most concrete post from the feedbackloop-first rationality sequence. I really appreciated the empiricist sort of frame of actually going out and trying to do a simple toy experiment to test some assumptions. Hope is blinding, and I remember more often to double-check my thinking for it after reading this post. 

(Self-review.) I started this series to explore my doubts about the "orthodox" case for alignment pessimism. I wrote it as a dialogue and gave my relative non-pessimist character the designated idiot character name to make it clear that I'm just exploring ideas and not staking my reputation on "heresy". ("Maybe alignment isn't that hard" doesn't sound like a smart person's position—and in fact definitely isn't a smart person's for sufficiently ambitious conceptions of what it would mean to "solve the alignment problem." Simplicia isn't saying, "Oh, yeah, w... (read more)

This was beautifully written. I give it +4.

I like that it includes opposite examples. Is nature gentle, friendly, and harmonious? Or is it indifferent, hostile, and murderous? This is the wrong question to ask. 

A surprisingly large fraction of people I talk to try to convince me that some kind of irrationality is good actually, and my overly strong assumptions about rationality are causes me to expect AI doom. I fairly sure this is false and I'm making relatively weak assumptions about what it means to be rational. One assumption I am happy to make is that an agent will try to avoid shooting itself in the foot (by its own lights). What sort of actions count as "shooting itself in the foot" depends on what the goals are about, and what the environment is like, and... (read more)

I still think this post is pretty good and I stand by the arguments. I'm really glad Peter convinced me to work on it with him.

In Sections 1,2&3 we tried to set up consequentialism and the arguments for why this framework fits any agent that generalizes in certain ways.

There are relatively few posts that try to explain why inner alignment problems are likely, rather than just possible. I think one good way to view our argument in Section 4 is as a generalization of Carlsmith's counting argument for scheming, except with somewhat less reliance on intent... (read more)

In this post John describes a method by which functioning democracies can attempt to prevent tyranny of the majority - giving each major faction a de-facto veto over new legislation.

Whilst this is a method used in some countries, it is by no means the only, or indeed the most common, method for achieving this. I am only properly familiar with the UK as a counter-example but Dumbledore's army lists Frace, Germany, Italy and Canada as some others.

The method described in the post is likely more useful in situations where there are 2-3 major factions, whose va... (read more)

A neat progress update, though largely obsoleted by more recent posts (see below).

LessWrong is famously obsessed with the trainable skills and social practice of rationality, and with the prospect of very strong computer intelligence. The biological pathway to improved cognition doesn't get as much discussion, and the timelines to impact are (probably!) somewhat longer, but I remain convinced of its importance. I also strongly agree with the ethical position that

nobody’s civil rights should be in any way violated on account of their genetic code, and t

I consider this idea essential social technology on the level of "the map is not the territory." super basic, but it's everywhere once you learn to see it. I think about and make use of this concept on a weekly basis. Definitely seems worthy of consideration for Best Of to me. 

Unfortunately this post hasn't gotten much attention, but I do think it offers some value to the conversation. Its main weakness is that it's very hypothetical, and it would have been better if I could have made it more formal and concrete. Also I could have probably wrote it better. 

I sent the post to @abramdemski and @Gordon Seidoh Worley; Abram said it looks totally correct, and Gordon said it's thinking in the right direction.

I think this post makes an important observation which hasn't been made elsewhere (at least on LW) - even if any worldview ... (read more)

I found this post valuable for stating an (apparently popular on LW) coherentist view pretty succinctly. But its dismissal of foundationalism seems way too quick:

• It's not at all "pretty clear" to me that "any foundations are also subject to justificatory work". Foundationalist philosophers don’t pick arbitrary beliefs (or seemings/appearances/etc.) to call foundational. They try to identify foundations that seem intuitively to (1) not be the sort of thing that requires justification, and (2) provide justification for non-foundational beliefs.
  • Fair enou

"Superbabies or other cognitive enhancement" seems like one of the major projects going on on LW right now and seemed like this post was something I'd like to see reviewed.

I don't really feel able to do a good job of that myself. Maybe interested in a review from @GeneSmith , @TsviBT , @Zac Hatfield-Dodds or @kave. 

"Get everyone who could use one a Thinking Assistant" still feels like one of the higher order interventions available. It at the very least raises the floor of my productivity (by being accountable to someone for basically working at all). 

I have found it fairly hard to hire for the role, and I think the main problem is still "reliability", which IMO is more important than having any particular skills.

I've had a Thinking Assistant for the past year. One benefit I positively updated on is "it makes it easy to gain new habits." Instead of every new hab... (read more)

I've spent a lot of time figuring out how to implement this exercise, which I wrote up in The "Think It Faster" Exercise (and slightly more streamlined "Think it Faster" worksheet).

I've reviewed it more thoroughly over there.

So I think this post is pointing at something very important for my personal rationality practice, but it gives me almost none of what I need to actually do it successfully. 

I would not normally vote on this post, as the technique of "How could I have thought that faster?" seems extremely obvious to me but also very important if you are not in fact trying to improve your thinking after being surprised (or any other shortcoming). Since this post has 241 upvotes and multiple comments from people (example: Said Achmiz, who is not an idiot!) and others disagreeing with the framing, I have review-upvoted this post.

I think the framing of "think it faster" is specifically something you should track, beyond just "What did I learn here... (read more)

Most people (possibly including Max?) still underestimate the importance of this sequence.

I continue to think (and write) about this more than I think about the rest of the 2024 LW posts combined.

The most important point is that it's unsafe to mix corrigibility with other top level goals. Other valuable goals can become subgoals of corrigibility. That eliminates the likely problem of the AI having instrumental reasons to reject corrigibility.

The second best feature of the CAST sequence is its clear and thoughtful clarification of the concept of corrigibili... (read more)

One of the most accurate classification and description of how to recover from procrastination. When I first read this post I was already very impressed by how much I relate to.

However, this post isn't quite actionable for me and I have never remembered to read this post when I am in a laziness death spiral.

This still qualitatively rings true, and this insurance frame helped me understand human relationships better. I always found it kind of game theoretically wrong for people to stay with their extremely sick husband/wife, but this post changed my mind. It would be interesting to see some discussion of relationships that are partly transactional, partly insurance.

I like this a lot. Finding good tacit knowledge videos is difficult in the current internet, and this post managed to successfully create a Schelling point in LW for sharing such information.

Scott Alexander has an argument (You Have Only X Years To Escape Permanent Moon Ownership) which seems partly directed against this post. I'm still siding with Rudolf.

Scott's argument depends more than I'm comfortable with on expectations that the wealthy will be as altruistic toward distant strangers. I expect that such altruism depends strongly on cultural forces that we're poor at predicting. I expect that ASI will trigger large cultural changes. Support for such altruism seems fragile enough that it seems like a crap-shoot whether it will endure. I fin... (read more)

To be honest, I now think the post has far less value than people thought in in 2024, and this is not because it's factual statements are wrong, but rather that Richard Ngo and a lot of LWers way underestimated how intractable it was to avoid polarization of AI safety issues in worlds where AI safety is highly salient politically without deep changes to our electoral systems.

(To be clear, this is a US-centric view, but this is justified given that the by far most likely countries to have ASI in their borders initially are the US and China, due to both comp... (read more)

My fear of equilibrium

Carlsmith’s series of posts does much to explore what it means to be in a position to shape future values, and with how to do so in a way that is “humanist” rather than “tyrannical”. His color concepts have really stuck with me; I will be thinking in terms of green, black, etc. for a long time. But on the key question addressed in this post — how we should influence the future — there is a key assumption treated as given by both Lewis and Carlsmith that I found difficult to suspend disbelief for, given how I usually think about the fu... (read more)

In this post and its successors Max Harms proposes a novel understanding of corrigibility as the desired property of the AIs, including an entire potential formalism usable for training the agents to be as corrigible as possible. 

The core ideas, as summarized by Harms, are the following:

I like the title of this post! The content of the post isn't bad. 

This was supposed to be a grand post explaining that belief. In practice it’s mostly a bunch of pointers to facets of truthseeking and ideas for how to do better.

I want the grand post! (I want clear articulations of the thing I feel is true and important.) Especially after you point out that it might have been.

The points in the post aren't bad, though it feels like fewer examples in greater depth that I could better memorize would have more value than a lot of short ones. I think the al... (read more)

This is a good post. The examples are clear and it deepened my intuition (though I'm judging from the reread, I don't remember the delta from before my first reading). From the second-read, I think I might notice more instances in the wild of adverse selection, though I don't think the first read had much impact on me.

The intended subsequent posts look really great and like they'd have interesting models I don't yet have. I think I had the concept of adversarial selection before this, so wasn't a conceptual breakthrough.

Then again, maybe the title should h... (read more)

This post is an insightful attempt to explore a novel issue of kids whom parents are to prepare for a world drastically more different from the previous eras than the eras are from each other. 

Before reading the post, I didn't think much about the issue, instead focusing on things like existing effects on kids' behavior. What this post made me do is to try and reason from first principles, which also is a way to test the post's validity. 

I found this essay frustrating. Heavy use of metaphor in philosophy is fine as long as it's grounded out or paid off; this didn't get there.

Edit: Okay, I went back and reread the entire sequence. it did get places, but in very low ratio to its length. And the overall message between the many lines in this particular post, "green is something and it's cool", is not one that's well-argued or particularly useful. Is it cool, or does it just feel cool sometimes? Joe doesn't claim to know, and I don't either. And I'm still not sure green is a coherent thing wor... (read more)

I appreciate this post (still, two years later). It draws into plain view the argument: "If extreme optimization for anything except one's own exact values causes a very bad world, humans other than oneself getting power should be scary in roughly the same way as a papperclipper getting power should be scary." I find it helpful to have this argument in plainer view, and to contemplate together whether the reply is something like:

1. Yes
2. Yes, but much less so because value isn't that fragile
3. No, because human values aren't made of "take some utility function and

The main takeaway from the post is Zvi's concept of levels of friction, which he developed later. As of the time of writing the post, Zvi had in mind the following:

I am coming around to a generalized version of this principle. There is a vast difference between:

1. Something being legal, ubiquitous, frictionless and advertised.
2. Something being available, mostly safe to get, but we make it annoying.
3. Something being actively illegal, where you can risk actual legal trouble.
4. Something being actively illegal and we really try to stop you (e.g. rape, murder).

We’ve pla

I think this was a badly written post, and it appropriately got a lot of pushback.

Let my briefly try again: clarifying what I was trying to communicate.

Evolution did not succeed at aligning humans to the sole outer objective function of inclusive genetic fitness.

There are multiple possible reasons why evolution didn't succeed, and presumably multiple stacked problems.

But one thing that I've sometimes heard claimed or implied is that evolution couldn't possibly have succeeded at instilling inclusive genetic fitness as a goal, because individuals humans don'... (read more)

This post provides important arguments about what goals an AGI ought to have.

DWIMAC seems slightly less likely to cause harm than Max Harms' CAST, but CAST seems more capable of dealing with other AGIs that are less nice.

My understanding of the key difference is that DWIMAC doesn't react to dangers that happen too fast for the principal to give instructions, whereas CAST guesses what the principal would want.

If we get a conflict between AIs at a critical time, I'd prefer to have CAST.

Seth's writing is more readable than Max's CAST sequence, so it's valuable to have it around as a complement to Max's writings.

This still seems like a valuable approach that will slightly reduce AI risks. This kind of research deserves to be in the top 10 posts of 2024.

When this post first came out, I felt that it was quite dangerous. I explained to a friend: I expected this model would take hold in my sphere, and somehow disrupt, on this issue, the sensemaking I relied on, the one where each person thought for themselves and shared what they saw.

This is a sort of funny complaint to have. It sounds a little like "I'm worried that this is a useful idea, and then everyone will use it, and they won't be sharing lots of disorganised observations any more". I suppose the simple way to express the bite of the worry is that I w... (read more)

I appreciate this post, as the basic suggestion looks [easy to implement, absent incentives people claim aren't or shouldn't be there], and so visibly seeing if it is or isn't implemented can help make it more obvious what's going on. (And that works better if the possibility is in common knowledge, eg via this post).

This visual reduces mental load, shortens feedback loops, and effectively uses visual intuition.

Before, my understanding of Shapley values mostly came from the desirable properties (listed at the end of this article). But the actual formula itself didn't have justification in my mind beyond "well, it's uniquely determined by the desirable properties". I had seen preceding justifications in terms of building up the coalition in a certain sequence, taking the marginal value a player contributes when they join, and then averaging over all ways coalition could... (read more)

I read this and think "ah, yes, this is valuable and important and I should be trying to do that more". And thought as much when I first read it. I don't think it stayed on my mind. It's too compressed and not a ready a cognitive strategy.

But taking a few moments to extrapolate it into something better, starting with why I'm not doing it to begin with:

• A reason I don't do more of this is I can't do this on the order of 30 seconds. My guess is constructing a picture of what mental operations I did and what I could have done alone is the work of many minutes.

I think this post does a good job of conveying the challenges here, grounded in actual cases. (It's hard for me to evaluate whether it does a great job because my pre-existing knowledge on the topic.) I think this stuff is hard and I have so much sympathy for anyone who's been caught up in it, if they're weren't the instigator.

I don't feel convinced it's impossible to do this much better. My own median world isn't very fleshed out, but my gut tells me that dath ilan has figured out some good wisdom and process here, and I trust it. I'd also guess that if L... (read more)

I think this is a valuable post. I say that less engaging with the specific ideas (they all seem like plausibly correct analyses to me), but for exploring the problem at all.

1. There's a societal taboo against discussions of intelligence and IQ that although it is much weaker on LessWrong, I wonder if it is not completely absent, and therefore we don't get that many posts like this one.

2. I often feel annoyed and judgmental that broader society doesn't clamor for longevity increases – it's seems so correct to think these are possible and important. Reading... (read more)

Feels like this points at correct things and I'm amenable to it being one of the top posts for 2024. It didn't change much for me (as opposed to @Ben Pace, who thinks about it many times per month according to his review) or feel so spot on that I'd want to give a high vote. I'll probably give something between 1-4.

Areas where I think it strikes me (admittedly with not that much thought or careful reading) as not perfectly right:

Notwithstanding the heading contra this, my instinct to want to reduce "believing in" statements to a combination of "I believe (... (read more)

This post makes a brave attempt to clarify something not easy to point to,  and ends up somewhere between LessWrong-style analysis and almost continental philosophy, sometimes pointing toward things beyond the reach of words with poetry - or at least references to poetry. 

In my view, it succeeds in its central quest: creating a short handle for something subtle and not easily legible.

The essay also touches on many tangential ideas. Re-reading it after two years, I'm noticing I've forgotten almost all the details and found the text surprisingly lo... (read more)

A year later: If you're going to do predictions, it's obviously IMO better if they are based around "what would change your decisions?". (Otherwise, this is more like a random hobby than a useful rationality skill)

And, it's still true that it's way better to be fluent, than not-fluent, for the reasons I laid out in this post. (I.e. you can quickly interweave it into your existing planmaking process, instead of clunkily trying to set aside time for prediction)

The question is "is it worth the effort of getting fluent?" 

When I first started writing this ... (read more)

This post helped me personally in two ways.

1. Recognizing that even picking 4 things to focus on is too much. And that focusing on only 1 or 2 (at least at any specific time) would be exponentially more effective. In this sense it served as a nice complement to the book “4000 weeks”.

2. Consciously splitting my time between exploring and exploiting allowed the exploration to be more free. I allowed myself to try things I otherwise may not have, by not feeling like I needed to commit to any of the explorations as itself being most worth doing.

An added bonus in reading this essay is that the prose is a pleasure to read.

I didn't pay this post much attention when it came out. But rereading it now I find many parts of it insightful, including the description of streetlighting, the identification of the EA recruiting pipeline as an issue, and the "flinching away" model. And of course it's a "big if true" post, because it's very important for the field to be healthy.

I'm giving it +4 instead of +9 because I think that there's something implicitly backchainy about John's frame (you need to confront the problem without flinching away from it). But I also think you can do great a... (read more)

This essay explained an idea which I think was implicit in many parts of the sequences but which I didn’t successfully identify or understand before now. It filled a gap that was one of the main reasons that I had difficulty in understanding and coming to my own conclusions about this worldview. It also provided a philosophical perspective in which I could rethink certain aspects of AI existential risk.

"Inventing Temerature" is excellent. It has helped me better understand the process and problems of attaining knowledge. It was also helpful in pointing to how to recognize gaps in my own theories and accepted paradigms. It would be nice to have a complementary work which translates these ideas into a practical toolbox but even on its own it still is helpful.

The book also showed why studying philosophy is lacking if it isn’t complemented with a study of the history of science and other ideas. (A gap in my education which I am still trying to remedy.)

The main problem of this review is noted within. Namely, that this is not the kind of area where reading a review will give you most of the benefit of the book itself.

I’m not sure to what degree this post can stand alone apart from the whole sequence. But in general I found the naturalism method a useful tool for understanding the world. (Is there reason that the Lesswrong review isn’t also done at the sequence level in addition to the post level)

Within the sequence, this post in particular stood out. Many people describe the importance of sitting through a period of being stuck without abandoning a project since often that is a stage towards clarity. This post pointed to a general actionable strategy for doing so which... (read more)

"On Green" is one of the LessWrong essays which I most often refer to in my own thoughts (along with "Deep Atheism", which I think of as a partner to "On Green"). Many essays I absorb into my thinking, metabolize the contents, but don't often think of the essay itself. But this one is different. The essay itself has stuck in my head as the canonical pointer to... well, Green, or whatever one wants to call the thing Joe is gesturing at.

When I first read the essay, my main thought was "Wow, Joe did a really good job pointing to a thing that I do not like. Sc... (read more)

I continue to use roughly this model often, and to reference it in conversation maybe once/week, and to feel dissatisfied with the writeup ("useful but incorrect somehow or leaving something out").

I really hope this post of mine makes it in. I think about "believing in" most days, and often reference it in conversation, and often hear references to it in conversation. I still agree with basically everything I wrote here. (I suspect there are clearer ways to conceptualize it, but I don't yet have those ways.)

The main point, from my perspective: we humans have to locate good hypotheses, and we have to muster our energy around cool creative projects if we want to do neat stuff, and both of these situations requires picking out ideas worthy of our atten... (read more)

This post is entertaining and was valuable for describing to me a group of people with whom I never interact (highly incompetent liars), but not all that useful given that I never interact with such people. I don't think I especially need an existence proof for lying at all; I do think it'd help to get a post about examples of lying that are closer to what I'd encounter, or at that sophisticated enough to pass if you're too credulous.

I have a feeling of distaste for this post from an unusual angle: when we were first introducing a recommendation engine, ou... (read more)

The first concept in this post really stuck with me, that of computational kindness vs <whatever the kindness of letting the other choose is>. The OP writes they got it from elsewhere, but I appreciate it having made it to me.

I'd really love it if it had a better solution of how to pick between kindnesses as I can find myself wondering which is more preferred.

The other concepts are great too. They hadn't stuck in my mind from original reading but perhaps will now.

I really wouldn't mind more posts just providing me with useful handles like this, so good stuff.

The review I have of this post is much the same as I left oh John's other delta post, My AI Model Delta Compared To Christiano:

speaking to the concept of deltas between views

I find that in reading this I end up with a better understanding of Paul and John (to the extent John summarized Paul well, but it feels right). Double-crux feels like a large ask given the hunt for a mutually shared counterfactual change that just seems like a lot to identify; ideological turing test means someone putting aside who they are and what they think too much; but "deltas" f

I disagree with your take.

Suffering is quite unlike shit in that once we get rid of shit, the shit you got rid of does not come back, crawling up the toilet. Suffering is not some "thing" you can get rid of, but rather a quirk of our neurophysiology. Get rid of the most immediate cause of suffering, and your brain adjusts its thresholds to seek the next worst thing; this is called upregulation.

Two related real-life examples: if you are walking in an uncomfortable shoe, you are aware of the uncomfortable shoe. Step on a bad thorn that pierces your shoe and ... (read more)

REVIEW
LessWrong is stuck in looped ways of being and thinking, and this was AN attempt at opening the door a bit, but I have the sense it wasn't a very effective attempt. 

I still appreciate the attempt. I'm not that good at meeting LW where it is, but I care to try.  

I am now thinking that using fiction or story would maybe be a better avenue. 

The main sticking points seem to be

• how does letting go actually make someone more effective (does the 'effectiveness' need to be legible to lots of ppl? I dont think so personally, but effectiveness ca

A fascinating experiment about short-term memory loss. 
Gives me some ideas I'd like to try if I ever get affected so.

This post is somewhat of a description of a test run of Lesswrong values and lessons, which is very important for showcasing these values, somewhat explaining how they work, and giving people concrete ideas about what to look for.

I've certainly read this post more than once. The advice described really does seem to be pretty solid and while it may sound somewhat obvious (use all your various skills to work on [thing] instead of using one or two skills while ignoring the rest because redirecting skills is fairly doable), the general picture isn't something I hear every day (or really any day).

Most questions that people ask you sound like "who do you want to be when you grow up", which I guess gives one a sense of needing to figure out a specific thing (be a doctor) and it's a little... (read more)

This is a fascinating article that I found interesting. But it is not something I'll likely be a part of, ever, and neither is most of the community (judging by the "profession" question of the 2024 census - Medicine (being the closest field) with 7 people - this article isn't particularly useful. Judging by the 2023 census, with 20 people across Biology and Medicine, this article is more useful but still doesn't quite compare).

And since most of the community doesn't seem to deal or be part of this field, and since it talks about an uncertain threat far in... (read more)

I like this post - it's a good reminder of what we can quite easily do better, paired with a personal anecdote that makes it feel all the more achievable.

I feel inordinately proud of this post, probably because this was a problem that I’ve been confused about since 2019, and I literally taught myself neuroscience in large part because I wanted to solve this problem, and I spent what amounts to several years of full-time effort building up an ability to tackle it … and this post represented the moment when I finally felt like I had my foot in the door towards a satisfying solution.

Granted, there’s still plenty more work to do, and indeed I’ve continued to follow up on this work in the past year since I wrote... (read more)

This is easily the most underrated AI security post of 2024, mostly because it points out that, unlike most files a defender has, AI weights are very, very large files, and that this means you can slow down exfiltration attempts enough to require physical attacks to be used.

It also inspired follow-up work like Defending Against Model Weight Exfiltration Through Inference Verification to achieve the goal of preventing model exfiltration.

The big question that will need to be answered is whether or not preventing exfiltration of model weights is positive EV, ... (read more)

I think this is a good and important post, that was influential in the discourse, and that people keep misunderstanding.

What people engaged with

What did people engage with? Mostly stuff about whether saving money is a good strategy for an individual to prepare for AGI (whether for selfish or impact reasons), human/human inequality, and how bad human/human inequality is on utilitarian grounds. Many of these points were individually good, but felt tangential to me.

What I was trying to write about

But none of that is what I was centrally writing about. Here is... (read more)

I have very mixed views about this, as someone who is myself religious. First, I think it's obviously the case that in many instances religion is helpful for individuals, and even helps their rationality. The tools and approaches developed by religion are certainly valuable, and should be considered and judiciously adopted by anyone who is interested in rationality. This seems obvious once pointed out, and if that was all the post did, I would agree. (There's an atheist-purity mindset issue here where people don't want to admit "The Worst Person You Know J... (read more)

These posts are about transformative psychological experiences that leave a person with a different sense of meaning in their lives, or ethics, or attitudes toward people / the world. That seems to me to be a real part of the world and human experience, and I think it's worthwhile exploring it and trying to understand its causes and what implications it may have for ethics/meaning/etc.

I think it's worth exploring religions as having been shaped substantially by some transformative psychological experiences, and purporting to bring them to adherents. It see... (read more)

"Yudkowskianism" is a Thing (and importantly, not just equivalent to "rationality", even in the Sequences sense). As I write in this post, I think Yudkowsky is so far the this century's most important philosopher and Planecrash is the most explicit statement of his philosophy. I will ignore the many non-Yud-philosophy parts of Planecrash in this review of my review, partly because the philosophy is what I was really writing about, and partly to avoid mentioning the mild fiction-crush I had on Carissa.

There is a lot of discussion within the Yudkowskian fram... (read more)

I think this makes a very important point, related to both AI Alignment and AI Welfare. I wish it had got more attention and discussion, so I'm going to nominate it.

I continue to like this post. I think it's a good joke, hopefully helps make more sticky in people's minds what muddling through is, and manages some good satirical sociopolitical worldbuilding. However, I admit in the category of satirical AI risk fiction it has been beaten by @Tomás B. 's The Company Man , and it contains less insight than A Disneyland Without Children

In retrospect, I think this was a good and thorough paper, and situational awareness concerns have become more prevalent over time. If I could go back in time, I would focus much more on the stages -type tasks, which are important for eval awareness, which is now a big concern about the validity of many evals as models are smarter, and where I think much more could've been done (e.g. Sanyu Rajakumar investigated a bit further). As usual, most of the value in any area is concentrated in a small part of it.

Many people seem to think the answer to the puzzle posed here is obvious, but they all think it's something different. This has nagged at me since it was posted. It's an issue that more people need to be thinking about, because if we don't understand it we can't fix it, and so the standard approaches to poverty may just fail even as our world becomes richer. Strong upvote for 2024 review.

There's a lot I don't like about this post (trying to do away with the principle of indifference or goals is terrible greedy reductionism), but the core point, that goal counting arguments of the form "many goals could perform well on the training set, so you'll probably get the wrong one" seem to falsely imply that neural networks shouldn't generalize (because many functions could perform well on the training set), seems so important and underappreciated that I might have to give it a very grudging +9. (Update: downgraded to +1 in light of Steven Byrnes's... (read more)

I’m a big fan of this series and voted for all of its posts for the 2024 best-of list. I think Joe is unusually good at understanding and conveying both sides of a debate, and I learned a lot from reading and pondering his takes.

I’ve read many of the posts in this series multiple times over the past 2 years since they came out, and I bring them up in conversation on occasion (example).

For what it’s worth, I tend to be in agreement with the things the series argues for directly, while disagreeing with some of the “anti-Yudkowsky” messages that the series in... (read more)

This post is great original research debunking a popular theory. It definitely belongs on the best-of-2024 list.

Great illustration of why I’ve always paid just as much attention (or more) to nitpicky blog posts by disagreeable nerds on the internet, than to big-budget high-profile journal articles by supposed world experts. (This applies especially but not exclusively to psychology.)

(Caveat that I didn’t check the post in detail, but I have other independent reasons to strongly believe EQ-SQ theory is on the wrong track.)

I'm pretty happy with this piece. I often look back at things I wrote a year or more prior and find I'd like to approach the whole thing very differently. But I don't feel that way at all here. I often want to point people to it as-is, usually without any caveats.

(The part I'd most rewrite is the section on how to use the framework to sometimes loosen Newcomblike self-deception. I've guided more people through that process since writing this article, and I've learned a few things about what's helpful for folk to hear. But even that part doesn't need substa... (read more)

I re-read this series in honor of the 2024 lesswrong review. That spurred me to change one of the big things that had been bugging me about it since publication, by swapping terminology from “homunculus” to “Active Self” throughout Posts 3–8. (I should have written it that way originally, but better late than never!) See Post 3 changelog. I also (hopefully) clarified a discussion in Post 3 of how preferences wind up seeming internalized vs externalized, by adding a figure, and I also fixed a few typos and so on along the way.

Other than that, the series is ... (read more)

This was very important. I don't think the specific example is timeless or general enough to belong in a Best Of 2024 collection, but the fact that prediction markets can behave this way - not just in theory, but in practice, in such a way that it potentially alters the course of history - is a big deal, and worth recording.

(Something the OP doesn't mention is the way this effect recurses. The people who shifted probability on prediction markets thereby shifted probabilities in real life, and thus ended up with more money.)

As someone who spent an unreasonable chunk of 2025 wading through the 1.8M words of planecrash, this post does a remarkable job of covering a large portion of the real-world relevant material directly discussed (not all - there's a lot to cover where planecrash is concerned). I think one of the main things lacking in the review is a discussion of some of the tacit ideas which are conveyed - much of the book seems to be a metaphor for the creation of AGI, and a wide range of ideas are explored more implicitly.

All in all, I think this post does a decent task of compressing a very large quantity of material.

Out of all the LW posts I read in 2024, I think this one was the most beneficial to my daily life. As a review of this article, I think it might be useful to link it to the current literature on communicating information from one person (often a teacher) to the other (often a student); I think it "fills in the blank" where I previously struggled to implement Raemon's knowledge.

I'm currently doing a masters degree in education, as such I think a useful contextual addendum to this article - with the goal of improving the required skill of "Modelling Others" ... (read more)

This is well-executed satire; the author should be proud.

That said, it doesn't belong among the top 50 posts of 2024, because this is not a satire website. Compare to Garfinkel et al.'s "On the Impossibility of Supersized Machines". It's a cute joke paper. It's fine for it to be on the ArXiV. But someone who voted for it in a review of the best posts of 2017 in the cs.CY "Computers and Society" categorization on ArXiv is confessing (or perhaps, bragging) that the "Computers and Society" category is fake. Same thing with this website.

On closer inspection, I believe this does not add much towards understanding the described people's psychology. 

Although the described reactions seem accurate, the analogy seems week and the posts jumps too quickly towards unflattering conclusions about the outgroup. In particular, the case of being forcibly moved by a company towards another location is an extremely radical action given our current social norms and thus people can be expected to be indignant.

On the other hand, organizations imposing large but longer term changes on societies without asking is the norm. such as introducing social media or the internet.  

I didn't manage to read this in 2024, but I was linked to it from Anna's end-2025 CFAR posts and reading it now, a bunch of lightbulbs went on that I wasn't yet ready for in 2024.

I did experience something like burnout around then, am now much more recovered, and I resonate a lot with especially the beginning--the idea that orgs need to be paying attention to the world, that there aren't very good forcing functions for them doing so, and that burnout is the pushing-on-a-string feeling caused by that, and recovery looks like putting yourself in places where the string has tension.

The model seems incomplete somehow, I'm not exactly sure what is missing from it, but regardless it is useful to me and resonant.

+4. This post stuck with me distinctly from 2024, and often when I mention it to people they have already read it. I think that during Covid, challenge trials seemed like a mystical thing from fantasy novels that would happen in dath ilan or some superior civilization. So it was a pleasant surprise to read an actual account of one from someone I already know, especially by a biologist who could teach me about this basic disease as well.

+4. In some regards it's sad to have to rehash this argument, but I feel that this argument has been going around in the public discourse, and so it's worthwhile to write up a thorough account of what's naive about it and how to move past it. My sense is that it has become less prevalent since the essay; perhaps the essay helped.

Many folks have distaste for Eliezer's style, or for perhaps implying that a weak-man argument is fully representative of positions he disagrees with; I think some of these criticisms are valid but do not mean the essay isn't (a) p... (read more)

+4. I recall learning about dimensional analysis as a teenager and it's still a basic element of my thinking, though I should practice it more. Fermis too. Anyway, this is a fantastic little explainer and fun introduction to these fundamental ways of thinking, I'm pretty confident it should make the top 50 list for the year. 

And, man, I wish LessWrong caused me to practice doing these sorts of arithmetic more often.

Plausibly the largest update to the foundations of my epistemology I've had since reading the sequences. Neat.

Feels like it captures and grounds out something critical about how to reason in a complex and uncertain domain without resorting to deference. Spot checking lots of parts of a world model and finding them consistent is genuine though not strong evidence of the overall conclusion, because they could have been inconsistent.

It's neat how this bootstraps from nothing, even if for high confidence you want other forms of evidence too.

Excellent post, plausibly the most rigorous explanation of the core reasons to expect doom, but really really needs a more memetic handle. Actually suggest the authors go back and pick one even now, perhaps "Catastrophic Misalignment is the Default", and make the current title a subtitle.

This post emphasizes that truthseeking is the foundation for other principles, and that you cannot costlessly deprioritise truthseeking. The post gives practical advice how to consciously apply the principle of truthseeking. It also encourages positive behavior like pushing back against truth-inhibiting behavior. In some parts, the article seems to me not to be fully focused and Elizabeth herself writes "In practice it’s mostly a bunch of pointers to facets of truthseeking and ideas for how to do better", but overall I think these pointers are very valuable.