The LessWrong 2019 Review is upon us. We're out of the nomination phase, and starting the review phase. 121 posts have passed into this second stage. 

Now we join forces to answer some crucial questions: 

  • Are the claims in these posts true?
  • Are they important?
  • Do we want to reify them in the LessWrong canon, as we did with winners of the 2018 Review?

If you've wanted to help in this endeavor, but haven't gotten around to it... then maybe this is for you. 

Eat that frog

Sometimes engaging with LessWrong is pure fun. Good opinions just keep coming to you. You ride a wave of excitement. 

Other times, however, you need to roll up your sleeves and Eat That Frog. 

In an inspirational haze, we spent the wild night of 2019 producing a ton of content. Now we have woken up the morning after. With the dry lucidity of hindsight, time has come to atone for our sins. Or celebrate our victories. 

One of the most beautiful things about LessWrong is our joint vision of a better way to do Science on the internet. We know that Facebook, Twitter, and the journal system are utterly broken. We know that, truly, so much more is possible.  

But, well, if we want good things to happen to us, we need to put in the work. 

All those things that seemed a bit schlepy, and like maybe-it-would-be-good-if-someone-but-not-me-did-them-some-day?

  • Operationalising fuzzy claims
  • Fact-checking and doing literature reviews
  • Putting in that effort to go "no, but actually, does this stuff hold up? Have I really turned over every stone in this argument, and not just spent time playing in the exciting corners?"
  • Making spreadsheets and Guesstimates
  • Making other models and simulations
  • Running model validations, sensitivity analyses
  • Doing replications
  • Running MTurk surveys
  • Formalising and checking proof steps
  • Spending long, meandering, 10h conversation with a burning curiosity to really know whether this essay is actually true, and refusing to accept a "maybe"
  • ...and more!

Now is the time. 

How to participate

Write a comment and commit to what you want to accomplish this Review season. When the end date comes, I will follow up on your comment to ask you how it went. It is expected you'll reply with a brief report (can be as short as a sentence).

I'll probably celebrate things that seem like victories, but I don't plan to give particular negative reinforcement otherwise. I trust that the act of committing and following up in public will handle that on its own.

Here are examples of commitments: 

  • I want to have fact-checked at least 10 claims across at least 3 posts, by January 1st.
  • I want to have spent enough time on at least 2 posts, that I feel I grok what they're trying to say and have formed my own strong model of whether they're right, by January 7th.
  • I want to have operationalised claims into interactive prediction questions that have gotten a total of 50 predictions, by January 11th.
  • I want to be able to say, in full honesty, true to my heart, swearing on what I hold dear, that I made tried my best to contribute to this Review, by January 8th.

Important disclaimer

This is not for everyone.

Some people hate public commitments. If so, don't join. It's cool. 

For others, like me, public accountability works pretty great. Knowing that people are watching just makes me want to be the best version of myself that I can. 

What's more, different people relate to the Review in different ways. If you work best being guided by motivations of pure curiosity and excitement, then, by all means, you do you. Don't mind this post and don't let it seep guilt into whatever mojo you've got going. :) 

To all, I wish you the best of skill. 

New Comment
6 comments, sorted by Click to highlight new comments since: Today at 3:34 PM

My commitments: 

  • Operationalize three forecasting questions
  • Run one MTurk/Positly survey
  • Have at least one 2h conversation about a particular post, and write up a review after, almost regardless of how I feel the conversation went

By January 11th. 

(Some of these are derived with an eye to experimenting with different onsite review methods, in addition to just reviewing)

I'm starting with these, and might add more later. 

I failed to meet all my commitments. 

Operationalize three forecasting questions

Smashed this one and created 20+ questions. 

Run one MTurk/Positly survey

I have a beginning draft of a survey for the Secret of Our Success. I hoped I could finish it up yesterday, but instead I had work on shipping the LessWrong Books. Will see if I can get it out later this week. 

Have at least one 2h conversation about a particular post, and write up a review after, almost regardless of how I feel the conversation went

Didn't happen and didn't really come close. 


My main post-mortem is that I had multiple calendar reminders about the commitment, but for all of them I postponed them into the future. Until it was the last weekend and I was out of time. I should've spent more meta-cognition during some of them, thinking about how much time I would need to complete the tasks on time. 

I want to write up at least two more reviews by January the 11th.

Following up on this: how did it go? 

I ended up writing 3 more reviews. They each took about an hour to write. I'm reasonably happy with them given those constraints - they could be better if I spent a day on each, but I think I hit the right part of the trade-off curve. I'd expect the reviews of Olah's post and the deep double descent post to be pretty useful for non-expert readers, telling them things they didn't already know. I'm a little bit sad that they didn't get somewhat more acclaim, but such is life.

Nice job. 

Reviews seem to me to have a lower karma on average than either posts, or comments on currently popular posts.