Voting Phase of 2018 LW Review

[-]gjm6y160

The posts available for review are presented in (what I guess is) a consistent order that is (so far as I know) the same for everyone. I expect this to mean that posts presented earlier will get more votes. If, as seems plausible, most of the things in the list aren't bad enough to get a "no" vote from most voters, this means that there is a bias in favour of the earlier posts in the list.

Related: keeping track of which posts I've looked at is a bit of a pain. Obviously I can see which ones I've voted non-neutral for, but there's no obvious way to distinguish "decided to stick with neutral" from "haven't looked yet". So long as the order of presentation is consistent, I can just remember how far through the list I am, but (see above) it's not obviously a good thing for the order of presentation to be consistent. And this phenomenon incentivizes me to process posts in order, rather than deliberately counteracting the bias mentioned above by trying to look at them in a random order.

[-]jimrandomh6y100

The list is now shuffled (as a tiebreak after sorting by your own vote). The shuffle is done once per user, so each user should see the posts in a random order, but it'll be the same order each time you revisit it. This change went live around the 13th.

[-]Ruby6y80

[EDIT: When I say that posts earlier in the list got 25-50% more votes, I mean simply the number of non-neutral votes cast on those items, regardless of direction of magnitude. It would perhaps be more accurate to say these posts had 25-50% more people vote on them.]

The posts available for review are presented in (what I guess is) a consistent order that is (so far as I know) the same for everyone. I expect this to mean that posts presented earlier will get more votes.

Good call. I looked into this and found an effect of somewhere between 25-50% more votes for posts being displayed earlier in the list. The team rolled out a fix to randomize loading this morning.

Interestingly, the default sort order was by number of nominations in ascending, so the most heavily nominated (approx, the most popular) posts were being displayed last. These posts were getting as many votes as those at the beginning of the list (though possibly not as many as they might have otherwise), and it's the posts in the middle that were getting less.

This was an oversight which we're glad to have caught. We've around halfway through the voting, plus the second half will have the deadline rush, so hopefully this bias will get countered in the coming week.

Unfortunately you just make mistakes the first time you're doing things. :/

[-]ESRogs6y150

I voted!

[-]Bird Concept6y130

I voted in category mode, and am some way through fine-tuning in quadratic mode.

[-]gjm6y100

Some of these posts have almost no content in themselves but link to other places where the actual meat is. (The two examples I've just run into: "Will AI see sudden progress?" and "Specification gaming examples in AI".)

Are we supposed to vote on these as if they include that external content? If the highest-voted posts are made into some sort of collection, will the external content be included there?

[-]DanielFilan6y100

One wish I have related to the voting is that either there were much fewer posts that got to the voting stage, or I could choose to just vote on a subset of posts, rather than all of them. As it is, I'm somewhat intimidated by the prospect of reading 75 posts to decide how to vote on each one.

[-]Ben Pace6y*80

You don't have to vote on all posts, you don't have to have read all the posts. I think it's fine for the vote to correlate with what posts people actually read. The default vote on all is ‘neutral’, which equals zero quadratic votes. I voted on about half of the posts, and I think I read way more than most people.

Edit: Rewritten.

[-]Scott Garrabrant6y80

Is it pro-social or anti-social to vote on posts I have skimmed but not read?

[-]habryka6y120

I think it would make sense if you weakly vote on them, by spending relatively few points of your quadratic budget on them. Voting very strongly on them feels wrong to me. Basically, vote in strength proportional to your confidence times the goodness/badness of your assessment of the post, would be my guess.

[-]Ben Pace6y60

+1 I have voted on a number of posts that I've mostly skimmed, but not voted with much weight.

(Quadratic voting makes the first few votes just very cheap, which was one part of my reasoning.)

[-]Vaniver6y40

I also note that I think there's signal in your decision to only skim a post, as opposed to reading it, but as noted in habryka's response, it's probably a weak signal.

[-]Raemon6y40

I think it's a particularly weak signal when you're trying to evaluate 75 posts at once.

[-]DanielFilan6y60

Have finished fine-tuning my votes.

[-]Ben Pace6y20

Woop! I did the same yesterday.

[-]Raemon6y20

woo!

[-]Matt Goldenberg6y30

Maybe a bit late to ask about this, but I realized how a vote really depends on the information you guys have from the votes (seeing as the votes are really used just to give a signal, and aren't actually deciding anything).

Do you see all the votes, along with the user names?

Do you see each users' votes, but anonymized?

Do you see only the final count, and not the votes?

[-]Raemon6y40

We see information about how much individuals vote (i.e. total amount of points you’ve assigned and how many posts you’ve voted on), but not the details of which votes.

The current plan is something like: carve up vote totals from a couple different cross sections (i.e. check if there’s a difference between how the highest karma users voted vs the overall vote score, and probably checking what AF users thought about AF posts). I expect us to make those totals public.

I’m not sure about all the ways we will look at data but I expect the general principle will be “look for ways to to get insights about how people used it, while keeping everyone anonymized”.

[-]Ruby6y70

We see information about how much individuals vote

For accuracy's sake, I'll add that we have all the data about who voted on what. Our internal policy is not to look at votes by specific users on specific posts unless we have really good reason to such as suspecting foul play.

Ray is correct about what we in fact look at, but feels important to say that we in principle could see it all if we chose to, and that we're requesting some trust from the community.

[-]leggi6y30

I've found the review process a good insight into what goes on on LW. From the posts nominated and the comments and feedback they have generated I have more of an understanding of general topics but I have a lot of questions about definitions...

A couple of thoughts:

improving the LessWrong community's longterm feedback and reward cycle.

Feedback risks punishment.

"Reward cycle" is an interesting phrase - More karma sharing between the collective? Or rewarding interactions and the development of ideas/processes/beliefs/understanding through debate and learning?

What is the aim/dream for the book? On a scale of globally-acclaimed to something solid for the collective to hold.

A nomination/review cycle for 2017 (and back) would be worth doing if creating a physical book for wider release.

[-]ChristianKl6y20

It seems to me like you extended the voting period and spent more effort to get people to vote by sending out emails because you believe the voting is important.

When it comes to this post it doesn't make a case of why you believe it's valuable to vote here. I'm curious about what your idea happens to be.

When it comes to the interface I think it would be great if the interface would show me my past karma votes on the post. It's useful to have the information of how I found the post after reading it the first time at hand when trying to evaluate 75 posts at once.

[-]Raemon6y80

I realize we didn't justify the Voting very hard. Here's my offhand attempt, which maybe we'll roll into the actual post after chatting about it more on Monday.

LessWrong runs, for good or for ill, off the same forces much of the rest of the internet runs on: people who are slightly bored at work. Naturally, posts get rewarded mostly by upvotes and comments, which disproportionately reward things for being exciting and for controversial (respectively). These are quite easy to goodhart on.

The Review (in general), and Voting (in particular) are an attempt to do a more nuanced thing – to take the accumulated taste of the LessWrong community, and use it to reflect hard on what was actually good, and then backpropagate that signal through people's more general sense of "what sort of posts are good to write and why?"

Without the Vote, the signal would basically be entirely "what the Mod Team Thinks Was Best", or, if we weren't doing this at all "what posts were memorable, and/or high karma". And this isn't ideal for a few reasons:

The Mod Team doesn't have domain expertise in all the areas that posts explore
Even though we're putting a lot of work into it, it's still a really daunting project to form opinions on all 75 posts. Having a mixture of people who've looked harder at different posts helps give more coverage of nuanced opinions.
Something something wisdom of crowds – each person is biased in some way, or has different knowledge. Getting many people to participate helps counterbalance various knowledge and biases that individuals have.

I meanwhile expect the voting here to be better than usual karma-voting, because it's more comparative. You're not just voting on "this post seems good!" but "this post seems better than this other post". What I found useful for my own voting was being forced to stop and think and build a model of what-sorts-of-posts-are-good-and-why.

[-]Raemon6y40

When it comes to the interface I think it would be great if the interface would show me my past karma votes on the post. It's useful to have the information of how I found the post after reading it the first time at hand when trying to evaluate 75 posts at once.

Yeah, I definitely agree with this. I think we've put about as much work into the UI as we're going to this year (I originally budgeted a couple weeks of time for the Review and ended up spending 1.5 months on it, but I think, assuming it stays in roughly the same form next year, this is an obvious thing to include)

[-]Bucky6y20

The right hand side of the interface isn't working for me - the scroll bar looks like its scrolling down but the text doesn't move. I'm using Firefox.

[-]Raemon6y40

Oh, I think a) that may get fixed (or at least replaced by newer and more interesting bugs) in another 30 minutes when our current deploy finishes, and b) in the meantime, try scrolling on alternate parts of the page? What I just found was that the scroll depended on whether my cursor was within a particular sub-window.