Boo votes, Yay NPS

[-]jimrandomh7y140

(I'm a member of the LW team, but this is an area where we still have a lot of uncertainty, so we don't necessarily agree internally and our thinking is likely to change.)

There are three proposed changes being bundled together here: (1) The guidance given about how to vote; (2) the granularity of the votes elicited; and (3) how votes are aggregated and presented to readers.

As you correctly observe, votes are serving multiple purposes: it gives information to other readers about what's worth their time to read, it gives readers information about what other people are reading, and it gives authors feedback about whether they did a good job. Sometimes these come apart; for example, if someone helpfully clears up a confusion that only one person had, then their comment should receive positive feedback, but isn't worth reading for most people.

These things are, in practice, pretty tightly correlated, especially when judged by voters who are only spending a little bit of time on each vote. And that seems like the root issue: disentangling "how I feel about this post" from "is this post worth reading" requires more time and distance than is currently going into voting. One idea I'm considering, is retrospective voting: periodically show people a list of things they've read in the past (say, the past week), and ask people to rate them then. This would be less noisy, because it elicits comparisons rather than ups/downs in isolation, and it might also change people's votes in a good way by giving them some distance.

Switching from the current up/down/super-up/super-down to 0-100% range voting, seems like the main effect is it's creating a distinction between implicit and explicit neutral votes. That is, currently if people feel something is meh, they don't vote, but in the proposed system they would instead give it a middling score. The advantage of this is that you can aggregate scores in a way that measures quality, without being as conflated with attention; right now if a post/comment has been read more times, it gets more votes, and we don't have a good way of distinguishing this from a post/comment with fewer reads but more votes per reader.

But I'm skeptical of whether people will actually cast explicit neutral votes, in most cases; that would require them to break out of skimming, slow down, and make a lot more explicit decisions than they currently do. A more promising direction might be to collect more granular data on scroll positions and timings, so that we can estimate the number of people who read a comment and skimmed a comment without voting, and use that as an input into scoring.

The third thing is aggregation--how we convert a set of votes into a sort-order to guide readers to the best stuff--which is an aspect of the current system I'm currently least satisfied with. That includes things like karma-weighting of votes, and also the handling of polarizing posts. In the long term, I'm hoping to generate a dataset of pairwise comparisons by trusted users, which we can use as a ground truth to test algorithms against. But polarizing posts will always be difficult to score, because the votes reflect an underlying disagreement between humans and the answer to whether a post should be shown may depend on things the voters haven't evaluated, like the truth of the post's claims.

[-]Gordon Seidoh Worley7y20

But I'm skeptical of whether people will actually cast explicit neutral votes, in most cases; that would require them to break out of skimming, slow down, and make a lot more explicit decisions than they currently do. A more promising direction might be to collect more granular data on scroll positions and timings, so that we can estimate the number of people who read a comment and skimmed a comment without voting, and use that as an input into scoring.

This is very much a problem in collecting NPS data in its original context, too: you get lots of data from upset customers and happy customer and meh customers stay silent. You can do some interpolation about what missing votes mean, and coupled with scrolling behavior you could get some sense of read count that you could use to make adjustments, but that obviously makes things a bit more complicated.

[-]Said Achmiz7y90

My personal experience is mainly with writing heretical posts of good quality such that I get more up votes than down but also a lot of down votes (maybe 1⁄3 down and 2⁄3 up)

How the heck can you tell what the upvote/downvote breakdown is…? I’ve never seen anything about this in the UI.

[-]Gordon Seidoh Worley7y40

Well, I do say "maybe"; this is a guess based on how the score evolved over time and how the the total score compares to the number of votes.

[-]Said Achmiz7y40

It seems to me that variability in vote weights adds so much noise here that you can’t make anything resembling the kind of inference that you’d like to make…

[-]Raemon7y40

I think this is particularly hard with posts, after the first few votes (esp. once someone has strong upvoted it). But I think it's easier with comments if you're paying attention, esp. if you see that your karma is fairly low but has a largish number of votes.

[-]Dagon7y20

Yeah, I comment far more than I post, and I have a general idea that if I get no downvotes, it means I'm optimizing too much for approval rather than exploration of non-obvious ideas. The vote-delta feature makes it a little easier to see the downvotes (I've disabled the hiding of negatives), but mostly I have to look and figure out if the average votes-per-voter is particularly low.

[-]Gordon Seidoh Worley7y30

An interesting feature would be to show the ratio instead of only the vote count in addition to the score.

[-]Gordon Seidoh Worley7y60

Additional evidence of boo/yay voting culture: Ben Hoffman's "Downing children are rare" post (on EAF, on LW, my comment about votes)

[-]Wei Dai7y40

It could also be (partly) caused by inferential distance, i.e. the LW audience can understand Ben's ideas better because we've read more of his recent posts. ETA: or we're not as familiar with existing counterarguments, which the EAF audience might expect the post to better address.

[-]Gordon Seidoh Worley7y40

I agree there are other possible interpretations; mainly wanted to document for myself in case I wanted to reference it later, and it seems potentially relevant, especially if we wanted to go back and interview the voters or analyze the comments.

[-]Dagon7y20

I'm not sure it makes sense to put much effort into this kind of gamification. Karma, voting, reactions, etc. are all cost-free, and therefore very weak signals.

It means different things to different people, and it's so cheap that it's hard to imagine that a change in text or mechanism will radically change it's use (though tweaks may be able to moderately reframe things).

As long as voting has the general effect of increasing a sense of involvement, even for those who don't post/comment, it's probably worth keeping.

[-]habryka7y20

A related concept to all of this is the idea of adding more react-like functionality to comments. See my thoughts here: https://www.lesswrong.com/posts/EQJfdqSaMcJyR5k73/habryka-s-shortform-feed#HefgGrp3nMwf5gKkd

Do you have a sense of how much this would solve the problems you are worried about?

[-]Gordon Seidoh Worley7y20

I'm not sure how much it would help, but that's mainly because I am both not troubled by voting up things I disagree with or voting down things I disagree with and because I have a history of leaving constructive feedback comments, especially in cases where I feel like a comment/post is being treated overly harshly or where the author seems unfamiliar with community norms. I can imagine that others who are less willing to do that might be more willing to leave reacts that at least convey some of that information.

[-]Raemon7y60

I think whatever plan addresses the situation definitely needs to address that most people are much more lazy than you in this regard.

[-]mako yass7y10

it lets you express yourself in two ways (unlike on Twitter where the only option is to vote up something, and a "downvote" requires writing your own tweet expressing dislike)

Weird oversight in not observing that twitter's retweets not only represent the question "would you recommend this to a friend", but are also guaranteed to yield a truthful answer, because retweeting is an act of recommendation to friends, for which the user is then held accountable.

One of the other things I like about resharing is that the resultant salience is completely subjective. Users can share space even if they're looking for different things. There can be a curator for any sense of quality.

[-]Trinley Goldenberg7y30

would you recommend this to a friend", but are also guaranteed to yield a truthful answer, because retweeting is an act of recommendation to friends, for which the user is then held accountable.

Note that even in this relatively straightforward case, the meaning of retweets can become conflated, from "I would recommend this to a friend" to "I agree with this". I sometimes have to be careful about what I retweet because of this confusion, and things that I would otherwise recommend people read I hesitate to retweet because I don't want people thinking I agree.

[-]knite7y10

What are your thoughts on *re-labeling* the voting UI so that it's more clear to voters what the site norms are for the meaning of up- and downvotes?

[-]Gordon Seidoh Worley7y40

Maybe that's useful, but we'd have to figure out what votes are supposed to mean in the first place, i.e. I'm not sure there is a well defined notion of what votes are for now such that we could change the UI to encourage using them in the expected manner.

I have an idea of what problems I'd like solved and voting is one way to solve some of those problems and in that context we might have some sense of how we would like to ask users to use voting, but that only makes sense in that context. On it's own voting is just incrementing/decrementing counters in the database and that counter is used to inform some algorithms about the order in which content is displayed on the site; we have to decide what that means to us and what we would like it to do beyond what it naturally does on its own such that providing instruction and shaping the UI to encourage particular behaviors is meaningful.

So that's a long way to say yes, but conditional on having explicit norms.

LESSWRONG
LW

LESSWRONG
LW

29

29

29

TL;DR

Motivation

Boo Votes

Yay NPS