LessWrong Has Agree/Disagree Voting On All New Comment Threads

Ben Pace

Starting today we're activating two-factor voting on all new comment threads.

Now there are two axes on which you can vote on comments: the standard karma axis remains on the left, and the new axis on the right lets you show much you agree or disagree with the content of a comment.

How the system works

For the pre-existing voting system, the most common interpretation of up/down-voting is "Do I want to see more or less of this content on the site?" As an item gets more/less votes, the item changes in visibility, and the karma-weighting of the author is eventually changed as well.

Agree/disagree is just added on to this system. Here's how it all hooks up.

Agree/disagree voting does not translate into a user's or post's karma — its sole function is to communicate agreement/disagreement. It has no other direct effects on the site or content visibility (i.e. no effect on sorting algorithms).
For both regular voting and the new agree/disagree voting, you have the ability to normal-strength vote and strong-vote. Click once for normal-strength vote. For strong-vote, click-and-hold on desktop or double-tap on mobile. The weight of your strong-vote is approximately proportional to your karma on a log-scale (exact numbers here).

Ben's personal reasons for being excited about this split

Here's a couple of reasons that are alive for me.

I personally feel much more comfortable upvoting good comments that I disagree with or whose truth value I am highly uncertain about, because I don’t feel that my vote will be mistaken as setting the social reality of what is true.
I also feel very comfortable strong-agreeing with things while not up/downvoting on them, so as to indicate which side of an argument seems true to me without my voting being read as “this person gets to keep accruing more and more social status for just repeating a common position at length”.
Similarly to the first bullet, I think that many writers have interesting and valuable ideas but whose truth-value I am quite unsure about or even disagree with. This split allows voters to repeatedly signal that a given writer's comments are of high value, without building a false-consensus that LessWrong has high confidence that the ideas are true. (For example, many people have incompatible but valuable ideas about how AGI development will go, and I want authors to get lots of karma and visibility for excellent contributions without this ambiguity.)
There are many comments I think are bad but am averse to downvoting, because I feel that it is ambiguous whether the person is being downvoted because everyone thinks their take is unfashionable or whether it's because the person is wasting the commons with their behavior (e.g. belittling, starting bravery debates, not doing basic reading comprehension, etc). With this split I feel more comfortable downvoting bad comments without worrying that everyone else who states the position will worry if they'll also be downvoted.
I have seen some comments that previously would have been "downvoted to hell" are now on positive karma, and are instead "disagreed to hell". I won't point them out to avoid focusing on individuals, but this seems like an obvious improvement in communication ability.

I could go on but I'll stop here.

Please give us feedback

This is one of the main voting experiments we've tried on the site (here's the other one). We may try more changes and improvement in the future. Please let us know about your experience with this new voting axis, especially in the next 1-2 weeks.

If you find it concerning/invigorating/confusing/clarifying/other, we'd like to know about it. Comment on this post with feedback and I'll give you an upvote (and maybe others will give you an agree-vote!) or let us know in the intercom button in the bottom right of the screen.

We've rolled it out on many (15+) threads now (example), and my impression is that it's worked as hoped and allowed for better communication about the truth.

One virtue of the new voting axis is how it allows users to clearly express great
appreciation for high-quality comments that many users disagree with.

Starting today we're activating two-factor voting on all new comment threads.

How the system works

Agree/disagree is just added on to this system. Here's how it all hooks up.

Agree/disagree voting does not translate into a user's or post's karma — its sole function is to communicate agreement/disagreement. It has no other direct effects on the site or content visibility (i.e. no effect on sorting algorithms).
For both regular voting and the new agree/disagree voting, you have the ability to normal-strength vote and strong-vote. Click once for normal-strength vote. For strong-vote, click-and-hold on desktop or double-tap on mobile. The weight of your strong-vote is approximately proportional to your karma on a log-scale (exact numbers here).

Ben's personal reasons for being excited about this split

Here's a couple of reasons that are alive for me.

I personally feel much more comfortable upvoting good comments that I disagree with or whose truth value I am highly uncertain about, because I don’t feel that my vote will be mistaken as setting the social reality of what is true.
I also feel very comfortable strong-agreeing with things while not up/downvoting on them, so as to indicate which side of an argument seems true to me without my voting being read as “this person gets to keep accruing more and more social status for just repeating a common position at length”.
Similarly to the first bullet, I think that many writers have interesting and valuable ideas but whose truth-value I am quite unsure about or even disagree with. This split allows voters to repeatedly signal that a given writer's comments are of high value, without building a false-consensus that LessWrong has high confidence that the ideas are true. (For example, many people have incompatible but valuable ideas about how AGI development will go, and I want authors to get lots of karma and visibility for excellent contributions without this ambiguity.)
There are many comments I think are bad but am averse to downvoting, because I feel that it is ambiguous whether the person is being downvoted because everyone thinks their take is unfashionable or whether it's because the person is wasting the commons with their behavior (e.g. belittling, starting bravery debates, not doing basic reading comprehension, etc). With this split I feel more comfortable downvoting bad comments without worrying that everyone else who states the position will worry if they'll also be downvoted.
I have seen some comments that previously would have been "downvoted to hell" are now on positive karma, and are instead "disagreed to hell". I won't point them out to avoid focusing on individuals, but this seems like an obvious improvement in communication ability.

I could go on but I'll stop here.

Please give us feedback

We've rolled it out on many (15+) threads now (example), and my impression is that it's worked as hoped and allowed for better communication about the truth.

Hmm didn't really find anything similar, but here are some examples of rating systems I found that looked interesting (though not necessarily relevant):

2-factor rating systems

SaidIt: (1) Insightful & (2) Fun

SaidIt is a Reddit alternative which seeks to "create an environment that encourages thought-provoking discussion". SaidIt has two types of upvotes to choose from: 1) insightful, and 2) fun.^[1]

Goodfilms: (1) quality & (2) rewatchability

Goodfilms is a movie site for users to rate, review, share films and find movies to watch. Users rate movies on two dimensions: quality and rewatchability. The ratings are displayed as a scatterplot, giving users a better sense of the type of movie (e.g. most people agree it is highly rewatchable, but there is disagreement on its quality => may not be very good, but is fun to watch).^[2]

Scatterplot for Starship Troppers ratings

Suggestion by Majestic121: (1) Agree/Disagree & (2) Productive/Unproductive

A Hacker News comment by Majestic121 suggests a 2-factor voting system:

Up/Down : Agree/Disagree Left/Right : Makes the discussion go backward/forward
This way you could express disagreement while acknowledging that the point is interesting, or like a joke without having it drown a conversation

Suggestion by captainmuon: (1) Promote/Bury & (2) Reward

Hacker News comment by captainmuon: Promote/Bury and Reward buttons

Up/downvotes always have multiple conflicting dimensions.
The post is factually right / wrong
Confirms to the site rules / breaks the rules
I agree / disagree
I want to promote / bury this post
Reward poster with XP / punish
I am usually very pragmatic and upvote a post when I want other people to read it (because I want to see the discussion, or because I want to spread the idea). I also upvote to reward the poster.
I don't tend to downvote factually wrong posts when they are still interesting, because that limits the chance they get good discussion, and because I don't want to punish somebody for Being Wrong On The Internet. I do downvote positions that I find bad in order to reduce their reach.
It would be probably possible to have a site that implements two dimensions as a cross (maybe only for users with a certain XP) although the UX might not be to great. Maybe it is a good idea to have "promote/bury" and "reward" buttons?

Others

Pol.is: displays a cluster graph of participants based on their voting patterns on statements

Pol.is is a platform where participants submit statements for others to vote on (agree/disagree/pass), and participants are then clustered based on their votes. People can see this map of voters and are incentivized to craft statements that also appeal to members of other groups to gain more support, thus converging on a consensus.^[3]

Description of graph extracted from Pol.is report:

In this graph, statements are positioned more closely to statements which were voted on similarly. Participants, in turn, are positioned more closely to statements on which they agreed, and further from statements on which they disagreed. This means participants who voted similarly are closer together.

Example of cluster graph from Taiwan's Uber case study

Example case study: vTaiwan Uber

People in Taiwan were invited to discuss the regulation of Uber. At the beginning, there were two groups: pro-Uber and anti-Uber. As people tried to submit statements that would gain more supporters, they converged on a set of seven comments that majority agreed on, such as "It should be permissible for a for-hire driver to join multiple fleets and platforms." These suggestions shaped the regulations that were eventually adopted by the government. ^[3]

Other Pol.is case studies and reports: https://compdemocracy.org/Case-studies

Tweakers: users can assign scores from -1 to +3, with detailed guidelines on how to vote

Tweakers is a Dutch technology website. Users can assign a score of +3 (insightful), +2 (informative), +1 (on-topic), 0 (off-topic/irrelevant), -1 (unwanted) to comments. The median score is displayed, but users can click to view the breakdown of votes.^[4] 0 scores with 0 votes are displayed in a different color from 0 votes with >= 1 votes.

There are detailed guidelines on what each score means, as well as how to handle common scenarios, such as:

Comments about spelling, typos etc.: -1, because they should be reported in the Dear Editorial Forum instead
Inaccurate comments: no less than 0 if it is well-intentioned, -1 is for socially inappropriate behavior
Opinions you disagree with: comments should not receive lower ratings simply because they conflict with your opinion. To express dissatisfaction, reply the comment with your refutation.
Strategic voting e.g. upvoting a comment because you think its score is too low, rather than because you think it deserves the high score: do not do this

The guidelines also vary based on the context, with the guidelines explaining how moderation practices should differ for downloads, product reviews, and other pages.

(Disclaimer: site was in Dutch so I used Google Translate)

Slashdot: voting by assigning different types of labels (e.g. insightful, redundant)

Moderators can assign different labels to comments, which will add or deduct a point from the comment's score. There are descriptions of what each label means in the FAQ. The labels are as follows:

Normal (default setting)
Offtopic
Flamebait
Troll
Redundant
Insightful
Interesting
Informative
Funny
Overrated
Underrated

Placebo button :p

Hacker News comment by kevin_thibedeau:

They should have dummy agree/disagree buttons that disappear once selected and has no function other than to satisfy the malcontents that the machine has recorded their opinion.

157

LessWrong Has Agree/Disagree Voting On All New Comment Threads

157

How the system works

Ben's personal reasons for being excited about this split

Please give us feedback

157

157

LessWrong Has Agree/Disagree Voting On All New Comment Threads

157

How the system works

Ben's personal reasons for being excited about this split

Please give us feedback

157

2-factor rating systems

Others