The programming language community blog Lambda the Ultimate had similar problems some years ago and seems to have emerged successful. Guess having a benevolent dictator editor is important. Here's the relevant discussion from 2006. My own two cents there, that I still stand by today:

The only way to improve the quality of discussions is to filter out the bozos. Don't moderate postings; moderate people.

For me, read-only access to "old LtU" would be more pleasant than read-write access to "new LtU". In other words: I don't think bans are &... (read more)

It happens to me regularly - my karma drops by roughly the number of comments I've posted since the last time it happened. It's annoying, but since my overall karma still grows, it's clearly noise.

Also, it's possible the same person is downvoting everybody in this thread; I had a drop of about 10 points today. (I skipped posting any comments this weekend, so my overall comment rate has gotten kind of low lately.)

Perhaps someone is just trying to "win" at karma?

2Vladimir_Nesov11yI just lost 20 points in half an hour.
3thomblake11yI disagree. Most people who make high-quality comments also make a large number of low-quality comments. Filtering out bad comments is essential if we want the good ones to float to the top.

On Comments, Voting, and Karma - Part I

by thomblake 4 min read7th Apr 200947 comments

9


There has been a great deal of discussion here about the proper methods of voting on comments and on how karma should be assigned.  I believe it's finally reached the point where a post is warranted that covers some of the issues involved.  (This may be just because I find myself frequently in disagreement with others about it.)

The Automatic Upvote

First, there is the question of whether one should be able to upvote one's own comment.  This actually breaks apart into two related concerns:

(1) One is able to upvote one's own comments, and

(2) One gains a point of karma just for posting a comment.

These need not be tied.  We could have (2) without (1) by awarding a point of karma for commenting, without changing the comment's score.  We could also have (1) without (2) by simply not counting self-upvotes for karma.

I am in favor of (2).  The main argument against (2) is that it rewards quantity over quality.  The main argument for (2) is that it offers an automatic incentive to post comments; that is, it rewards commenting over silence.  As we're community-building, I think the latter incentive is more important than the former.  But I'm not sure this is worth arguing further - it serves as a distraction from the benefits of (1).

I am also in favor of (1).  As a default, all comments have a base rating of 0.  Since one is allowed to vote on one's own comments, and upvoting is the default for one's own comments, this makes comments effectively start at a rating of 1.  The argument against this is that it makes more sense for comments to start with a rating of 0, so that someone else liking a comment gives it a positive rating, while someone disliking it gives it a negative rating.  I disagree with this assessment.

If I post a comment, it's because it's the best comment I could think of to add to the discussion.  I will usually not bother saying something if I don't think it's the sort of thing that I would upvote.  When I see someone else's comment that I don't think is very good, I downvote it.  Since they already upvoted it, I'm in effect disagreeing that this was something worth saying.  The score now reflects this - a score of 0 shows that one person thought it was a worthwhile comment, and one person did not.

Furthermore, if I was not able to vote on my own comments, I would be much more reluctant to upvote.  Since I would not be able to upvote my comment, upvoting someone else's comment would suggest that I think their comment is better than my own. But by hypothesis, I thought my comment was nearly the best thing that could be said on the subject; thus, upvotes will be rare.

And so I say that we implement a compromise - (1) and not (2).

What should upvote/downvote mean?

I think it is established pretty well that upvote means "High quality comment" or "I would like to see more comments like this one", while downvote means "Low quality comment" or "I would like to see fewer comments like this one".  However, this definition still retains a good bit of ambiguity.

It is too easy to think of upvote and downvote as 'agree' and 'disagree'.  Even guarding myself against this behavior I find the cursor drifting to downvote as soon as I think, "Well that's obviously wrong".  But that's clearly not what the concept is there for.  Comments voted up appear higher on the page (on certain views), which allows casual readers to see the best comments and discussions on any particular post.  If we use upvote and downvote to mean 'agree' and 'disagree', then this is effectively an echo chamber, where the only comments to float to the top are the ones that jive with the groupthink.

Instead, upvote and downvote should reflect overall quality of a comment.  There are several criteria I tend to use to judge a good comment (this list is not all-inclusive):

  1. Did the comment add any information, or did it just add to the noise? (+)
  2. Does the comment include references or links to relevant information? (+)
  3. Does the comment reflect a genuinely held point-of-view that adds to the discussion? (+)
  4. Is the comment part of a discussion that might lead somewhere interesting? (+)
  5. Is the comment obvious spam / trolling? (-)
  6. Is the comment off-topic? (-)

Since we feel the need to voice whether we agree or disagree with comments, but 'I agree' and 'I disagree' comments are noisy, it's been suggested that there should be separate buttons to indicate agreement and disagreement.  Thus, someone posting a well-argued on-topic defense of theism can get the upvote and 'disagree', while someone posting an off-topic 'physicalism is true' can get the downvote and 'agree'.  Presumably, we'd only count upvotes and downvotes for karma, but we could use 'agree' and 'disagree' for "most controversial" or other views/metrics.

Whether votes should require an explanation

It has been suggested that votes, or downvotes specifically, should require an explanation.  I disagree with both sentiments.  First, requiring explanations for downvotes but not upvotes would bias the voting positively, which would have the effect of rewarding quantity over quality and decrease the impact of downvotes.

But requiring explanations for votes is in general a bad idea.  This site is already a burden to keep up with; for those of us that do a lot of voting, writing an explanation for each one would be too much time and effort.  Requiring an explanation for every vote would doubtless result in a lot less voting.  Also, explaining votes is almost always off-topic, so adds to the noise here without really contributing to the discussion.

Note Yvain's more personal rationale:

I'm not prepared to write an essay explaining exactly was wrong with each of them, especially if the original commenter wasn't prepared to take three seconds to write a halfway decent response.

Adding to the burden of those already performing the service of voting unduly penalizes those who are doing good, to the end of appeasing those who are contributing to the noise here.

Relevant Comments

For reference, some links to relevant posts and sections of comments.  I tried to be inclusive, since there have been a lot of discussions about these issues - more relevant ones hopefully near the top. (Please comment if you know of any other relevant discussions)

1 2 3 what upvote and downvote should mean and whether there should be agree/disagree buttons

4 whether karma should be the sum of individual post scores, or (perhaps) an average

5 super-votes

6 The utility of comment karma

7 whether one should unselect the self-upvote

8 9 whether downvotes should require explanation

10 whether Eliezer Yudkowsky gets fewer upvotes than others

11 whether karma can be used to gauge rationality

12 whether people downvote for disagreeing with groupthink

13 whether karma promotes a closed-garden effect

14 whether administrators should delete comments entirely

15 Lesswrong Antikibitzer: tool for hiding comment authors and vote counts

ETA: I might concede that this post is possibly off-topic for Less Wrong - but the blog/community site about "Less Wrong" does not exist yet, so this seems like the best place to post it.

ETA2: Public records of upvotes/downvotes might solve some of these problems; discuss.

9