Time and attention is scarce. You gotta have quality metrics to decide whether you want to read something, before you read it. I read maybe 10% of things I scroll past. If you put the quality metrics at the bottom, you can't use them to decide whether to read a piece of content at all.
You can hide the information and still sort by the information, but this doesn't work for post-items where we have to intersperse low-karma and high-karma content to get signal out of people, and for comment threads, it of course still makes a huge difference whether a comment is at +2 or +35 on whether I want to read it.
People read a minority of content they encounter. A UI that makes it impossible to make an informed decision whether to start reading something basically breaks the central cognitive operation that a user performs on the site.
Edit: To be clear, you acknowledge this when you say "These ideas aren't perfect. For example, karma is genuinely useful for selecting which comments you'd like to read. By making the karma less prominent, it's harder to skim for comments above a karma threshold". My comment here is meant to just restate that to me, all things considered, the tradeoffs don't seem worth it, and that it seems to me that "hiding/deemphasizing information about what other users think about the quality of this content" and "making it as easy as possible to decide whether a comment is worth reading without reading it" are more fundamentally in conflict, so it's not clear there are clever solutions here.
3This proposal is compatible with the algorithm you just stated. You would skim comments looking by the bottom and then go to the top of the comment if they're highly rated. You'd be moving your eyes to a different part of the page for a moment—hardly the "impossible to make an informed decision" you rally against!
I also engaged with this critique in the post. Did you read this part?
These ideas aren't perfect. For example, karma is genuinely useful for selecting which comments you'd like to read. By making the karma less prominent, it's harder to skim for comments above a karma threshold. Consider two cases:
- The comment is not collapsed. In this case, while skimming the webpage, you can scroll down and just learn to look at the bottom of comments instead of the top. If the comment passes a threshold, read it by scrolling up slightly. This is mildly inconvenient.
- The comment is collapsed. Then the karma count isn't visible at the bottom (since otherwise it'd be visible early on). This is a problem.
The fix might be to modify proposal (2) to keep "karma" at the top of the comment but keep "username" and "agreement" at the bottom. I'm open to other ideas which do an even better job of minimizing costs and maximizing gains!
You would skim comments looking by the bottom and then go to the top of the comment if they're highly rated.
I unfortunately think this is too difficult and time-consuming for long comments or posts. This can work if you truncate the content so that scrolling to the bottom is always pretty easy, and if you don't have nesting so you can make the bottom of a comment easy to visually identify, which we both have on the frontpage feed, and so indeed on the frontpage feed we have the relevant information on the bottom:
I also engaged with this critique in the post. Did you read this part?
Yep, that's why I left my other two comments! Sorry for that probably producing a slightly disorienting comment experience, I considered leaving all three comments of mine in a combined form, but then decided it would be better to break it out to make voting and responding to them easier.
I think your engagement in the post isn't really responding to the heart of the difficulty though, and my top-level comment is more supposed to be read as "you say there are tradeoffs, but no, this is like, the core thing users do on the site, you can't really trade off much against that".
1I unfortunately think this is too difficult and time-consuming for long comments or posts. This can work if you truncate the content so that scrolling to the bottom is always pretty easy, and if you don't have nesting so you can make the bottom of a comment easy to visually identify, which we both have on the frontpage feed, and so indeed on the frontpage feed we have the relevant information on the bottom:
That makes sense. And even if you did truncate the content for this reason, people might just learn to reflexively look down to the bottom of the comment instead of the top. I expect I'd have to put in effort to resist.
Perhaps karma really should stay up top. The site has already done the admirable work of (imperfectly) disentangling "quality" from "agreement." So why not use that work and lend readers trust in their ability to decouple "knows comment has high karma" with "is anchored positively on agreeing with the comment." So I'm warming to "non-voting karma up top, full karma + agreement panel at the bottom."
There'd still be the "sees high karma" -> "will think it's high quality" coupling, but perhaps the "sees high karma" -> "will agree" coupling is weaker (and that's the more important one IMO).
1Yes. If I had to read everything carefully, I would mostly agree with the post's proposal, but it is crucial that I don't have to read everything.
Agree. Hiding metrics by default will be too annoying. I would be happy to opt-in to something like "At the top of the comment section, randomly show me n=3 comments with hidden stats" though. Hiding karma/agreement for the first couple hours is also good.
At the top of the comment section, randomly show me n=3 comments with hidden stats
This is getting into details, but even very popular posts only tend to have on the order of 5-10 top-level comments, so giving you 3 random ones is basically getting rid of sorting.
It can be customizable, or reduced to n=1, or my intention is that you actually show the same comment twice in total.
Likely the biggest win. Hide karma and agreement indicators in the hour after a comment is posted. This would reduce the initial "luck" of someone strong-upvoting a comment, leading to a cascade of other positive votes due to anchoring.
I do think something like this is kind of reasonable, but I actually don't want this for the opposite reason. I frequently rescue random comments and posts that clearly accidentally triggered someone in a way that doesn't seem like it should result in being downvoted. Those are not usually comments or posts I would strong-upvote, so if I can't see their karma total I can't rescue them by strong-voting, I think the situation on the site would actually be worse in terms of negative visibility cascades.
1Hmmmm, interesting.
- If the comment is actually good, it should presumably come out of the visibility haze with a good score on average. Unless you're worried about the low engagement regime? But then rescuing those comments seems less important anyways.
- This seems like a niche use case (which doesn't mean it's not legitimate). I, at least, very rarely engaged in this while using the site. But it does suggest that an account-level toggle would let you (specifically) engage in this activity. WDYT?
- If there's a lot of brigading, that seems bad. But also people might just legitimately be using their votes in ways you disagree with? Sometimes "I'm confident they're wrong" leads to the perception "so the only way you could downvote this is if you're a triggered idiot, I must reverse it." Hard to say without more insight into the incidents you have in mind, though. You could have a lot of data I'm missing.
If the comment is actually good, it should presumably come out of the visibility haze with a good score on average. Unless you're worried about the low engagement regime? But then rescuing those comments seems less important anyways.
I do actually care about low-quality contributions being sorted to the bottom and collapsed pretty immediately, so this is a tricky situation.
Also, I was thinking more of posts and quick takes, where if you get downvoted early it can really hurt your visibility. For comments on posts I am less worried, and IDK, maybe I am just sold that we should do this for comments on posts (but not for quick takes and top-level posts, for that reason).
If there's a lot of brigading, that seems bad. But also people might just legitimately be using their votes in ways you disagree with? Sometimes "I'm confident they're wrong" leads to the perception "so the only way you could downvote this is if you're a triggered idiot, I must reverse it." Hard to say without more insight into the incidents you have in mind, though. You could have a lot of data I'm missing.
I often remove my strong-upvote when the comment then later on gets upvoted by other people. Agree that I don't want to cancel out other people's votes, but I do think it's worth reducing random variance around the null-point, especially since getting into negative karma early basically guarantees no one will see it (since at -5 it's hidden by default even from the All Posts page).
I don't want to cancel out other people's votes, but I do think it's worth reducing random variance around the null-point, especially since getting into negative karma early basically guarantees no one will see it (since at -5 it's hidden by default even from the All Posts page).
I wonder whether we can do something programmatically to reduce the high-variance there. For instance, a comment could only be collapsed if there are at least 2 downvotes on it (of any strength).
The fix might be to modify proposal (2) to keep "karma" at the top of the comment but keep "username" and "agreement" at the bottom. I'm open to other ideas which do an even better job of minimizing costs and maximizing gains!
I think username is maybe the most important quality signal and also relational signal (I read a comment very differently if I know it's by the post author on their own post, or is by someone who has left another comment in the same thread), so I don't think it makes sense to move that to the bottom.
I think moving the karma + agree-voting to the bottom is often the right choice. For example, it's what we do in the feed!
On the feed we make sure we collapse comments to a height where you can always see the bottom of the comment if you are looking for it. Doing this in other contexts it's pretty tricky, but maybe worth figuring out.
There are two big reasons why I don't want to just move the agreement-karma to the bottom:
- It just adds visual clutter by needing to have substantially taller spacing for comments at the bottom. This alone seems overcomeable.
- Making it clear to users that on LessWrong, we factor out agreement from approval is a really important cultural touchstone, and if you separate them visually then that factorization is much less clear
1Making it clear to users that on LessWrong, we factor out agreement from approval is a really important cultural touchstone, and if you separate them visually then that factorization is much less clear
Maybe you could repeat the karma at the bottom of the comment, next to the recently moved agreement?
When I tried this it felt pretty disorienting, especially to new users, but I am not confident it couldn't be made to work.
The way I would probably structure it is to have just the karma at the top, without any vote buttons, and then to have the full interactive version of the karma and agreement at the bottom. I probably have some mockups where I try this somewhere, but it never really quite came together.
1That's awesome, nice! I haven't used the new feed. I went to check the average comment case and the post case, but hadn't considered that feature.
When reading a comment, the first thing you see is what other people think. That design choice reduces your ability to form your own opinion and makes the site's karma rankings less related to the comment's true value. I think the problem is fixable and propose some ideas for consideration.
The LessWrong interface prioritizes social information
You read a comment. What information is presented, and in what order?
The order of information:
This is unwise design for a website that emphasizes truth-seeking. You don't have a chance to read the comment and form your own opinion first. However, you can opt in to hiding usernames (until moused over) via your account settings page.
A 2013 RCT supports the upvote-anchoring concern
From Social Influence Bias: A Randomized Experiment (Muchnik et al., 2013):[1]
Inline reaction indicators also seem anchoring
Inline reactions are shown as little icons to the right of the line of text. Here's an image of sidelined reactions to a comment of mine:
I find these "reactions" distracting. They discourage people from forming independent opinions and probably have produced too much agreement with my comment.
When I'm reading LessWrong content and see an icon on the side, the icon grabs my attention and distracts me from the content.
In order to avoid people's first impressions being anchored by these reactions, I sometimes redirect users from LessWrong to my website.
Concrete proposals
A mock-up of how (2) might be implemented. This assumes that Matthew's comment was not collapsed (and just ends as shown). A more modest (but still good) change would be to just move the agreement score to the bottom.
These ideas aren't perfect. For example, karma is genuinely useful for selecting which comments you'd like to read. By making the karma less prominent, it's harder to skim for comments above a karma threshold. Consider two cases:
The fix might be to modify proposal (2) to keep "karma" at the top of the comment but keep "username" and "agreement" at the bottom. I'm open to other ideas which do an even better job of minimizing costs and maximizing gains!
(And to anyone about to type "this can't be fixed", have you spent five minutes (by the clock) thinking about the issue first?)
Prior discussion and results
In 2021, Max Harms talked about Improving on the Karma System. His proposal focused on augmenting the entire system, not just the way karma is displayed.
The LessWrong team has made changes in related areas. Total karma is deliberately not displayed prominently. Side-comments default to "just an icon" until you mouse over them. Karma used to be much more prominent at the top of posts, but now (on desktop) it's a smaller number in the top right. These seem like good choices.
In 2013, gwern shared the results of a highly relevant experiment. gwern followed the posts made by eight participating authors. gwern used an alternate account to randomly upvote or downvote the article and post a comment with a boilerplate rationalized "explanation" for the vote. For example: "downvoted, not enough math." A month later, gwern came back to measure the total karma of the post. After controlling for an outlier popular post by Scott Alexander, the data indicated that an early up/downvote produced a non-statistically significant effect (with a difference-in-mean karma of about 10). However, as gwern notes, the sample size was small, so it wasn't highly powered to begin with.
While gwern's experiment measured a proxy of the bandwagon-y-ness of LessWrong at that point in time, it measured how an initial comment affected the final karma of the post. Not how the visual prominence of karma- and agree-counts on a comment affected the final karma- and agree-counts of that comment. Related, but not quite the same. This is evidence against super strong versions of the effect (e.g. "most voters bandwagon off of the existing score"), but compatible with meaningful anchoring due to current design choices.
Also, being able to see agreement right away can be a stronger effect than a single comment saying "upvoted" or "downvoted" (and a +1 or -1 to post karma), since the initially displayed agreement might be quite strong (e.g. +25). Seeing a tally of multiple votes all at once likely has a stronger impact on decision-making.
Please show social signals after the comment!
To me, the most valuable part of LessWrong was how it encouraged interesting contrarian comments. Many of us value truth-seeking, so I hope users and moderators optimize the website to better reflect that value.
Inspired to finally share this critique due to Ryan Greenblatt's comment arguing that more people should post on LW rather than X.
Appendix: Filter list
Here's what I use in my Brave browser to filter out AF / LW karma and agree-votes.
Unlike LessWrong's design, this study didn't increase the visibility of highly rated posts. That would likely have strengthened the effects, as an initial upvote increases view count, which can lead to a compounding "rich get richer" outcome.