Here's what I consider the relative merits of each Tag Relevance system. (Note: it'd be helpful if you read the OP and form your own opinion before reading this)
...
...
Karma-Style
Advantages:
Disadvantages:
I expect many of these problems to mostly effect less-visited tags. More visited tags will probably end up with lots of users contributing, resulting in a "wisdom of crowds". But I expect at least half of Tag Pages to be sparsely attended, and for the relevance votes to be much noisier than just sorting by karma.
Multiple Choice Voting Style
Advantages:
Disadvantages:
I found the current karma-style relevance to be surprisingly intuitive. My impression of tags is that the main problem is figuring out what is allowed to be a tag or not - I'm not sure how this was done, but the tags seem to be good.
I think "top" posts can be some combination of high karma, high tag relevance, and curated.
There is the additional problem of "Meditations on Moloch" was read by a lot of people, so it will get high karma and high tag relevance, and maybe be curated because it is high quality. This suggests that the multiple choice system is better able to bucket posts.
I think I want any tagging system to be able to separate between posts that use the concept and posts that try and explain the concept. There will not be that many of the later, but they should be prioritized on the tag's page (there are possibly only 1 or 2, in which case a form of curation but for specific tags might solve the issue).
I like the current karma style relevance system and think it's worthy to be given a good trial period. If after a year we find that not enough people vote on tags, we can introduce other features, but I think the tag functionality as it is functional enough to warrant a good trial period.
Meta on format: Why didn't you use the question feature for this post?
Meta on content: yay, I really like those kind of improvement!
Object: epistemic status: thinking outloud right after having read the post, before reading the comments (as requested), and not having thought about this much before; not confident
it seems like system 1 asks "which categories belong to the post the most?" whereas system 2 asks "which posts belong to the category the most?", sort of. I think most of the time, categorization is useful for system 2's purpose (ie. searching for articles in a specific category), and when you want to know the categories of a specific post before reading it, system 2 should still be able to play system 1's role.
also, system 2 shows the average instead of the sum, which might better represent the categorization. on the other hand, knowing the number of votes that the categorizations received is useful to know how large the sample was.
both systems seems to allow for 5 options, so are equally fine-grained (although maybe system 2 for more as it seems you can select two options from the video?)
voting system is more intuitive (flows better?) because we already use it
I wonder if the post writer should have more votes or something
Voting seems to fit if you're okay with things not fitting together. If you want things to fit together, a set of "Economic" sequences/posts with an intentional order might work better.
We recently launched our Tagging Beta (See here for the full anouncement). We've spent the past few months thinking a lot about our longterm vision for tags on the site, but still have some uncertainty about exactly what shape tagging should take.
One major question that the team hadn't yet come to agreement on is how exactly to vote on Tag Relevance.
When you go to a tag page (such as the Rationality Tag), somehow we have to decide which posts to list first. An obvious answer is "just sort by karma", but this isn't quite right: High karma posts aren't necessarily the most relevant posts.
For example, if "Meditations on Moloch" had originally been a LessWrong post, one might have tagged it "economics". (It does touch on economic concepts). And it might have gotten lots of upvotes because it was an evocative, powerfully written piece. But this wouldn't make it appropriate to be listed first on the Economics Tag Page.
So, we want some other system to determine how relevant a post is. And we want that system to be scalable, where lots of users could potentially go tagging posts and determining their relevance. But this leaves us needing a dispute mechanism – what happens if two people disagree?
We shipped the Tagging Beta with one such system, but are considering an alternative. I'd love your feedback.
We shipped with a system that simply re-used our existing Karma System. For each Tag-Post relationship, you can vote on how relevant the post is to the tag, using your normal array of upvotes/downvotes/weakvotes/strongvotes.
Here, relevance is a simple number. Higher relevance posts are sorted higher on the tag page. Posts are considered "tagged" if they have a positive amount of relevance. If you think a post was tagged incorrectly, you can downvote it to zero.
This system has some advantages and disadvantages. I'm going to describe my thoughts on those in the comments, so that you have a chance to form your own opinion if you want.
There were some aspects of the Karma-Relevance system that I personally found dissatisfying. So I created an alternate system and pitched it to the team, and we collectively decided to present them both and give you guys the chance to give some feedback.
This system uses multiple-choice voting, where you sort the tag into one of five buckets. Within those buckets, posts are sorted by karma.
Here's a video showing how that would work. (I recommended watching on fullscreen mode to more easily read the text)
Instead of adding up votes to get a total relevance, this uses the median-vote to determine which bucket to put the post in. Then, on the tag page, all Top Relevance posts will be listed first (in karma order), followed by all High Relevance posts, then Medium, then Low.
Votes would still get vote-weight based on your karma, so established users can more easily counter an overzealous newbie who went on a tagging spree.
In the comments, I'll go into more details about how each system works, and I think their relative merits are. But an important question is "how intuitive is each system?".
So for now, I'm just going to end the post asking for you to try out the current karma-relevance voting system (you can try upvoting this post for the Site Meta tag to get an idea of how that works), and to watch the video so you have an idea of what the new system would look like.
If you're up for it, it'd be quite helpful if you could write up your "first impressions" before reading the more detailed comments.