Not sure if you’re aware of this but there is an anti-kibitzer mode which hides author names and karma. I haven’t used it but have used the greaterwrong.com version (eye icon) which doesnt require any installation.
I use/don’t use it for more or less the reasons you describe - I use it when I think I’m likely to be reading lots of posts for a week or so and filtering for time constraints is less important.
Not sure whether it would be possible to compare votes with and without the feature on and whether there would be any concerns about accidentally deanonymising the votes if only a few people are using it.
If authors’ posts get a similar number of upvotes compared to when their names were public, then that’s a sign that every post is evaluated independently.
Controlling for the amount of views seems important.
A lot of posts (and comments) are context-dependent, either as part of a series, or as part of a topic exploration by a set of authors who are discussing both here and in other media (including offline conversations). In these cases, knowing the author's previous work is necessary to the evaluation of a given post. Basically, posts aren't fully independent.
Also, within this site, "high status" correlates pretty well with "high quality posts", so that bias doesn't actually lead to incorrect results very often.
Shower thought: How much are new posts evaluated based on authors’ old reputation?
We could find out if all posts were anonymous for a duration of x weeks and posts could only be upvoted during that time. [If it’s considered important that posts can be upvoted later as well, then there could be a second karma tally for the non-anonymous period after the anonymous one.]
If authors’ posts get a similar number of upvotes compared to when their names were public, then that’s a sign that every post is evaluated independently.
Whether it’s good or bad to evaluate posts based on the author is unclear imo:
- good, because it’s a quick style filter, e.g. I’m in the mood to read a post, what shall I read? Well I will read this author’s post, because I liked all their previous posts, and I reasonably believe that I will like their new post.
- bad, because it biases evaluations in favour of posts by high status people, and does not evaluate posts on merit alone.