I should post separately about this at some point.

Suppose we have a Collective Judgment of Science system in which scientific karma enters the system at highly agreed-upon points, e.g. very well-replicated, significant findings. Is there a system with the following properties:

• The karma entry points need not necessarily be the most trusted people. Let's say you made a significant discovery, but 70% of the field disagrees with most of your opinions, and someone who hasn't made a significant discovery is trusted by 95% of the people who make significant discoveries. We should perhaps believe the latter person over you; making one discovery is not proof of perfect epistemic reliability.

• If someone goes rogue and endorses a thousand trolls, who in turn endorse a million trolls, the million trolls can do no more karmic damage / produce no more karmic distortion, than the original person.

• If I make three significant discoveries or write three good papers, there is no incentive to spread those papers out over 3 pseudonyms, or coauthor them with 3 others, in terms of how much influence I will have afterward. There may potentially be some incentive to centralize, although this would also not be good.

• Downvoting or strongly downvoting an idea that many reliable epistemic voters think is correct may potentially be taken as Bayesian evidence by the system that you sometimes downvote good ideas. It's probably worth distinguishing this from concluding that you sometimes upvote bad ideas, without separate evidence.

• Rather than give people an incentive to waste labor by systematically downvoting everything that person X said, there is a centralized "I think this person is a complete idiot" button. After pressing this button, further systematic downvoting has no effect. Obviously the order of operations should not be significant here, i.e., this button must have as much effect as downvoting everything. Perhaps you might be asked to look at the person's 3 highest-karma nodes and asked if you really want to downvote those too (vs. an "I hate most but not all things you say" rating) given that indicating "I uniformly hate everything you say" may then potentially reflect poorly on your reliability.

• Within these constraints, it should be generally true that one person who's gotten a large karma prize cannot outvote 100 people who were all endorsed by trusted epistemics with karma originating from sources outweighing that single prize.

• We're okay with this system using terabytes or even petabytes of memory to scale, so long as it's not exabytes and it can compute updates in real time, or at least less than an hour.

• Being able to run on upvotes and downvotes is great, failing that having people click on a 5-star level or a linear spectrum is about as much info as we should ask, since most users will not provide more info than this on most occasions. We could potentially have a standard 5-star scale which by leaving the mouse present for 5 seconds can go to 6 stars, or a 7-star rating which can be given once per month, or something. We can't ask users to rate along 3 separate dimensions.

• We should take into account that some people have pickier standards and downvote more easily or upvote more rarely than others, or conversely someone who endorses almost everything is only providing discriminatory Bayesian evidence about a threshold on the low end of the quality scale.

• We can suppose that nodes are clustered in a 3-level hierarchy by broadest area, subject, and subspecialization but probably shouldn't suppose any more clustering in the data than this. It's possible we shouldn't try to assess it at all.

• A consequence of this system is that as a philosopher, you can potentially achieve great endorsement of your perspicacity, but only by convincing people who were upvoted by people who delivered well-replicated significant experimental results. This strikes me as a feature, not a bug. I don't know of any particularly better way to decide which philosophers are reliable.

• It can potentially be possible to bet karma on predictions subject to definite settlement a la a prediction market, since this can only operate to increase reliability of the system. If an open question that people opinionated about is definitely settled, anyone who was bold in predicting a minority correct answer should have their karma in some way benefit. Again we do not want an incentive to create pseudonyms to get independent karma awards here. (We can perhaps imagine such a question-node as a single source which endorses everyone who endorsed its correct answer.)

• Presentation ordering of new nodes takes into account a value-of-information calculation, not just the highest confidence in current karma. (Obviously, under such a calculation, more prolifically voting users will see more recent nodes. This is also fine.)