The way we handled this with Verity was to pick a series of values, like "good judgement", "integrity," "consistency" etc. Then the community would select exemplars who they thought represented those values the best.
As people voted on which proposals they liked best, we would weight their votes by:
1. How much other people (weighted by their own score on that value) thought they had that value.
2. How similarly they voted to the examplars.
This sort of "value judgement" allows for fuzzy representation of high level judgement, and is a great supplement to more objective metrics like Brier score which can only measure well defined questions.
Eigentrust++ is a great algorithm that has the properties needed for this judgement-based reputation. The Verity Whitepaper goes more into depth as to how this would be used in practice.