Thanks for this. In principle, you could use KBCs for any kind of evaluation, including evaluation of products, texts (essay grading, application letters, life plans, etc), pictures (which of my pictures is the best?), etc. The judicial system is very high-stakes and probably highly resistant to reform, whereas some of the contexts I list are much lower stakes. It might be better to try out KBCs in such a low-stakes context (I'm not sure which one would be best). I don't know what extent KBCs have tested for these kinds of purposes (it was some time since I looked into these issues, and I've forgotten a bit). That would be good to look into.
One possible issue that one would have to overcome is explicit collusion among subsets of raters. Another is, as you say, that people might converge on some salient characteristics that are easily observable but don't track what you're interested in (this could at least in some cases be seen as a form of "tacit collusion").
My impression is that collusion is a serious problems for ratings or recommender systems (which KBCs can be seen as a type of) in general. As a rule of thumb, people might be more inclined to engage in collusion when the stakes are higher.
To prevent that, one option would be to have a small number of known trustworthy experts, who also make evaluations which function as a sort of spot checks. Disagreement with those experts could be heavily penalised, especially if there are signs that the disagreement is due to (either tacit or explicit) collusion. But in the end only any anti-collusion measure needs to be tested empirically.
Relatedly, once people have a history of ratings, you may want to give disproportionate weights to those with a strong track record. Such epistocratic systems can be more efficient than democratic systems. See Thirteen Theorems in Search of the Truth.
KBCs can also be seen as a kind of prediction contests, where you're trying to predict other people's judgements. Hence there might be synergies with other forms of work on predictions.
Note that 'guilt' and 'innocence' is normally settled by a jury (for serious cases), and that most (interesting) judicial decisions are on cases that don't have a binary outcome, and the reasoning by which they make the decision is an important part of the precedent set by their decision. It seems like this method can still work for that, but this exacerbates concerns that things will be 'decided by the lowest common denominator' instead of whatever the 'legal truth' should be.
I liked how your idea resembles the aggregation used in prediction markets. But, I'm not sure how it would reflect in the number of cases judged by judge after considering that they receive the same ones to review. If the ratio of KBCJ to normal ones is high, there will be a waste from too many people review the same cases, and you are trading efficiency for more predictability, but it might cost too much in money terms. If it is low, and the pay rate is higher than the normal ones, it can create two different judging modes, the normal where they will still have more freedom to choose, and the KBCJ where they pay more attention to an aggregator to aim for conformity. It feels like the two modes students have, one more relaxed to learn, and another one more focused to get a better grade on a test, even if it involves reducing the amount learn to get a better grade.
Would this system be expanded to cover both sides lawyers by punishing those who make claims that are expected to be losers considering the KBCJ judgement?
You could keep the current legal system and use the "Keynesian beauty contest" as a way to evaluate judges for retention. Are their actual decisions likely to win the "Keynesian beauty contest"? If not, that could inform the decision whether to retain. Once the community has seen the "KBC" functioning as a useful tool, it may be easier to consider implementing something like your KBCJ system.
The job of the judicial system is to interpret the law. It's valuable for society that the law is interpreted consistently--for example, so we can make legal agreements and trust that judges will enforce those agreements in a way that's similar to how they enforced such agreements in the past. This is why legal precedents exist.
I think one possible problem facing democracy, at least here in the US, is that incentives for judges to follow precedents aren't strong enough. Because judges have so much flexibility in how they interpret the law, the appointment of judges has become a high-stakes political contest that our democracy wasn't designed to do robustly. Because judges have significant leeway in sentencing, their apparent capriciousness leads people to distrust the judicial system. Here's a proposal for fixing these problems and reducing noise in the judicial system--which could be useful, but is probably flawed and mainly just fun to think about.
A Keynesian beauty contest is a scenario John Maynard Keynes used to describe the behavior of financial markets. The idea is that a newspaper prints photographs of 100 faces and challenges readers to pick the 6 faces who will be picked most frequently by the readership as a whole. The interesting thing is that choosing the faces you personally find attractive isn't a good strategy. You're better off choosing faces which are attractive to the population at large--or, if you believe other newspaper readers are savvy, you might instead choose faces people think people think people are attracted to (and so on).
Let's suppose that the newspaper editor makes a mistake and happens to run the same 100 faces a second time. You'd be a fool not to pick the six winning faces from last time! Not only does the data show that those faces are likely to win--everyone else knows that the data shows that those faces are likely to win, and they're going to make their choices accordingly.
In other words, the Keynesian beauty contest is structured to incentivize predictability and conformism in one's judgments.
Now for fixing the judicial system.
Imagine if anyone who passed the bar was allowed to sign up to be a "Keynesian beauty contest judge" (KBCJ hereafter). Every month, they'd be sent some legal cases to make decisions on. Perhaps there'd be a transcript of some kind of court investigation to review, with identifying information removed. (The KBCJs should remain anonymous too, so they would be harder to intimidate if, for example, the defendant is a mafia boss. And they should remain unknown to each other so they can't coordinate.) Then the KBCJs review the transcript, pronounce guilt or innocence, and if the person is thought guilty, they sentence them. But there's a catch--they only get paid if their judgments are close enough to the average of the other KBCJs. So just like the beauty contest participants, they have a natural incentive to look at precedent information in order to make predictable judgments.
Of course there are a lot of issues that would need to be worked out. For example, the KBCJs might gradually get lazy over time. If all the KBCJs know that all the KBCJs usually just look at the first page of the transcript to determine guilt or innocence, ignoring subsequent pages is actually the best strategy for getting paid! To deal with this problem, maybe the KBCJs could justify why they made each decision in writing, and those justifications could be subject to random checks. They could be limited in the number of cases they took on, and fired if their opinions deviated from the majority too often. Maybe only publishing the final results of KBCJ decisions, and never publishing the voting numbers, could also serve as a bulwark against gradual shift.
Additionally, there might be cases that cover new legal ground. In those cases, however, I'd argue that trying to squeeze the case into some sort of existing law or precedent is the wrong approach, and the right strategy is for someone to do some original thinking about what's best for society, which feels like it ought to be a different process. Perhaps in addition to standard outcomes like "guilty" and "innocent", the KBCJs would have a new option available, "send it to a citizen's assembly". If enough KBCJs voted for this option, a citizen's assembly (advised by experts) would be spun up to try & figure out what to do in that kind of case.
Another possible objection to the KBCJ system is that it could work too well, and the KBCJs might faithfully continue to enforce precedents long after the population at large decided they were unjust. One hopes that the legislative arm of the government is competent to repeal unjust laws, but it's good to build in some redundancy if that isn't the case. Perhaps the KBCJs could also make use of the "send it to a citizen's assembly" option in cases where it seems obvious that a particular law could use a second look.
I'm also not sure about the idea of working from a transcript instead of seeing live people in front of you. Working from a transcript has the advantage that, for example, the decisions of the KBCJs could not be influenced by the defendant's race or gender (assuming those identifying details were removed, which should be possible in many cases). However, the KBCJs might feel their work isn't meaningful if it feels too much like shuffling papers, and they might grow cynical about their job. I have a theory that the reason why public healthcare systems tend to work OK (compared to nationalizing most other industries) is that medical workers are intrinsically motivated to help patients, so profit incentives aren't essential. On the other hand, very few people are intrinsically motivated by shuffling papers at a bank, and that's why state-run banks don't work as well. Perhaps having the KBCJs sometimes sit down for tea with the people involved in a case after a decision is made could go a ways toward solving this problem. To help prevent drama, maybe only one of the KBCJs would participate in the tea ritual, so the people involved wouldn't know for sure what decision that KBCJ made.