Interesting idea. I think that rather than a Brier score or log-score (which just indicates how accurate the prosecutor was) it would be better to use something that also indicates the extent to which they specifically over-stated charges. Maybe just something like "On average, the total sentence for defendants charged by this prosecutor was 4.7% of the total of all charges made", though it seems like there would be all sorts of difficulties actually doing that in a way that makes sense.
I think that accuracy is better for a prosecutor to optimize then not overstating the likelihood.
If I'm charged with a crime and the prosecutor tells me that the real chance of getting put into jail is 60% when it's actually 90% then I'm harmed by having more hopes then reasonable about being able to stay out of jail.
Besides the main number, all the history should however be available so that a defense lawyer could also calculate whether a prosecutor with a bad Briers score under or overcharges.
In addition to accuracy being useful to optimize for, I expect it to be easier to find a political consensus around "prosecutor should be accurate when making charges" as that's desirable for both the right and left.
I think you want to give prosecutors incentives both to be accurate and not to overstate, and also to give people accused of crimes some indication of what they're actually likely to end up with if they go to trial.
You might be right that doing something that looks "bidirectional" might be easier politically.
For something that's impossible to implement, this seems like a surprisingly small step toward EITHER futarchy and the embedding of good conditional predictions in more policy and behavioral decisions, OR to resolving the broken incentives of prosecutors (convictions, not truth).
For something that's impossible to implement, this seems like a surprisingly small step
I don't believe that it's impossible to implement. Good policy making is about adding a lot of small steps together. It's one small step that can be added to a legal reform bill in any state.
We have a climate where a liberal US state might very well say: "Let's put all reasonable proposal to improve our legal system into a bill and put it up for a vote".
Here it's also worth noting that we are at a time where crypto people are inventing new legal systems. https://kleros.io/ for example is really interesting.
OR to resolving the broken incentives of prosecutors (convictions, not truth).
This system doesn't completely change the incentives of prosecutors for convictions to truth but it moves them in that direction.
Chance of being convicted is very weakly correlated with actual guilt, or even procecutor knowledge of guilt. Consider the case of an illegal search that definitively proves guilt, but cannot be presented at trial. The prosecution would probably want to proceed with their 10% case, since they know the defendent is guilty, even though the jury won't have key information.
Also, too, predicting outcomes is a skill separate from ability to generate outcomes. The Tails Come Apart, Goodheart, and all that..
Also, also, too almost no prosecutors are directly elected, judging people by any raw metric is wierd and not done - there would be some arbitrary cutoffs added to translate it to a quantified grading scale, and, as other commenters said, the root cause here is way deeper than "the DA charges 17 counts of assault, one for each blow, plus 3 Kidnapping, one for each grab, etc. Etc. Etc.
Consider the case of an illegal search that definitively proves guilt, but cannot be presented at trial.
Disincentivizing gathering information from illegal searches is a feature and not a bug. The fact that the process I propose has side-effects like discouraging illegal activity like that shows it's power.
I don't think the system should be designed to give people who do illegal things more room to maneuver but to reduce that room.
Also, too, predicting outcomes is a skill separate from ability to generate outcomes.
The present system in the United States doesn't select for the outcome of prosecutors persuing justice but often rewards them for doing unjust things like maximizing the amount of people they put behind bars.
I do think that prosecutors who's mental focus when charging a person isn't "how do I maximize the chances of getting this person behind bars" but who are also taking the outside view and think clearly through the likelihood of the person being convicted being are more likely to persue justice.
Also, also, too almost no prosecutors are directly elected, judging people by any raw metric is wierd and not done
The point is not to say that this should be the only metric that's used to judge prosecutor performance alone but to design the system so that it's one of the metrics that's visible when decisions about prosecutor performance get made.
As far as judging people by performance metrics being weird, we do have a problem that many people prefer to judge expertise by college degrees then by actual performance metrics.
An alternative approach that feels less gameable:
Every time a plea bargain is reached, there is a small chance (1 in 1000?) that we randomly select the case for review. The defendant is imprisoned as per the terms of the plea bargain, but then we also bring the case to trial (federally funding a defense).
If the review trial finds the defendant guilty of the charges they pled to, we say 'okay, seems like the plea bargain was fair, the defendant would have been found guilty anyway'.
If the review trial finds the defendant innocent, we release them and publicize this fact. If multiple reviews under the same prosecutor exonerate the defendant, it starts to look like that prosecutor is threatening charges to coerce defendants into unfavorable plea bargains.
A 1 in 1000 chance means that the cost of this should not grow too excessive compared to existing costs of the legal system, but also means that a prosecutor who is habitually coercing defendants into taking unfavorable plea deals is very likely to get caught.
It seems to me like your proposed system is more gameable by influencing the people who do the review and who select which case to randomly review then the system I proposed.
Furthermore, it does have no direct impact on the decision making process of the prosecutor when pressing charges.
The ability of the prosecutor to accurately access the likelihood can be measured via the Briers score or a Log score.
Scoring predictions requires knowing the outcomes. But wouldn't the outcome depend on whether the accused takes plea deals and such?
You only score the predictions that actually have the court decide the outcome or that have the prosecutor drop the charges without a deal and not those that get decided by plea deals.
Suppose that the prosecutor has some random noise in their charges, such that they sometimes overcharge a bunch and sometimes undercharge a bunch. In that case it seems reasonable to suppose that things are more likely to go to court when the prosecutor is overcharging and the accused therefore thinks they can get more of the accusations dropped. But this would mean that the prosecutors are evaluated on a subset of the charges that are systematically too high, and therefore to compensate they end up lowering their assessed probabilities below what is actually counterfactually accurate if people just went to court about it always.
I don't know how big a problem this would be, but it seems like something that would be good to evaluate in the proposal.
But this would mean that the prosecutors are evaluated on a subset of the charges that are systematically too high, and therefore to compensate they end up lowering their assessed probabilities below what is actually counterfactually accurate if people just went to court about it always.
While I do think such an effect could exist, it should be relatively constant and I would expect defense attornies to understand the effect and correct for it in their decisions when they advice their clients.
I would expect that many juries will automatically throw out a 10% or 20% charges because there will be people on the jury who would argue that the prosecutor thinking that there's only a 20% chance that a charge holds means that there's reasonable doubt against the charge.
So then nobody would assign low likelihood charges? I guess that’s a feature of the proposed system, but it seems like a flaw to me. (Probably minor)
I do think that the legal standard of reasonable doubt means not assigning low likelihood charges. That's how I understand the principle of "reasonable doubt". The feature of this system it that it lets the jurors decide what they think reasonable doubt means and the system doesn't set the meaning for it.
Yeah, I agree with that. But 20% seems within that range for me (especially for very bad crimes). Is 10%? Not sure.
The system as I proposed by design doesn't say "Prosecutors are only allowed to bring charges with at least X%" but works to let an equilibrium play out.
I think this feature is very neat because it's a mechanism that doesn't work through bureaucratic rules and arbitrary cutoffs but through jurors making decisions about what's right in individual cases and the prosecutors learning from the judgements the jurors make.
It would be a legal obligation for a prosecutor when making a charge to publically enter the charge along with the likelihood into a database. Making errors with filing charges will be problematic for the prosecutor the same way it's problematic now, it can make a court throw out the charges on procedural grounds.
Then when an election comes around the election commission has the job to calculate the Briers score to put it on the ballot.
The job of administrating the database could be done be done by the court where the charges get filled, so that it's outside of the realm that the prosecutor can influence.
If upfront assessments are provided, I expect the defense bar would gleefully keep track of such things.
They already informally track the behavior of the DA's offices they deal with. They're extremely organized in some areas and in near-constant communication with one another.
How would prosecutors be trained and incentivized to make and report accurate predictions? Who would be funding the defense attorneys, especially overworked public defenders, to learn to properly evaluate prosecutor's claims? The US legal system already isn't exactly known for being honest today even in terms of things like rules about disclosing exculpatory evidence to the defense. Also, the famous saying about indicting a ham sandwich is anecdotal evidence about how much credit (grand) jurors give prosecutors' likelihood assessments in a pre-trial, non-adversarial environment, and I don't find it encouraging.
Also, I understand that one of the big issues with plea deals is that it is often in a defendent's individual interest to plead guilty to a lesser charge they're innocent of, even if they're pretty sure they wouldn't be found guilty at trial, either because they're risk averse (not necessarily unreasonable when years of your life are on the line), or because the agreed-on punishment will be less than the time in jail, and cost in fees, they'd incur just by going to trial even if exonerated. Data on likelihood of conviction wouldn't help with that.
At a more basic level, I think this solves the wrong problem. If, as you say, it is in society's best interests to keep legal costs low, then it follows that there is little benefit to society from the larger penalties imposed by overcharging defendants. This suggests that the problem lies in the criminal code and sentencing guidelines, and that these need serious updating.
And given how biased our legal system is and has historically been, I believe a proposal based on conviction odds has basically the same problems as current AI tools for criminal sentencing and risk assessment. Large racial and economic biases, among other problems.
Edit to add: I do like the idea of giving defendents honest info about their likelihood of conviction if a case goes to trial. I'm extremely skeptical of the idea of having prosecutors provide that evidence, and wary of letting prosecutors, juries, or judges see it.
How would prosecutors be trained and incentivized to make and report accurate predictions? Who would be funding the defense attorneys, especially overworked public defenders, to learn to properly evaluate prosecutor's claims?
The main thing a prosecutor cares about is getting reelect as a prosecutor. Writing the metric that measures whether they are good at reporting accurate predictions and writing it on the ballot is one of the strongest ways to incentives prosecutor.
Even if the prosecutor wants to go to another office after being a prosecutor, doing badly at the metric for prosecutors that gets written on their ballot, will be hold against the prosecutor.
Shifting the incentives on which prosecutors get evaluated is part of what this proposal is about.
Also, I understand that one of the big issues with plea deals is that it is often in a defendent's individual interest to plead guilty to a lesser charge they're innocent of, even if they're pretty sure they wouldn't be found guilty at trial, either because they're risk averse (not necessarily unreasonable when years of your life are on the line), or because the agreed-on punishment will be less than the time in jail, and cost in fees, they'd incur just by going to trial even if exonerated. Data on likelihood of conviction wouldn't help with that.
Let's say a defendent Joe correctly believes that his chance of being found guilty at trial is 10%. This means that a prosecutor that's well calibrated because that's in his interest for getting reelected would also tell him that his chance of being found guilty is 10%.
If I'm a risk adverse person then knowing that not only I believe that my chances of being found guilty is 10% but that the prosecutor whose has access to evidence that I don't see also believes the chance is 10% is very helpful.
Then when I'm in the court room my lawyer can ask the prosecutor "Why did you bring that case when you only think that there's a 10% chance that my client will be found guilty?". Journalists could write story "Prosecutor Smith brings a case against Joe to trial even so he only believes that there's a 10% chance that Joe would be convicted".
That's not good for the public reputation of prosecutor Smith and thus won't charge Joe in the first place and has no leverage to get Joe to accept the plea deal.
Who would be funding the defense attorneys, especially overworked public defenders, to learn to properly evaluate prosecutor's claims?
The same way they are currently learning to evaluate the charges that the prosecutor makes. If the defense completley ignores the numbers nothing is lost over the status quo.
At a more basic level, I think this solves the wrong problem.
Rejecting a proposal because it doesn't solve the problem you most care about it is bad. I agree that it would be good if the penal code would be changed and sentencing guidelines be reduced but that's not the issue my post is addressing.
that's not the issue my post is addressing.
Sorry, then I think I made a bad assumption about why you were trying to solve the problem of overcharging. I added that at the end of my comment and probably shouldn't have. Still, it's not the reason I initially rejected the proposal. I honestly don't think your proposal actually solves the overcharging problem or would make plea deals much fairer.
Then when I'm in the court room my lawyer can ask the prosecutor "Why did you bring that case when you only think that there's a 10% chance that my client will be found guilty?" Journalists could write story "Prosecutor Smith brings a case against Joe to trial even so he only believes that there's a 10% chance that Joe would be convicted".
Journalists could write that, true. But if we're relying on juries and voters to properly evaluate that kind of statistic as a reflection on the prosecutor's job performance or the case's merits, then hopefully they'll also understand that the data point is immaterial to the question of whether the defendant is guilty. I can be very confident a policeman is guilty of murder, or a politician is guilty of accepting a bribe, or a frat boy is guilty of rape, and still correctly believe the probability of conviction is low. I'm not sure how a court would regard such an argument, but to this layman it seems that to whatever extent the percentage is based on the evidence, giving the number as an additional data point is double-counting evidence (or a way of introducing information based on evidence not presented at trial or inadmissible at trial, which is just as bad or worse), and to whatever degree it is based on assumptions about the judge and jury's behavior and thinking, it's speculation.
Overall I'm very skeptical that the enforcement mechanism you proposed to incentivize prosecutors to be honest is anywhere near strong enough. It might be better to scale the prosecutor's pay to how well calibrated their estimates are, for those cases that make it to trial, with additional penalties like removal from office for being too far off. Or better yet, do a literal randomized trial where the prosecutor who offers the plea deal and makes the estimate is different from the prosecutor who goes to trial, and evaluate the former by accuracy and the latter by conviction rate.
Overall I'm very skeptical that the enforcement mechanism you proposed to incentivize prosecutors to be honest is anywhere near strong enough.
The incentive it's as strong as the desire for people who evaluate the prosecutors want it to be. You can also make it stronger by linking it to bonus payments if you want. If you do pay prosecutors any bonuses it would make sense to do that, but I think it's generally better to pay prosecutors a fixed salary.
In any case, even if many prosecutors give crapy numbers those numbers can just be ignored by the defendent and produce no disadvantage over the status quo.
I'm not sure how a court would regard such an argument, but to this layman it seems that to whatever extent the percentage is based on the evidence, giving the number as an additional data point is double-counting evidence
Evidence is by it's nature very complex. If a journalist writes an article then a reader doesn't know whether the journalist gives a fair representation of the issue at hand.
It takes a lot of effort to understand all the evidence that goes into all the cases where a prosecutor charges people to evaluate the prosecutor. If you have the prosecutor summarize the strength of the evidence that evaluation gets easier.
Or better yet, do a literal randomized trial where the prosecutor who offers the plea deal and makes the estimate is different from the prosecutor who goes to trial
I don't think you can easily do a trial because it's a systematic intervention that needs to run a few years for people to adept to the new system before it leads it's provides most of it's benefits.
It takes a lot of effort to understand all the evidence that goes into all the cases where a prosecutor charges people to evaluate the prosecutor. If you have the prosecutor summarize the strength of the evidence that evaluation gets easier.
That's true, but I think you're being very optimistic, both in the ability of defendants and defense council to ignore or evaluate information the other side in an adversarial system claims is their true opinion, and in the ability and interest of the public in properly evaluating the job performance of prosecutors in local elections based on actual data. I think both are possible, and would be very valuable, but can't be achieved without much deeper and broader reforms to make the underlying justice system more open, transparent, and trustworthy.
I don't think you can easily do a trial because it's a systematic intervention that needs to run a few years for people to adept to the new system before it leads it's provides most of it's benefits.
Sorry, I didn't mean a trial as an experiment, I meant literally running legal trials this way, where in general the prosecutor that tries a case is not the one that produces the conviction probability estimate. Then, grade each both on the accuracy of their assessments, and separately on their conviction rates in trials they prosecute. I'd say either the one trying the case or a separate third prosecutor should have final say on which charges to bring. I think this would eliminate a lot of the potential for perverse incentives.
That's true, but I think you're being very optimistic, both in the ability of defendants and defense council to ignore or evaluate information the other side in an adversarial system claims is their true opinion
At would expect that in the beginning after the reform defendants and defense council would not trust the probability at all.
I would expect that trust in the numbers will only come when the system works well that they provide valid information.
I think both are possible, and would be very valuable, but can't be achieved without much deeper and broader reforms to make the underlying justice system more open, transparent, and trustworthy.
Good political reforms aren't about doing one thing but multiple things. There's a reason why Obamacare is 906 pages.
While I think there are some flaws with this proposal, my biggest question is "Why not do other things that are done abroad and we know work?"
The US system for deals is really strange. In the UK and many european countries there's no such thing as a deal. The police can charge you with a crime. The earlier in the process you admit guilt, the bigger % a sentence reduction you get. This seems to solve the overcharging problem, a prosecutor can't charge you with crime X and then give you a deal for crime Y, and aligns incentives better as prosecutors can't fear people into taking plea deals as easily. It also preserves much of the benefit of plea bargaining by still allowing suspects to confess when the evidence is overwhelming and hence save lot's of time and money.
In Germany where I live a prosecutor has a legal obligation for prosecuting cases of crimes if he believes the defendent is guilty. A prosecutor is not allowed to say "Well this law is stupid so I won't prosecute people for breaking it" while in the US that's within the rights of a prosecutor. This right allows for plea deals to happen in the US.
If you change this basic part about what the mandate of the prosecutor happens to be, you will suddenly get a lot of very bad laws enforced that currently aren't enforced. If laws against oral sex in some states suddenly get enforced that will create pressure to get rid of those laws but it's won't be pretty.
But even if the US would get rid of their current legal system and switch to the German one (which is obviously much better thought out and superior ;) ) and have a culture change to make American bureacrats as law abiding as German one's there's still a process whereby plea bargining could cut down on legal costs and time of the trial. There's a benefit in getting a verdict in a few days instead of years through plea bargaining.
In many situation a plea deal is about more then just getting the person to admit that they are guilty but also to reveal certain other information that's of interest to the prosecutor. Being able to make a deal to get the information can be useful for the government.
Another idea: as things stand today, prosecutors are only allowed to file charges when they have "probable cause". Courts won't put a number on what that means, but you might reasonably approximate it as 25% certainty that the defendant did it. We could insist on a higher standard - preponderance of the evidence (which courts do define as > 50% certainty) or clear and convincing evidence (which courts won't put a number on but you could reasonably approximate as 75% certainty).
As I wrote, I don't think such a number should be set directly by law.
A judge can however take the number into account when deciding whether there's "probable cause". Individual jurors might also decide to categorically treat charges with are under some number as baseless charges.
A voter might decide that he doesn't want to reelect prosecutors that fill charges with <50% or <75% certainty. Prosecutors who fill charges that they believe to be low probability can be judged for that behavior democratically and different voters can disagree with each other what standards they have for prosecutors.
Having new dimensions based on which voters can evaluate prosecutors and thus incentvise different behavior for prosecutors then just a maximum of convictions is part of the point.
I think you are forgetting a major player in the criminal justice system: the defense attorney. The defense attorney already has both the expertise and the incentive to accurately advise the defendant as to the likelihood of a conviction on each charge.
The defense attorney doesn't have access to all the evidence that the prosecutor has. This proposal essentially forces the prosecutor to state at the beginning how strong his evidence is.
Having common knowledge about what the prosecutor believes about the likelihood that his charges stick is very useful for the defense attorney when it comes to negotiating a plea deal.
Being able to argue in court "Even the prosecutor only believes that there's a 10% chance that this charge sticks" is a valuable move even if the defense attorney knows that there's a 10% chance.
That's not a bug, that's a feature! The prosecutor knows what evidence the prosecutor has, but the defendant knows whether he did the crime. We want the defendant to make the plea decision blind to the strength of the prosecutor's evidence, because guilty defendants will guess that the prosecutor has strong evidence and plead guilty (even if that guess is wrong), and innocent defendants will guess that the prosecutor has weak evidence and proceed to trial. This is how we want the system to work.
I don't think the intention of the plea deal system should be to get people who wouldn't be found guilty in a court of law to plead guilty for any reason.
I think subverting the right to a trial by jury by creating a system that encourages people to plead guilty who the jury wouldn't find guilty is bad. Overall I think the defendant has to little power in the US system and moving the system to give them more powers is good even if that occasionally means guilty people go free.
In criminal cases in the United States prosecutors often add a lot of charges for a defendant to have ammunition for coercing the defendant into a plea deal. This is toxic because the defendant doesn't know which of those charges are likely to hold up in court if the case would be decided by a jury. Given that there's little cost to the prosecutor for adding additional charges, defendants are often overcharged.
I propose that whenever a prosecutor files a charge for a defendant, the prosecutor should state the likelihood that in the absence of a deal the court will find the defendant guilty of the charge. The ability of the prosecutor to accurately access the likelihood can be measured via the Briers score or a Log score.
The current score should be publicly accessible on the website of the court. This allows the defendant to know whether they can trust the likelihood values the prosecutor gives. The score should also be printed on ballots when the prosecutor seeks reelection to create much higher incentives for the prosecutor to give the correct likelihood then convicting a lot of people.
After the prosecutor provides the likelihood for the charges it's much easier for a defendant to make a good decision about whether taken a given plea deal is in their interest. Prosecutors with a good Briers score will be able to make more plea deals to reduce their overall workload because it's easier for the defendant to know that a deal is in their interest.
While this reform wouldn't fix all problems with plea deals, as some plea deals are due to the defendant being given charges that would actually hold up in court given the existing criminal code, the reform will provide defendants with fairer plea deals. Defendants getting fair plea deals is good for the system given that it keeps overall legal costs down.
I would expect that many juries will automatically throw out a 10% or 20% charges because there will be people on the jury who would argue that the prosecutor thinking that there's only a 20% chance that a charge holds means that there's reasonable doubt against the charge.
This reform is both in the interests of citizens who care about law and order and citizens who care about reducing sentencing overall as everybody should be interested in prosecutors providing fair plea deals.