The IPCC has a nice mapping from words to probabilities that they use when talking about global warming claims:
In this Summary for Policymakers, the following terms have been used to indicate the assessed likelihood, using expert judgement, of an outcome or a result: Virtually certain > 99% probability of occurrence, Extremely likely > 95%, Very likely > 90%, Likely > 66%, More likely than not > 50%, Unlikely < 33%, Very unlikely < 10%, Extremely unlikely < 5%.
I like the principle, but 5% is "extremely unlikely"? Something that happens on the way to work once every three weeks?
I like the principle, but 5% is "extremely unlikely"? Something that happens on the way to work once every three weeks?
It can be a bit scary, but in a lot of domains that's exactly what people mean when they say extremly unlikely.
It's extremly unlikely that humans aren't responsible for global warming.
It can be a bit scary,
And it's not even as scary as people saying “beyond a reasonable doubt” to mean something like ‘P > 75%’.
Which you can see being done by a LW regular in the LW post Thinking Bayesianically, with Lojban. So it's not like this is something no one does, or something only idiots do.
I'm being slightly unfair. The actual figure being described in those terms is nearer to 76%.
You want to translation from numbers to certainty to be
0->1/2=50%
1->2/3=67%
2->4/5=80%
3->8/9=89%
4->16/17=94%
5->32/33=97%
6->64/65=98.5%
7->128/129=99.2%
8->256/257=99.6%
9->512/513=99.8%
10->1024/1025=99.9%
n->2^n/(2^n+1)
Here, the percent p is given the number n, such that it would take n more bits of information to convince you that you are wrong than it would take to convince you that you are correct. These numbers are very natural, and for some purposes, it would be better to use these numbers than to use the percents.
Notice that this luckily (Maybe you planned this) fits with your number system, except that 99.9 is really really certain compared to what you would expect would come one number after 95%. I know that it would be very hard to get that kind of detail in the intermediate values between 4 and 5, but even if you only ever say 0, 1, 2, 3, 4, 10, I still think it is worth it to emphasize the big difference between 95% and 99.9%.
Very good point. I like how this could go higher or have in-between values quite easily. In retrospect an equation like this makes much more sense than an intuitive guess, what I wrote down was mostly to use as a start.
I'm not sure if this is exactly the perfect equation for this, given that I think I'd probably want them to be a bit more spaced out if they went to 10 (going further in confidence past 99.9% perhaps) ~i1.
I like the relation to this system to bits of evidence and I agree that having some form of short hand would potentially be a lot more useful then having to add a large number of verbal ticks and caveats. ~i1.
I am concerned that a possible flaw in this shows that condensing all of the evidential certainty into a single number makes it easier to cause a typo in it, which would change the entire tone of your post. Example: ~i1: ~i11. While that is one character, it is the difference between "I'm just sort of spitballing this, based on a quick gut feeling." And "This is very self evident to me." However, you can have the same problem with emoticon typos, such as :) :(, so it isn't as though this is exclusive to the proposed system. ~i1r1
That being said, I find the general idea very helpful, because I could note someone replying to one of my posts, showing its flaws, and seeing "Oh, I didn't believe that all that firmly anyway, so I should update to the other side." In general, I think this would make me feel less defensive when given counter evidence, which would be a good thing. ~i2.
Why not just use parenthetical probabilities where it's useful? I'm pretty sure (80%) that this is more likely to catch on than your proposal.
I feel like probabilities are longer (80 as opposed to 3) and in some ways contain more information (1 out of a 100) than we can really state ~i1. In effect, we have few significant figures.
That said, the understandability of percentages may outweigh this benefit (-~i1).
Reminds me of the discussion in Through The Language Glass of the Matsés people of the Amazon.
Their language has a built-in concept of evidentiality -- every time they say anything about anything, their language requires them to express the amount of evidence for the statement -- 'seen with my own eyes' until 'mere hearsay' -- paper.
I'm ~.9 confident that it's more understandable to state confidence levels.
That said, I do like the idea of grammatical evidential tags to conveniently express types of evidence -- "Someone I trust told me this", "I observed this directly", "this seems to follow from other things I know", etc. English has a few of these (e.g., "There must be an X" can clearly mean, if said a certain way, that I don't know there is an X from direct observation, but I infer it from other available data) but not very many and they're not very robust.
Actually, I think that while this might be true now, it would actually be helpful for us to start thinking about probabilities as integers on a log scale like I described. If we all switched to this system, I think we would be better off. Also, as it is now, we do not state confidence levels much, and this might encourage us to do so.
I agree that a community that conventionally used a log scale like you describe to communicate confidence levels ("probabilities") would have advantages. It is not clear to me how, or if, this community can make a transition to that convention.
Perhaps a place to start would be by modeling the behavior you suggest, and adopting the habit yourself? (e.g., you say you "think" that starting to think about probabilities as integers on a log scale would leave us better off... what is that expressed in your suggested notation?)
An even bigger problem is expressing estimate stability -- the difference between "50% confidence the coin will land heads, because I'm very confident it is a fair coin" and "50% confidence the coin will land heads, because I know the coin is either two-headed or two-tailed, but I have no information about which one". Whereas confidence levels at least have vague language, like "very unlikely" or "somewhat likely", estimate stability doesn't, except for even more useless phrases like "but I could very well be wrong" or "but that may change as we find out new information".
This will never catch on unless someone can read the statement on its own, and through google know what it means. (logit=2)
How certain am I of the above statement? [pollid:566]
I agree that standardized expressions for levels of confidence are desirable. However, anything that needs explanation before use is unlikely to catch on.
I suggest we try and make a habit of saying "I'd bet against that ". Such as: "I'd bet $100 against $50 that Democrats will win the next US presidency."
That quantifies confidence, gives others an opportunity to challenge the claim and is frankly more colorful than cryptic codes.
I think most debates that we have aren't as easily verified as who wins the next US presidency. If you do make claims that are easily verified post them to predictionbook and link to the predictionbook entry.
If you could get past the bootstrap problem, it'd be used more than the existing version, simply due to being faster to type. ~c2
There is the issue of fitting it into existing grammatical rules, but that's not strictly speaking required.
Several years ago some people made the language E-Prime in large part to make this uncertainty crystal clear
E-prime doesn't do anything in particular for uncertainty as such; mostly what it does is prevent the mind projection fallacy and keep various kinds of opinions and confused thinking from masquerading as facts.
I believe that we don't need different kinds of uncertainty to have a shorthand. I would just go with a single number, along with some code telling you that this is a certainty measurement. Also, it should probably be something long enough to be searchable. It should be something we put on the wiki, and that can be searched just from seeing it in practice, and then some of us could use it and some of us could choose not to use it.
While this sounds like it would be useful, it would also turn a lot of people off to the site ~i4
I'm not suggesting that we make this a mandated LessWrong policy, but I think it may be fun to play with personally ~i4. I imagine that something more sensical would be created after some thought and work ~i2.
Also it could make a lot of sense for an organization to accept a standard like this rather than a blog community. For instance, many startups or EA orgs already use a lot of internal jargon ~e3i2, so something like this doesn't seem like a clear negative in that light ~i2.
Concerning the appearance aspect, it may be possible to hide the phrases (similar to the markdown mention earlier) with a small symbol that one could "inspect" when interested.
Is there a significant and communicated difference between i3 and i4 in most circumstances where you'd use this terminology? Is this scale -- or any single-digit scale -- going to provide enough accuracy for c5 to be truthful?
At a deeper level, how confident are you for your evaluation of your own confidence levels? You mention that humans aren't very good at accessing probability, but I think it needs to be examined further. A 15% variance in the meaning of "very certain" might as well be noise when even the individual studies that make up consensus knowledge provide significantly greater ranges to justify their confidence levels.
((More shallowly, do you discuss these matters often enough with the same individuals that explaining this standard is likely to save time, even when compounded over a few years, compared to insisting on the arabic numerals? Or are you focusing on other benefits?))
Most things I find I discuss are highly uncertain, but it can be really confusing and wordy to state that uncertainty in writing. In this last sentence for example I felt the need to write “I find” to point out uncertainty, for example.
First, people are really bad at agreeing on probabilities. So if I say something is “very certain”, that could mean 80% chance to me and 95% chance to you. This is rigorously explained in the Failure of Risk Management (by the same author from How to Measure Anything), where it is explained further to say that this is especially true of risk managers.
Second, there aren’t too many words to use to indicate uncertainty. I find that I need to repeat the same ones over and over again. And when they are used, these words can be quite wordy and confusing.
Several years ago some people made the language E-Prime in large part to make this uncertainty crystal clear.
E-Prime (short for English-Prime, sometimes denoted É or E′) is a prescriptive version of the English language that excludes all forms of the verb to be. E-Prime does not allow the conjugations of to be—be, am, is, are, was, were, been, being— the archaic forms of to be (e.g. art, wast, wert), or the contractions of to be—’s, ‘m, ‘re (e.g. I’m, he’s, she’s, they’re).
Some scholars advocate using E-Prime as a device to clarify thinking and strengthen writing.[1] For example, the sentence “the film was good” could not be expressed under the rules of E-Prime, and the speaker might instead say “I liked the film” or “the film made me laugh”. The E-Prime versions communicate the speaker’s experience rather than judgment, making it harder for the writer or reader to confuse opinion with fact.
While I do intend to look more into E-prime, it seems like a bit much to use on a routine basis.
I propose that we instead use a symbol at the end of our sentences or propositions to indicate uncertainty.
A scale would have to be created of course in order to indicate what these levels are. My guess is that the optimal (for usefulness, popularity, and accuracy) amount of levels would be around 5-10, especially because we aren’t very good at accessing probability.
Here’s one example that makes sense to me:
0. ~50% 1. ~65% 2. ~80% 3. ~90% 4. ~95% 5. ~99.9%
In cases where something is unlikely, this would just work the opposite way (50% to 0.01%).
I think that any representation of certainty would have to be achievable with ASCII characters, if not the English keyboard. Here are some possibilities. Each is shown to be representative for a level of 4/5, according to a scale similar to what is shown above.
My personal favorite at this point is to have a number with the tilda sign “~”, with a symbol for indication (like the “c” or “?”). The dashes are be difficult to read and more confusing to newcomers (c3).
So far we’ve assumed that the definition of ‘uncertainty’ is relatively clear, but sometimes there are different definitions of uncertainty.
For instance, there’s the certainty of “the existing scientific literature strongly agrees that evolution is true”, and the certainty of “I personally am very certain that the Paleo diet is good, even though others might disagree.”
These could be indicated by different symbols. This would require a small dictionary of symbols/standards, but this may not be very unreasonable.
Say we use ‘c’ to indicate ‘consensus’ and ‘i’ to indicate ‘personal intuition’, and ‘r’ to indicate ‘personal research/rationality’. Not all of these would need to be used in every instance, only the ones that are instantially relevant.
Some statements could be as follows:
Of course, we’d need a definition for this, which is effectively a standard. For now I’ll call it “Uncertainty Notation V0.1” I’ll try it out in future posts as an experiment. HTML Codes Reference