Suppose Alice is thinking about some feature of the real world. To make the graphs easier to draw, this example is a continuous 1d variable. Alice has a probability distribution that looks like this.
In order to help talk about this distribution, Alice labels the 2 peaks.
So now Alice can describe the situation as a foo or a bar. Of course, such description looses some information. In particular, it is bad at describing outcomes that are far from either peak, but Alice doesn't think this is a problem as she thinks such outcomes are very unlikely.
Now Bob comes along. Bob's probability distribution looks like this. (The green peaks that are slightly taller and to the right are bob's distribution. Alice's distribution is shown for comparison.)
Now Alice thinks P(Foo)=0.3 and bob thinks P(Foo)=0.4
They can have a productive discussion just using the words Foo and Bar to talk about the possibilities, without ever really mentioning the underlying distribution explicitly.
Now comes Carl. Here is his probability distribution.
Now the Foo and Bar approximation is really starting to break down. Most of Carl's probability mass lies in regions that aren't clear examples of either.
What is the probability that god exists? What is the probability covid leaked from a lab? Surely you can assign a probability to any statement? No.
For the god question, possibilities involving timetraveling humans playing at being Jesus using advanced tech to create "miracles", or pranking aliens, or simulators etc is orders of magnitude higher than the probability I assign to the fully stereotypical Judaeo-Christian god. Its edge cases. Its all edge cases. On one extreme, if someone defined god as the space-less timeless creator of all life, you might point out that evolution technically fits the bill. The space of hypothesis is very high dimensional. The stereotypical religious persons probability distribution focusses in on a tiny volume in that space. Mine doesn't. So if they try to define a region that contains their blob of probability, then most of my probability mass in that region is what they consider to be a weird edge case.
The probability I put on god existing varies by orders of magnitude depending on precisely how "god" is defined.
On the other hand, the probability covid is a lab leak, I put it around 80%. If you had an atomically precise recording of reality, it would be an easy question to answer. Just pick the RNA of covid in any covid patient and trace that RNA back through time. Watch as it self replicates in reverse and see if it enters one of the few buildings considered labs.
I mean there are cases where the covid starts in the lab canteen for reasons totally unrelated to its function as a virology lab, but those are unlikely compared to scenarios where it is either from a lab, or not from a lab.
Things I liked about this post:
Things I didn't like:
Nice post, so many hidden assumptions behind the words we use.
I wonder what are some concrete examples of this in alignment discussions, examples like your one about the probability that god exists.
One that comes to mind is a recent comment thread on one of the Late 2021 Miri Conversations posts where we were assigning probabilities to "soft takeoff" and "hard takeoff" scenarios. Then Daniel Kokotajlo realized that "soft takeoff" had to be disambiguated because in that context some people were using it to mean any kind of gradual advancement in AI capabilities, whereas others meant it to mean specifically "GDP doubling in 4 years, then doubling in 1 year".