I strongly value learning and therefore feedback - from positive to negative, and from the very minor (e.g., typos) to the very major (e.g., an entire post of mine is pointless or wrong - "that which can be destroyed by the truth should be", and all that).

Also, in my comments and posts, I’ll likely often try to explain things multiple times with different phrasings, and to provide (ideally multiple) concrete examples, to aid readers in “triangulating” what I actually intend to convey (which may be abstract). This is based partly on some takeaways from my studying and experience while I was a teacher for 2 years (specifically, some of the takeaways which I’m relatively confident are not just unreplicable garbage). (Just thought I’d mention that in case anyone wonders about that aspect of my writing style.)

MichaelA's Comments

MichaelA's Shortform

Thanks for the feedback!

I think sometimes posts with a lot of synonyms are hard to have take aways from, because it's hard to remember all the synonyms. What I think is useful is comparing and contrasting the different takes, creating a richer view of the whole framework by examining it from many angles.

Yeah, I'd agree with that, and it's part of why fleshing this out is currently low priority for me (since the latter approach takes actual work!), but remains theoretically on the list :)

Siren worlds and the perils of over-optimised search

I've just now found my way to this post, from links in several of your more recent posts, and I'm curious as to how this fits in with more recent concepts and thinking from yourself and others.

Firstly, in terms of Garrabrant's taxonomy, I take it that the "evil AI" scenario could be considered a case of adversarial Goodhart, and the siren and marketing worlds without builders could be considered cases of regressional and/or extremal Goodhart. Does that sound right?

Secondly, would you still say that these scenarios demonstrate reasons to avoid optimising (and to instead opt for something like satisficing or constrained search)? It seems to me - though I'm fairly unsure about this - that your more recent writing on Goodhart-style problems suggests that you think we can deal with such problems to the best of our ability by just modelling everything we must already know about our uncertainty and about our preferences (e.g., that they have diminishing returns). Is that roughly right? If so, would you now view these siren and marketing worlds not as arguments against optimisation, but rather as strong demonstrations that naively optimising could be disastrous, and that carefully modelling everything we know about our uncertainty and preferences is really important?

Risk and uncertainty: A false dichotomy?
The casino owners don't gamble on their income.

Maybe this is a matter of different definitions/connotations of "gamble". Given that the odds are in the casino's favour, and that they can repeat/iterate the games a huge number of times, the results do indeed tend to converge to the expected value, which is in the casino's favour - I'm in total agreement there. The odds that they'd lose out, given those facts, are infinitesimal and negligible for pretty much all practical purposes. But it's like they asymptotically approach zero, not that they literally are zero.

It seems very similar to the case of entropy:

The Second Law of Thermodynamics is statistical in nature, and therefore its reliability arises from the huge number of particles present in macroscopic systems. It is not impossible, in principle, for all 6 × 1023 atoms in a mole of a gas to spontaneously migrate to one half of a container; it is only fantastically unlikely—so unlikely that no macroscopic violation of the Second Law has ever been observed.

But in any case, it seems your key point there, which I actually agree with, is that the deal is better for the casino (partly) because they get to play the odds more often than an individual gambler does, so the value they actually get is more likely to be close to the expected value than the individual gambler's is. But I think the reason this is better is because of the diminishing marginal utility of money - losing all your money is way worse than doubling it is good - and not because of the risk-uncertainty distinction itself.

(Though there could be relevant interplays between the magnitude of one's uncertainty and the odds one ends up in a really bad position, which might make one more reluctant to avoid "bets" of any kind when the uncertainty is greater. But again, it's helpful to consider whether you're thinking about expected utility or expected value of some other unit, and it also seems unnecessary to use a categorical risk-uncertainty distinction.)

Seem one implication of your position is that people should be indifferent to the following two settings where the expected payoff is the same:
1) They toss a fair coin as many times as they want. If they get heads, they will receive $60, if they get tails they pay $50.
2) They can have the same coin, and same payoffs but only get one toss.
Do you think most peoples decision will be the same? If not, how do you explain the difference.

Regarding whether I think people's decisions will be the same, I think it's useful to make clear the distinction between descriptive and normative claims. As I say in footnote 1:

Additionally, it’s sometimes unclear whether proponents of the distinction are merely arguing (a) that people perceive such a distinction, so it’s useful to think about and research it in order to understand how people are likely to think and behave, or are actually arguing (b) that people should perceive such a distinction, or that such a distinction “really exists”, “out there in the world”. It seems to me that (a) is pretty likely to be true, but wouldn’t have major consequences for how we rationally should make decisions when not certain. Thus, in this post I focus exclusively on (b).

So my position doesn't really directly imply anything about what people will decide. It's totally possible for the risk-uncertainty distinction to not "actually make sense" and yet still be something that economists, psychologists, etc. should be aware of as something people believe in or act as if they believe in. (Like how it's useful to study biases or folk biology or whatever, to predict behaviours, without having to imagine that the biases or folk biology actually reflect reality perfectly.) But I'd argue that such researchers should make it clear when they're discussing what people do vs when they're discussing what they should do, or what's rational, or whatever.

(If your claims have a lot to do with what people actually think like, rather than normative claims, then we may be more in agreement than it appears.)

But as for what people should do in that situation, I think my position doesn't imply people should be indifferent to that, because getting diminishing marginal utility from money doesn't conflict with reality.

In the extreme version of that situation, if someone starts with $150 as their entire set of assets, and takes bet 2, then there's a 50% chance they'll lose a third of everything they own. That's really bad for them. The 50% chance they win $60 could plausibly not make up for that.

If the same person takes bet 1, the odds that they end up worse off go down, because, as you say, the actual results will tend to converge towards the (positive in dollar terms) expected value as one gets more trials/repetitions.

So it seems to me that it's reasonable to see bet 1 as better than bet 2 (depending on an individual's utility function for money and how much money they currently have), but that this doesn't require imagining a categorical risk-uncertainty distinction.

Risk and uncertainty: A false dichotomy?

(As per usual, my comments are intended not to convince but to outline my thinking, and potentially have holes poked in it. I wouldn't be willing to spend time writing as many paragraphs as I do if I thought there was 0 chance I'd end up learning something new as a result!)

I don't think either the gambling or market analogies really shows either that the risk-uncertainty distinction makes sense in categorical terms, or that it's useful. I think they actually show a large collection of small, different issues, which means my explanation of why I think that may be a bit messy.

The house sets up the rules so they have something like a 3% edge on all the customers. They have no idea on what any given bet will pay off for them but over all the bets over the whole year, they can be pretty sure that they keep 3% if all money put down during the year.

I think this is true, but that "something like" and "pretty sure" are doing a lot of the work here. The house can't be absolutely certain that there's 3% edge, for a whole range of reasons - e.g., there could be card-counters at some point, the house's staff may go against their instructions in order to favour their friends or some pretty women, the house may have somehow simply calculated this wrong, or something more outlandish like Eliezer's dark lords of the Matrix. Like with my points in the post, in practice, I'd be happy making my bets as if these issues weren't issues, but they still prevent absolute certainty.

I don't think you were explicitly trying to say that the house does have absolute certainty (especially given the following paragraph), so that's sort of me attacking a straw man. But I think the typical idea of the distinction being categorical has to be premised on absolute certainty, and I think you may still sort-of be leaning on that idea in some ways, so it seems worth addressing that idea first.

But lets know thing about a case of a gambler. What if he can get the same edge as the house was getting? Is he really in the same situation of risk management as the casino? I think that depends. We might know what the probabilities are and the shape of the function but we don't really know how many times we need to play before our sampling starts to reflect that distribution -- statistics gives us some ideas but that also has a random element to it. The gambler's has to decide if he has a budget to make it through far enough to take advantage of the underlying probabilities -- that is to take advantage of "managing the risk".
If the gambler cannot figure that out, or knows for a fact there are insufficient funds, do those probabilities really provide useful information on what to expect? To me this is then uncertainty. The gamble simply doesn't get the opportunity to repeat and so get the expected return.

I think what's really going on here in your explicit comment is:

  • differences in the size of the confidence intervals; the house can indeed be more confident about their understanding of the odds. But it's not an absolute difference; they can't be sure, and the gambler can know something.
    • So I think it's sort-of true to say "We might know what the probabilities are and the shape of the function but we don't really know how many times we need to play before our sampling starts to reflect that distribution -- statistics gives us some ideas but that also has a random element to it." But here the "don't really know", "some ideas", and "a random element" seem to me to be doing a lot of the work - this isn't absolutely different from the house's situation; in both cases, there can be a prior, there's some data, and there's some randomness and uncertainty. The house has a way better grounded prior, way more data, and way less uncertainty, but it's not a categorical difference, as far as I can see.
  • an extreme case of diminishing returns to money. Becoming totally broke is really bad, and once you get there you can't get back. So even if he does have really good reason to believe gambling has positive expected value in dollars, that doesn't mean it has positive expected utility. I think it's very common to conflate the two, and that this is what underlies a lot of faulty ideas (i.e., that we should be genuinely risk-averse, in terms of utility - it makes a lot of sense to avoid the colloquial sense of risk, and it makes sense to avoid many gambles with positive expected dollar value, but that all makes sense if we're risk-neutral in terms of utility).

So it's very easy to reach the reasonable-seeming conclusion that gambling is unwise even if there's positive expected value in dollar terms, without leaning on the idea of a risk-uncertainty distinction (to be honest, even in terms of degrees - we don't even need to talk about the sizes of the confidence intervals, in this case).

I also think there's perhaps two more things implicitly going on in that analogy, which aren't key points but might slightly nudge one's intuitions:

  • We have a great deal of data, and very strong theoretical reasons, pointing towards the idea that, in reality, gambling has negative expected value in dollar terms. This could mean that, if Framing A seems to suggest one shouldn't gamble, and Framing B suggests one should, Framing A scores points with our intuitions. And this could occur even if we stipulate that there's positive expected value, because system 1 may not get that memo. (But this is a very small point, and I do think it's acceptable to use analogies that break our intuitions a bit, I just think it should be acknowledged.)
  • We also probably have a strong prior that gamblers are very often overconfident. It seems likely to me that, if you have a bias towards overconfidence, then the less grounding you have for your probabilities, the more likely they are to be wrong. That is, it's not just that the value you happen to receive could be further from what you expect, in either direction (compared to if you had a better-grounded probability), but that your perceived expected value is probably off the reasonable expected value by more, because your beliefs had more "room to manoeuvre" and were biased to manoeuvre i none direction in particular. So the less trustworthy our probability, the more likely it is we shouldn't gamble, as long as we're biased towards overconfidence, but we can discuss this in terms of degrees rather than a categorical distinction.
In this type of situation, perhaps rather than trying to calculate all the odds, wagers and pay-offs maybe a simple rule is better if someone wants to gamble.

I think that's likely true, but I think that's largely because of a mixture of the difficulty of computing the odds for humans (it's just time consuming and we're likely to make mistakes), the likelihood that the gambler will be overconfident so he should probably instead adopt a blanket heuristic to protect him from himself, and the fact that being broke is way worse than being rich is good. (Also, in realistic settings, because the odds are bad anyway - they pretty much have to be, for the casino to keep the lights on - so there's no point calculating; we already know which side of the decision-relevant threshold the answer must be on.) I don't think there's any need to invoke the risk-uncertainty distinction.

And finally, regarding the ideas of iterating and repeating - I think that's really important, in the sense that it gives us a lot more, very relevant data, and shifts our estimates towards the truth and reduces their uncertainty. But I think on a fundamental level, it's just evidence, like any other evidence. Roughly speaking, we always start with a prior, and then update it as we see evidence. So I don't think there's an absolute difference between "having an initial guess about the odds and then updating based on 100 rounds of gambling" and "having an initial guess about the odds and then updating based on realising that the casino has paid for this massive building, all these staff, etc., and seem unlikely to make enough money for that from drinks and food alone". (Consider also that you're never iterating or repeating exactly the same situation.)

Most of what I've said here leaves open the possibility that the risk-uncertainty distinction - perhaps even imagined as categorical - is a useful concept in practice (though I tentatively argue against that here). But it seems to me that I still haven't encountered an argument that it actually makes sense as a categorical division.

Making decisions when both morally and empirically uncertain

Basically just this one comment sent me down a (productive) rabbit hole of looking into the idea of the risk-uncertainty distinction, resulting in:

  • A post in which I argue that an absolute risk-uncertainty distinction doesn't really make sense and leads to bad decision-making procedures
  • A post in which I argue that acting as if there's a risk-uncertainty distinction, or using it as a rule of thumb, probably isn't useful either, compared to just speaking in terms along the lines of "more" or "less" trustworthy/well-grounded/whatever probabilities. (I.e., I argue that it's not useful to use terms that imply a categorical distinction, rather than a continuous, gradual shift.)

Given that you're someone who uses the risk-uncertainty distinction, and thus presumably either thinks that it makes sense in absolute terms or is at least useful as rough categorisation, I'd be interested in your thoughts on those posts. (One motivation for writing them was to lay out what seems to me the best arguments, and then see if someone could poke holes in them and thus improve my thinking.)

Risk and uncertainty: A false dichotomy?

Update: I've now posted that "next post" I was referring to (which gets into whether the risk-uncertainty distinction is a useful concept, in practice).

Risk and uncertainty: A false dichotomy?

I think I agree with substantial parts of both the spirit and specifics of what you say. And your comments have definitely furthered my thinking, and it's quite possible I'd now write this quite differently, were I to do it again. But I also think you're perhaps underestimating the extent to which risk vs uncertainty very often is treated as an absolute dichotomy, with substantial consequences. I'll now attempt to lay out my thinking in response to your comments, but I should note that my goal isn't really to convince you of "my side", and I'd consider it a win to be convinced of why my thinking is wrong (because then I've learned something, and because that which can be destroyed by the truth should be, and all that).

For the most part, you seem to spend a lot of time trying to discover whether terms like unknown probability and known probability make sense. Yet, those are language artifacts which, like everything language, is merely a use of a clarification algorithm as means to communicate abstractions. Each class represents primarily its dominating modes, but becomes increasingly useless at the margins.

From memory, I think I agreed with basically everything in Eliezer's sequence A Human's Guide to Words. One core point from that seems to closely match what you're saying:

The initial clue only has to lead the user to the similarity cluster—the group of things that have many characteristics in common.  After that, the initial clue has served its purpose, and I can go on to convey the new information "humans are currently mortal", or whatever else I want to say about us featherless bipeds.
A dictionary is best thought of, not as a book of Aristotelian class definitions, but a book of hints for matching verbal labels to similarity clusters, or matching labels to properties that are useful in distinguishing similarity clusters.

And it's useful to have words to point to clusters in thingspace, because it'd be far too hard to try to describe, for example, a car on the level of fundamental physics. So instead we use labels and abstractions, and accept there'll be some fuzzy boundaries and edge cases (e.g., some things that are sort of like cars and sort of like trucks).

One difference worth noting between that example and the labels "risk" and "uncertainty" is that risk and uncertainty are like two different "ends" or "directions" of a single dimension in thingspace. (At least, I'd argue they are, and it's possible that that has to be based on a Bayesian interpretation of probability.) So here it seems to me it'd actually be very easy to dispense with having two different labels. Instead, we can just have one for the dimension as a whole (e.g., "trustworthy", "well-grounded", "resilient"; see here), and then use that in combination with "more", "less", "extremely", "hardly at all", etc., and we're done.

We can then very clearly communicate the part that's real (that reflects the territory) from when we tried to talk about "risk" and "uncertainty", without confusing ourselves into thinking that there's some sharp line somewhere, or that it's obvious a different strategy would be needed in "one case" than in "the other". This is in contrast to the situation with cars, where it'd be much less useful to say "more car-y" or "less car-y" - do we mean along the size dimension, as compared to trucks? On the size dimension, as compared to mice? On the "usefulness for travelling in" dimension? On the "man made vs natural" dimension? It seems to me that it's the high dimensionality of thingspace that means labels for clusters are especially useful and hard to dispense with - when we're talking about two "regions" or whatever of a single dimension, the usefulness of separate labels is less clear.

That said, there are clearly loads of examples of using two labels for different points along a single dimension. E.g., short and tall, heavy and light. This is an obvious and substantial counterpoint to what I've said above.

But it also brings me to really my more central point, which is that people who claim real implications from a risk-uncertainty distinction typically don't ever talk about "more risk-ish" or "more Knightian" situations, but rather just situations of "risk" or of "uncertainty". (One exception is here.) And they make that especially clear when they say things like that we "completely know" or "completely cannot know" the probabilities, or we have "zero" knowledge, or things like that. In contrast, with height, it's often useful to say "short" or "tall", and assume a shared reference frame that makes it clear roughly what we mean by that (e.g., it's different for buildings than for people), but we also very often say things like "more" or "less" tall, "shorter", etc., and we never say "This person has zero height" or "This building is completely tall", or the like, except perhaps to be purposefully silly.

So while I agree that words typically point to somewhat messy clusters in thingspace, I think there are a huge number of people who don't realise (or agree with) that, and who seem to truly believe there's a clear, sharp distinction between risk and uncertainty, and who draw substantial implications from that (e.g., that we need to use methods other than expected value reasoning, as discussed in this post, or that we should entirely ignore possibilities we can't "have" probabilities about, an idea which the quote from Bostrom & Cirkovic points out the huge potential dangers of).

So one part of what you say that I think I do disagree with, if I'm interpreting you correctly, is "These quotes and definitions are using imagery to teach readers the basic classification system (i.e. the words) to the reader, by proposing initial but vague boundaries." I really don't think most writers who endorse the risk-uncertainty distinction think that that's what they're doing; I think they think they're really pointing to two cleanly separable concepts. (And this seems reflected in their recommendations - they don't typically refer to things like gradually shifting our emphasis from expected value reasoning to alternative approaches, but rather using one approach when we "have" probabilities and another when we "don't have probabilities", for example.)

And a related point is that, even though words typically point to somewhat messy clusters in thingspace, some words can be quite misleading and do a poor job of marking out meaningful clusters. This is another point Eliezer makes:

Any way you look at it, drawing a boundary in thingspace is not a neutral act.  Maybe a more cleanly designed, more purely Bayesian AI could ponder an arbitrary class and not be influenced by it.  But you, a human, do not have that option.  Categories are not static things in the context of a human brain; as soon as you actually think of them, they exert force on your mind.  One more reason not to believe you can define a word any way you like.

A related way of framing this is that you could see the term "Knightian uncertainty" as sneaking in connotations that this is a situation where we have to do something other than regular expected value reasoning, or where using any explicit probabilities would be foolish and wrong. So ultimately I'm sort-of arguing that we should taboo the terms "risk" (used in this sense) and "Knightian uncertainty", and just speak in terms of how uncertain/resilient/trustworthy/whatever a given uncertainty is (or how wide the confidence intervals or error bars are, or whatever).

But what I've said could be seen as just indicating that the problem is that advocates of the risk-uncertainty distinction need to read the sequences - this is just one example of a broader problem, which has already been covered there. This seems similar to what you're saying with:

So you get caught in a rabbit hole as you are essentially re-discovering the limitations of language and classification systems, rather than actually discussing the problem at hand. And the initial problem statement, in your analysis (i.e. certainty as risk+uncertainties) is arbitrary, and your logic could have been applied to any definition or concept.

I think there's something to this, but I still see the risk-uncertainty distinction proposed in absolute terms, even on LessWrong and the EA Forum, so it seemed worth discussing it specifically. (Plus possibly the fact that this is a one-dimensional situation, so it seems less useful to have totally separate labels than it is in many other cases like with cars, trucks, tigers, etc.)

But perhaps even if it is worth discussing that specifically, I should've more clearly situated it in the terms established by that sequence - using some of those terms, perhaps changing my framing, adding some links. I think there's something to this as well, and I'd probably do that if I was to rewrite this.

And something I do find troubling is the possibility that the way I've discussed these problems leans problematically on terms like "absolute, binary distinction", which should really be tabooed and replaced by something more substantive. I think that the term "absolute, binary distinction" is sufficiently meaningful to be ok to be used here, but it's possible that it's just far, far more meaningful than the term "Knightian uncertainty", rather than "absolutely" more meaningful. As you can probably tell, this particular point is one I'm still a bit confused about, and will have to think about more.

And the last point I'll make relates to this:

Whether the idea of uncertainty+risk is the proper tool can essentially only analyzed empirically, by comparing it, for example, to another method used in a given field, and evaluating whether method A or B improve s the ability of planners to predict date/cost prediction (in software engineering, for example).

This is basically what my next post will do. It focuses on whether, in practice, the concept of a risk-uncertainty distinction is useful, whether or not it "truly reflects reality" or whatever. So I think that post, at least, will avoid the issues you perceive (at least partially correctly, in my view) in this one.

I'd be interested in your thoughts on these somewhat rambly thoughts of mine.

Can we always assign, and make sense of, subjective probabilities?

In practice, I try to understand the generator for the claim. I.e. the experience plus belief structures that lead to a claim like it to make sense to the person.

I think that makes sense, and it's sort of like what I was interested here is thinking about what the generator could actually be in cases that seem so unlike anything one has actually experienced or had direct evidence of, and, in the most extreme case, something that, by its very nature, would never leave any evidence of its truth or falsity.

Also knightian uncertainty seems relevant but I'm not sure how quantitatively speaking.

This post was sort-of a spin-off from another post on the idea of a distinction between risk and Knightian uncertainty, which I've now posted here. So it's indeed related. But I basically reject the risk-uncertainty distinction in that post (more specifically, I see there as being a continuum, rather than a binary, categorical distinction). So this post is sort-of like me trying to challenge a current, related belief of mine by trying to see how subjective probabilities could be arrived at, and made sense of, in a particularly challenging case. (And then seeing whether people can poke holes in my thinking.)

(I've now edited the post to make it clear that this came from and is related to my thinking on Knightian uncertainty.)

Can we always assign, and make sense of, subjective probabilities?

(See my other comments for what I meant by probability)

I don't know much about dark matter and energy, but I'd say they're relatively much less challenging cases. I take it that whether they exist or not should already affect the world in observable ways, and also that we don't have fundamental reasons to expect we could never get more "direct observations" of their existence? I could be wrong about that, but if that's right, then that's just something in the massive category of "Things that are very hard to get evidence about", rather than "Things that might, by their very nature, never provide any evidence of their existence or lack of existence." I'd say that's way closer to the AGI case than to the "a god that will literally never interact with the natural world in any way" case. So it seems pretty clear to me that it can be handled with something like regular methods.

My intention was to find a particularly challenging case for arriving at, and making sense of, subjective probabilities, so I wanted to build up to claims where whether they're true or not would never have any impact at all on the world. (And this just happens to end up involving things like religion and magic - it's not that I wanted to cover a hot button topic on purpose, or debate religion, but rather I wanted to debate how to arrive at and make sense of probabilities in challenging cases.)

Can we always assign, and make sense of, subjective probabilities?

Ah, these two comments, and that of G Gordon Worley III, have made me realise that I didn't at all make explicit that I was taking the Bayesian interpretation of probability as a starting assumption. See my reply to G Gordon Worley III for more on that, and the basic intention of this post (which I've now edited to make it clearer).

Load More