I think for most of the examples given, the headline concept is not the main reason people might not immediately agree. for example:
some people prefer to take big effective actions, rather than tiny actions that add up across many people
Like the OP says, I have heard people say that not eating meat has no effect because production decisions are made in large chunks. That isn't about a preference for big actions.
I mean, it's probably true in the same way that "my vote will probably not change the election" is probably true. most people value a 1/1000 chance of saving 1000 chickens at way less than a 100% chance of saving one chicken. for meat production in particular, the connection between you choosing not to eat meat and less meat being produced is more tenuous than for elections. imo the main reasonable arguments for not eating meat are either deontological/signalling (it is bad to do immoral things, it is bad to be a hypocrite, etc), or fdt style arguments (I am not deciding just for myself, but for many people who use the same decision strategy); incidentally, these are imo also the main reasons to vote.
I very much know how continuous functions work and precisely what differentiability is, and I have filed taxes, but I would probably need a refresher on tax brackets to be sure I had everything right...
I have a similar comment: it seems to me the problem is not that people don’t have these fairly obvious concepts, it’s the unobvious application of these concepts to real world cases. Eg the three examples given for the Intermediate Value Theorem are not entirely obvious instances of it, and it’s not obvious that the IVT completely solves them.
I think some people don't have those concepts, and some people know those concepts at a shallow level but haven't integrated them into their worldviews. And for people who have them deeply, they assume these concepts are so obvious that it can't possibly be the reason other people are confused (instead they must have a deeper disagreement, or are trolling). Whereas I tend to believe that the world is full of more trivial mistakes.
But I don't have strong evidence that I'm right here. The nature of unknown knowns also means that, to the degree I'm right, it's harder to discuss than usual.
One indirect piece of evidence is an anecdote recounted by Thomas Shelling in the preface to the 1980 edition of The Strategy of Conflict. The anecdote suggests that we may overestimate people’s familiarity with seemingly obvious concepts.
The book has had a good reception, and many have cheered me by telling me they liked it or learned from it. But the response that warms me most after twenty years is the late John Strachey’s. John Strachey, whose books I had read in college, had been an outstanding Marxist economist in the 1930s. After the war he had been defense minister in Britain’s Labor Government. Some of us at Harvard’s Center for International Affairs invited him to visit because he was writing a book on disarmament and arms control. When he called on me he exclaimed how much this book had done for his thinking, and as he talked with enthusiasm I tried to guess which of my sophisticated ideas in which chapters had made so much difference to him. It turned out it wasn’t any particular idea in any particular chapter. Until he read this book, he had simply not comprehended that an inherently non-zero-sum conflict could exist. He had known that conflict could coexist with common interest but had thought, or taken for granted, that they were essentially separable, not aspects of an integral structure. A scholar concerned with monopoly capitalism and class struggle, nuclear strategy and alliance politics, working late in his career on arms control and peacemaking, had tumbled, in reading my book, to an idea so rudimentary that I hadn’t even known it wasn’t obvious.
Thanks, this is helpful.
Schelling's specific point actually feels relevant to me and a blindspot among (at least some) rationalists or EAs when they talk about "conflict" vs "mistake" theory. I've recently thought about the "conflict vs mistake theory" framing some more, and think it misses out a lot of the learnings that are standard in, eg, negotiation classes or bargaining theory, or international relations/game theory writ large.
I think a lot of the time a better position is something roughly like: "I have my interests and intend to pursue mine own interests to the best of my ability. I respect you as an agent with your interests and willing to pursue yours. Sometimes our interests come into conflict, and we take actions detrimental to each other. However, it is implausible that our interests are directly opposed, and there are often plausible gains from trade."
A plausible example of mistake theory inhibiting gains from trade is when (supposedly) Obama often tried to lecture Republican lawmakers about their mistakes, instead of taking their interests as a given and tried to negotiate more.
Of course, conflict theory can inhibit gains from trade if it prevents people from coming to the negotiation table, or just not notice that bargaining is almost always a better option than war.
Broadly agree - I overstated my point; of course some people don’t have these concepts. But I think there is a big gap between having these concepts as theory (eg IVT in pure math) and applying them in practice to less obvious cases.
(Cf Wittgenstein thought that understanding a concept just was knowing how to apply it - you don’t fully understand it until you know how to use it.)
I'm not wedded to the tax example. For the other(s), I disagree, but unfortunately neither of us have enough data to resolve this.
If a continuous function goes from value A to value B, it must pass through every value in between. In other words, tipping points must necessarily exist.
I propose more specific idea: if you are uniformly uncertain about fractional part of , then .
E.g., if you hurry on the way to the subway station without knowing when the next train arrives and got there 10 seconds earlier than if you didn't hurry, you win exactly the same 10 seconds in expectation.
Yeah, after getting enough people tripped up/upset at me about invoking IVT-like intuitions for discontinuous functions[1], I suspect something like the above is the subtler point I should've led with. Elsewhere I wrote I think part of the argument is that if you have a complicated distribution between a bunch of unknown discontinuous functions "in reality", from your epistemic state, it would often essentially look continuous to you when you combine the probabilities together, and you should treat them as such.
I think your formalism is helpful/might aid in thinking more clearly, but I'm also worried people would jump at it if their uncertainty is slightly non-uniform (without noticing that changing the math a tiny bit only changes the endline result a tiny bit).
In a lot of situations you can still treat your situation as locally linear despite non-global uniformity (see my third point on differentiable functions being locally linear), but that argument is more about "negotiating price," my first (IVT-inspired) point was establishing that it's possible to have an effect at all.
Total agree with the train example being a clear elucidation. I've used it before in other contexts when trying to explain EV-style reasoning more directly.
Obviously IVT doesn't hold for all discontinuous functions. But IVT-style intuitions still hold up for reasons like the ones you illustrate, most of the time.
other than very narrow pathological cases.
I think the more common underlying issue here is that people are confused about the tax code. Tax codes are very confusing. Income/benefits cliffs do exist. People get confused about what is and isn't an income cliff based on what they heard from more or less equally tax-confused people.
In the UK, exceeding £100,000 in household income for individuals making use of "Free Childcare for Working Parents" is indeed one of these pathological cases. Benefits cliffs frequently make things nonlinear unfortunately!
That said I do recognize that OP's original point was not about benefits. Even setting benefits aside however, my understanding is that there are cases where you might not want to be pushed into a higher bracket (e.g. drawing down from a retirement pot of fixed size, where funds drawn from it are considered "income" only in the year(s) in which they are withdrawn: often better to draw down relatively evenly over n years rather than most of it in a single year, due to tax brackets). Most talk of "not wanting to be pushed into a higher tax bracket" that I have heard comes from people in this and similar situations.
I’m confused—your examples for IVT reference “lumpy” functions. I’m not exactly sure what that means, but it seems like you mean functions with discrete, sharp steps. Such a function would be discontinuous, and IVT only applies to continuous functions.
Thanks, I really like these concepts, especially Grice's maxims were new to me and they seem very useful. Your list also got me thinking and I feel like I also have some (obvious) concepts in mind which I often usefully apply but which may not be so well known:
The data-processing inequality is often useful, especially when thinking about automated tools, like LLMs. It states that for any fixed channel K, the mutual information between X and Z is always larger than that between Y and Z, if Y is the output of X processed by the channel K. E.g., if you tell an LLM simple "refine the following paragraph", and the goal of the paragraph is to transmit information from your brain towards a reader, then using the LLM with this prompt can only destroy information (because the information is processed by a fixed channel which does not know the contents of your mind). Important are also the cases where the data processing inequality does not directly apply, e.g. with a prompt like "refine the following paragraph, note that what I want to communicate is [more detailed description than the paragraph alone]".
2. and 3. are just particularly useful concepts from game theory. I see public good games everywhere (e.g., climate, taking on duties in a community, etc.) and actually think many situations are sufficiently well explained by a very simple modeling as a public good game. Evolutionarily stable equilibrium is a stronger concept than Nash equilibrium and useful to think about which equilibria actually occur in society. E.g., especially for large games with many players and mixed strategies, it's a useful concept to think about cultural or group norms, globalization, etc.
The last one is probably well known to everyone who seriously played online poker or did sports betting, but it applies more generally. Roughly speaking, if you get money into the pot while having an 80% chance at winning, don't focus on whether you lose or win the hand eventually. The feedback for your actions should be the assessed correctness of your actions, without including factors completely independent of your actions. So basically: De-noise as much as possible (without destroying information).
Edit: Just to clarify the details of the data-processing inequality because I noticed Wikipedia uses different notation: The (Markov) model is Z-X-Y in my description, and in the example Z is the reader, X the brain and Y the output of the LLM.
I’d heard of the data processing inequality but don’t remember every understanding it. Now I feel like I do. Great example.
Interesting post! I am a bit confused about the application of the local linearity principle to the tax bracket issue.
The content of "differentiable functions are locally linear" is something like: if you nudge an input by ε, the output changes by ~f'(x)·ε. If you double your nudge, you approximately double your effect (to first order). I enjoyed the small goods insurance and altruism examples, since local curvature is negligible at the relevant scale (for altruist utility, and relatively unimportant decisions).
But the tax bracket example doesn't really make sense to me. The feared bad outcome ("being pushed into a higher bracket makes me worse off") is compatible with local linearity (in the sense that the segments were linear), since just you could just have f'(x) < 0. What rules out the bad outcome is that marginal tax rates are positive and less than 100%, not anything about smoothness or local approximation.
Example 3: People worry about “being pushed into a higher bracket” as if earning one more dollar could make them worse off overall. But tax liability is a continuous (piecewise linear) function of income. No additional dollar in income can result in greater than one dollar of tax liability, other than very narrow pathological cases.
Note the ASPE estimates 0.7% of households without children, and 0.6% of households with children (in the US) have marginal tax rates exceeding 100%.
Are these pathological cases? I don't know, but its nice to use numbers.
Thank you so much! I do appreciate the numbers here. When I was writing the post I was primarily thinking about taxes rather than welfare but I agree that this is an important part of the story as well.
Related to theory of mind: also realize that future you is an agent with motivations and intelligence and capacity to make decisions likely at least as well as you today. I've made this mistake before, where I ended up very anxious about some decision because I had this very implicit model of kind of shooting a bullet into the future where I had to aim just right to ensure it would hit the target. Took me a while to even realize I was doing this, and I was sort of treating future me as an "NPC" in a sense.
When I think of someone not having the intuitive concept of Grice's maxims, I don't think of someone who doesn't follow them (because part of the point is that people usually do so, and assume others will do so, without ever having conceptualized them; also because someone who actually didn't, on any level, know them would be pretty impaired in everyday life, so people who don't follow them in arguments are more often selectively suspending common sense). I think of someone analyzing the meaning of what someone said, or what the right thing to say is, overly literally without recognizing implicature.
Thanks for the Wiki article, was helpful to read!
Yeah, it's hard to balance the examples well. The most common examples of being wrong about X are often not the most central/clean examples of being wrong about X. This was also an issue for me in the Theory of Mind examples (neurotypical adults have at least some ToM in the developmental psychology sense, some of the most common failures are more sophisticated failures like typical mind fallacy[1], but in a sense, neither are the most interesting examples to bring up).
For me, an interesting example of Grice's maxims not being fully integrated is this post, which argues that you need to understand postmodern philosophy to get why "Stating true facts sometimes makes people angrier rather than updating their beliefs," whereas in practice I think in many (not all) cases, the people "stating true facts" and "just asking questions" that predictably make people angry are failing to integrate Grice's maxims on a normative level, and/or have poor theory of mind on a descriptive level.
Obviously there's more than one way to surmount a mountain, and continental philosophy has other teachings and benefits as well, so I don't want to begrudge people too much for becoming better at strategic empathy and conversational pragmatics through continental philosophy rather than the tools I'm more familiar with. But it does feel like overkill to me, and unfortunately continental philosophy seems to shackle people with other commitments and attitudes.
The problem with "typical mind fallacy" as a prototypical example of a cognitive error is that in many cases it can also be written correctly as a typical mind heuristic.
I think the category of 'unknown knowns' is itself a great example of an unknown known.
Personal anecdote: I was first told about about the idea of 'unknown unknowns' when I was in elementary school. I completed the to-me-obvious 2x2 and asked the (adult) person I was talking to about unknown knowns. They were frankly confused by the concept, and didn't think such a thing could really matter and thought there might not be such a thing.
I agree! Based on the title, I also kind of expected the article to cover that, but I guess it did so rather implicitly. :)
Personally though, I always thought of unknown knowns more like ~latent knowledge, so things you know without being entirely aware of them, or things you never thought of but that you immediately get when thinking about them (e.g. once somebody raises the question, you know the answer, but you never thought about that answer before and hence it wasn't really part of your world model until then), or things you could figure out by piecing together other things you know easily, but you never tried.
Yes, me too, and giving the latent knowledge a shape and a name helps turn (part of) it into explicit knowledge.
I highly recommend you look them up
Hotlinks to good expositions would have been nice.
Actually I know you don't endorse this, but I think you're right. The wording is slightly unclear and a holdover from the old shortform.
Great! Because I actually did as you had suggested, searched them each on Wikipedia, went "yes, I do know that, and use it on occasion", posted my comment, then read on and found your excellent expositions of them and the sorts of errors that not knowing them causes, felt slightly foolish, and then retracted it. Which might have been me doing the reading.
These IVT examples also remind me of a discussion I once had, where I argued with a person about whether it could be true that "Britney Spears ending her career as a musician" might in some sense be worse for the world than when a single (not that well-known) person dies. To me, this was obviously plausible, as it depends on how many Spears fans would feel how devastated about these news. If it's, say, millions who'll experience some degree of grief and despair about this, then this would surely cause more suffering than one death. Which is, I guess, primarily a question of scope sensitivity, but there's also this aspect to it where you can think of a continuous scale of degrees of suffering/grief, on which both "a loved one dies" and "a celebrity I like end their career" exist. They may not seem immediately comparable, but they do once you find sufficiently many intermediate points on that scale. At which point they become comparable and it's a relevant consideration how many individuals experience these states of mind.
I wonder if the disagree votes here are trying to say that people disagree with my argument being sound, or rather disagree with this being in any way similar to the IVT example. With the exception of kbear, for who it appears to be the former.
Just in case this wasn't clear, by "comparable" I don't mean they're similar, but that you can put them on a scale where the difference seems quantitative rather than qualitative. Perhaps a possible sequence would be:
To me, all two consecutive pairs of this chain seem to be within something like an order of magnitude of badness (solely from the isolated perspective as me being the person experiencing this from the outside and being emotionally affected by it). If this is true, and this is transitive, then a celebrity ending their career should be (per individual affected) ~upper-bounded by being 10^-6 as intense as a loved one dying. So, if someone has millions of fans, this could be in the same ballpark of badness as someone dying (and I'd argue that for many of these steps in the chain, the badness ratio is much smaller than 10).
yes! i understood your meaning, and intended to respond to it! i see the argument you are making, but disagree with its conclusion: asking "which of these events is worse?" is a type error. they are incomparable.
should britney spears be forced to perform in order to save a life? should the trolleyman make a Toxic choice? <-- these questions are so contrived that they are not meaningfully answerable. the 'correct' response is "please stop placing me in hypothetical situations!"
referring to your ladder, what i mean is this. hearing the news of event number 1 may have a larger impact on me than the news of event number 5. however, this is not at all the same as making a choice between the two events. if presented with such a choice, one 'should' kobayashi maru. any other response is morally and ethically bankrupt. at the very least one 'should not' make the decision frivolously: "oh, yep. option 5. already considered it. give me something harder next time, huh?" disgusting.
(it may be possible to appeal to an exoself, who is able to sort between the potential news items, but this brings in rather more metaphysics than i am comfortable making claims about. specifically, it's not clear to me how to reason about the interaction between the self's and exoself's desiderata.)
to resolve your slope, it is 'obvious' to me that items 1-5 are equal, while 6 and 7 are in different categories. i myself may have specific revealed preferences between 1-5 (though, see above), but my selfishness is not an ethical principle. nor are my emotional responses my barometer of what's good.
Merry Christmas! Today I turn an earlier LW shortform into a full post and discuss "unknown knowns" "obvious" ideas that are actually hard to discuss because they're invisible when you don't have them, and then almost impossible to unsee when you do.
Hopefully this is a fun article for like the twenty people who check LW on Christmas!
__
There are a number of implicit concepts I have in my head that seem so obvious that I don’t even bother verbalizing them. At least, until it’s brought to my attention other people don’t share these concepts.
It didn’t feel like a big revelation at the time I learned the concept, just a formalization of something that’s extremely obvious. And yet other people don’t have those intuitions, so perhaps this is pretty non-obvious in reality.
Here’s a short, non-exhaustive list:
If you have not heard any of these ideas before, I highly recommend you read up on the relevant sections below! Most *likely*, they will seem obvious to you. You might already know those concepts by a different name, or they’re already integrated enough into your worldview without a definitive name.
However, many people appear to lack some of these concepts, and it’s possible you’re one of them.
As a test: for every idea in the above list, can you think of a nontrivial real example of a dispute where one or both parties in an intellectual disagreement likely failed to model this concept? If not, you might be missing something about each idea!
Photo by Roberto Nickson on Unsplash
The Intermediate Value Theorem
Concept: If a continuous function goes from value A to value B, it must pass through every value in between. In other words, tipping points must necessarily exist.
This seems almost trivially easy, and yet people get tripped up often:
Example 1: Sometimes people say “deciding to eat meat or not won’t affect how many animals die from factory farming, since grocery stores buy meat in bulk.”
Example 2: Donations below a certain amount won’t do anything since planning a shipment of antimalarial nets, or hiring a new AI Safety researcher, is lumpy.
Example 3: Sometimes people say that a single vote can’t ever affect the outcome of an election, because “there will be recounts.” I think stuff like that (and near variants) aren’t really things people can say if they fully understand IVT on an intuitive level.
The core mistake? People understand there’s some margin where you’re in one state (eg, grocery store buys 2000 pounds of chicken) and some margin where you’re in another state (eg, grocery store buys 3000 pounds of chicken). But without the IVT, people don’t realize there must be a specific decision someone makes that tips the situation from the first state to the second state.
Note that this mistake (IVT-blindness) is recursive. For example, sometimes people understand the reasoning for why individual decisions might matter for grocery store orders but then don’t generalize, and say that large factory farms don’t make decisions on how many animals to farm based on orders from a single grocery store.
Interestingly, even famous intellectuals make the mistake around IVT. I’ve heard variants of all three claims above said by public intellectuals.[1]
Net Present Value
Concept: The value today of a stream of future payments, discounted by how far away they are. Concretely, money far enough in the future shrinks to nearly nothing in present value, so even infinite streams have finite present value[2].
Example 1: Sometimes people are just completely lost about how to value a one-time gain vs benefits that accumulate or compound over time. They think the problem is conceptually impossible (“you can’t compare a stock against a flow”).
Example 2: Sometimes people say it’s impossible to fix a perpetual problem (e.g. SF homelessness, or world hunger) with a one-time lump sum donation. This is wrong: it might be difficult in practice, but it’s clearly not impossible.
Example 3: Sometimes people say that a perpetual payout stream will be much more expensive than a one-time buyout. But with realistic interest rates, the difference is only like 10-40x.
Note that in many of those cases there are better solutions than the “steady flow over time” solution. For example, it’d be cheaper to solve world hunger via agricultural and logistical technology improvements than the net present value of “feeding poor people forever.” But the possibility of the latter creates an upper bound for how expensive this can be if people are acting mostly rationally, and that upper bound happens to be way cheaper than current global GDP or wealth levels.
Differentiable functions are locally linear
Concept: Zoom in far enough on any smooth curve and it looks like a straight line.
Example 1: People might think “being risk averse” justifies buying warranties on small goods (negative expected value, but shields you from downside risks of breaking your phone or something). But this is not plausible for almost any realistic risk-averse utility function, which becomes clear once you realize that any differentiable utility function is locally linear.
Example 2: People often have the intuition that altruists should be more careful with their money and more risk-sensitive than selfish people, even though the opposite is true. Altruistic people care about global welfare, which is a large function, so zoomed in, almost any individual altruist’s donation budget is linearly good for the world at large.
Example 3: People worry about “being pushed into a higher bracket” as if earning one more dollar could make them worse off overall. But tax liability is a continuous (piecewise linear) function of income. No additional dollar in income can result in greater than one dollar of tax liability, other than very narrow pathological cases.
Understanding that differentiable utility functions are locally linear unifies a lot of considerations that might otherwise confuse people, for example, why one sometimes ought to buy insurance for health and life but almost never for small consumer products, why altruistic people should be more risk-seeking with their investments, why bankroll management is important for poker players, etc.
Grice’s maxims
Concepts: Grice actually has four maxims:
I think disputes where one or both sides don’t follow each of Grice’s maxims should be fairly self-explanatory.
Many forms of trolling break one or more of these maxims, but not all of them. For example, a gish gallop is breaking the maxim of informativity. Bringing up Hilary Clinton’s emails, or the last Trump escapade, in an otherwise non-political discussion is breaking the maxim of relevance. The bad forms of continental philosophy often break the maxim of manner, which is why many analogize their writings to trolling. And of course, many trolls lie, breaking the maxim of quality.
For a longer, and somewhat ironic, meditation on the importance of Grice’s maxims, consider reading my earlier post:
The Pig Hates It
Theory of Mind
Concept: ToM has many components, but the single most important idea is that other people are agents too. Everybody else has their own goals, their own model of how the world works, and their own constraints on what they can do.
Example 1: Sometimes people ascribe frankly implausible motivations to their enemies, like “Republicans just hate women”, “Gazans don’t care about their children,” “X group just wants to murder babies” etc.
Example 2: Sometimes people don’t even consider that their enemies (and allies, and neutral third parties) even have motivations at all. The Naval War College Historian Sarah Paine calls this “half-court tennis”: sometimes US government officials and generals think about war and peace in relation solely to US strategic objectives. They don’t even consider that other countries have their own political aims, and do not primarily define their own politics in relation to US objectives.
Example 3: Do you often feel like characters in a novel seem “flat?” Like they’re characters who think they should be characters in a novel to advance a narrative point, not fully-fleshed out people with their hopes and dreams.
The core idea is very simple: treat other agents as real. It sounds banal, until you realize how rare it can be, and how frequently people mess up.
I think a full treatise on a theory of mind failures and strengths is worthy of its own blog post, and that’s what I’m working on next! Subscribe if you’re interested! :)
Why this all matters
Well, first of all, I think all of the concepts above are important, and neat, and it’d be good if more of my readers know about them!
More importantly, I think ideas matter. I deeply believe that ideas are extremely important and behind much of civilizational progress (and backsliding).
This is one of the central themes of this blog: ideas matter, and if we try harder and work smarter, if we approach every problem with simultaneous dedication and curiosity, together we can learn more ideas, integrate them into our worldviews, and use those ideas to improve our lives, and the world.
I don’t just mean big, all-encompassing, ideological frameworks, like Enlightenment or Communism. I also don’t just mean huge scientific revolutions, like evolution or relativity.
I mean small ideas, simple concepts like the ones above, that help us think better thoughts and live better lives.
I’m interested in a category I think of as Unknown Knowns: concepts that, once acquired, feel less like models you learned and more like obvious features of reality. They’re invisible until you have them, and then, once acquired, almost impossible to unsee. So you never truly notice them.
Today, almost 2000 years after some Jewish dude was nailed to a tree for championing the idea of how great it would be to be nice to people for a change, I want to actually see these ideas again. I want to take some time to appreciate all the ideas that have made my reality better, and all the people who made sacrifices, great and small, to find and propagate those ideas.
Merry Christmas.
Photo by Shalom Ejiofor on Unsplash
More subtly, Derek Parfit is arguably the single most original ethicist in the second half of the 20th century. Yet, his discussion of “imperceptible torture” in Reasons and Persons is probably not compatible with the Intermediate Value Theorem.
The actual math here has to do with summations of geometric series, which is not worth getting into here but is fairly intuitive for those who want to study up.