by Mark Xu1 min read10th Mar 202033 comments
Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.
New Comment
32 comments, sorted by Click to highlight new comments since: Today at 2:55 AM

My current taxonomy of rationalists is:

  • LW rationalists (HI!)
  • Facebook rationalists
  • Twitter rationalists
  • Blog rationalists
  • Internet-invisible rationalists

Are there other types of rationalists? Maybe like group-chat rationalists? or podcast rationalists? google doc rationalists?

Alternative taxonomy:

  • rationalists belonging to Eliezer
  • cryopreserved rationalists
  • rationalists trained by CFAR
  • aspiring rationalists
  • rationalists working for MIRI
  • legendary rationalists
  • metarationalists
  • those commenting on this taxonomy
  • those that tweet as if they were mad
  • Bayesians
  • et cetera
  • Zvi
  • those that from afar look like paperclips


This made me chuckle. More humor

  • Rationalists taxonomizing rationalists
  • Mesa-rationalists (the mesa-optimizers inside rationalists)
  • carrier pigeon rationalists
  • proto-rationalists
  • not-yet-born rationalists
  • literal rats
  • frequentists
  • group-house rationalists
  • EA forum rationalists
  • academic rationalists
  • meme rationalists


I like this one.

I think there is a community of discord rationalists and tumblr rationalists.

I am a google doc rationalist! (Or I would like to be. Google docs are great.)

What are you using the word "rationalist" to mean? If you just mean "members of any subculture bearing some line of memetic descent from early Less Wrong" (which I don't think deserves the pretentious term rationalist, but putting that aside), why is "communication platform" a useful way to chop up that social grouping? A lot of the same people use Less Wrong and Facebook and Twitter and Discord and Google Docs, and a lot of the people who use the same platform wouldn't be in the same cluster if you were to analyze the graph of what actual conversations people are using these platforms to have.

It's a natural way to cut it up from one's own experience. Each platform has different affordances and brings out different aspects of people, and I get pretty different experiences of them on the different platforms mentioned.

John Flanagan: “An ordinary archer practices until he gets it right. A ranger practices until he never gets it wrong.”

I want to reword this in to make it about rationality in a way that isn't pretentious.

Cavilo, The Vor Game: "The key to strategy... is not to choose a path to victory, but to choose so that all paths lead to a victory." is close to what I want, but not quite.

I'm not sure what to recommend, but a few words that come to mind that might be relevant and help you spark an idea:

  • resilient
  • robust
  • reliable
  • antifragile
  • high availability
  • self healing
  • overdetermined

Is the victory bit important in the quotation?

If it's not about the victory/winning, and rather about the path/journey ....

A first draft that springs to mind as I type:

The key to rationality.... is not to chose the label, but to choose to take every opportunity to improve/update your thinking.

(Can't ... stop ... myself ... from commenting: From what I've observed too much ego gets in the way of rational thinking sometimes.)

Why not something along the lines of "your rationality is measured as much by your worst case performance as your average case"?

Maybe "your worst case contributes to your average just as strongly as your best case"?

Epistemic status: rambles

Quarantine preparation has made me realize that a days worth of food is actually really cheap, doesn't require that much time to cook, and can be made fairly tasty for not much more, i.e. a day's worth of easy-to-cook, relatively tasty food is about $5.

This requires some amount of amortized costs for the easy-to-cook and relatively tasty part, but not immensely large upfront costs (instantpot, spices, etc.).

This reference says that 40 million people dealt with hunger is the US. I am... super confused? I find it extremely difficult to believe that people literally couldn't afford to buy food, so the explanation is probably something like "hunger is worth it to get dietary variety/convenience/etc." or "trapped by local incentive gradients" or "have incentives to not save money, go hungry when hit by proverbial 'rainy days'" or "people don't realize how cheap food actually is"

I'm still confused though. I feel like there might be some room for someone to write some infographic that has information like "here's a rotating course of 10 meals that are cheap and tasty, with lists of exactly what to buy on what day, how to cook everything with various types of kitchen equipment, substitutes in case the store doesn't have various ingredients, possible variants in case you get bored". Crucially, the infographic would have to be really good. A possible explanation is that people who potential might have to deal with hunger don't have the slack to plan their meals so they don't and none of the existing meal plans are understandable enough or something.

I notice I'm still confused.

Also mildly confused by why soup kitchens make complicated foods instead of simple foods, but that confusion is nearly entirely resolved by various signaling considerations.

Maybe people who struggle with hunger don't plan a rotating course of 10 meals because they are signaling that they aren't so poor to have to plan their meals so meticulously. Maybe planning a rotating course of 10 meals is much harder than I think it is. Maybe I'm far below where I think I am in terms of "ability to endure eating the same food over and over again" and most people just can't eat a rotating course of meals.

I notice I'm still confused.

I feel like I might be missing something really clear. Like something along the lines of "most people who go hungry don't have kitchens/space to store ingredients/stable living situations/any slack at all whatsoever/something".

It seems to me that during the quarantine I eat less than usual; either I am deluding myself, or it is a combination of having less physical activity (such as walking to/from job/lunch/shops/playground), being able to eat whenever I want (so there is no pressure to "eat a lot now, because the next opportunity will be 7 hours later"), making less superstimulating food (using less sugar and salt), and having other ways to get some hedons (e.g. taking a nap). Sometimes I cook a soup, and that's most of my daily food.

And soups are really cheap. You take $1-2 worth of ingredients, cook them in water, add little salt and spices; optionally eat with bread. Bread is cheap, salt is cheap, most spices are cheap (per portion), potatoes, broccoli, carrot, onion, and beans are cheap. Most of these things are like $1 per 1kg.

Okay, soups are not super healthy; cooking destroys vitamins. You should also get some fresh fruits and vegetables. Apples, tomatoes, cucumbers are $1-2 per 1 kg. You should definitely be able to eat healthy food for less than $5 a day.

What is expensive? Chocolate and other sweets, cheese, berries, nuts; I probably forgot something. You shouldn't eat sweets; and you can afford the other things now and then even under $5 a day on average. (It is not an optimal diet; some people recommend eating berries and nuts every day. But still healthier than many people eat, including those who spend more money on food.)


On the other hand, we usually spend more than $5 per person per day, even during the quarantine. We spend a lot on sweets and cheese. The tastier ones are even more expensive than the basic ones, which are already more expensive than the actually useful food. Instant gratification -- it's addictive! The more stress I have during the day, the more I need something that will improve my mood instantly, even if it's only for a moment.

Poor people probably have more stress, and thus less willpower to resists things full of sugar and salt. (Also alcohol and tobacco. Okay, the last one is technically not food, but still comes from the same budget.)

Very poor people, e.g. the homeless, don't have the place to cook. So many cheapest things are ironically out of their reach. Not having a fridge is also a problem.

Then, I assume many poor people don't have the good skills and habits. Some of them don't have the necessary IQ, some have mental problems, some had shitty upbringing.

There are a few things to keep in mind:

1) The claim that 40 million Americans "deal with hunger" is, um, questionable. Their citation leads to, which cites USDA's Household Food Security in the United States report ( The methodology used is an 11-question survey (18 for households with children), where answering 3 questions in the affirmative marks you as low food security. The questions asked are (naturally) subjective. Even better, the first question is this: “We worried whether our food would run out before we got money to buy more.” Was that often, sometimes, or never true for you in the last 12 months? That's an a real concern to have, but it is not what people are talking about when they say "dealing with hunger". You can be running on a shoestring budget and often worry about whether you'll have enough money for food without ever actually not having enough money for food.

2) A significant percentage of the population has non-trivial issues with executive function. Also, most of the population isn't familiar with "best practices" (in terms of effective life strategies, basic finances, etc). Most people simply don't think about things like this systematically, which is how you get the phenomenon of ~50% of the population not being able to cover a $400 emergency (or whatever those numbers are, they're pretty close). This would be less of an issue if those cultural norms were inherited, but you can't teach something you don't know, and apparently we don't teach Home Economics anymore (not that it'd be sufficient, but it would be better than nothing). This is a subject that deserves a much more in-depth treatment, but I think as a high-level claim this is both close enough to true and sufficient as a cause for what we might observe here. Making an infographic with a rotating course of 10 cheap, easy-to-prepare, relatively healthy, and relatively tasty meals is a great idea, but it'll only be useful to the sorts of people who already know what "meal prep" means. You might catch some stragglers on the margin, but not a lot.

3) The upfront costs are less trivial than they appear if you don't inherit any of the larger items, and remember, 50% of the population can't cover a mid-3-figure emergency. "Basic kitchen equipment" can be had for under $100, but "basic kitchen equipment" doesn't necessarily set you up to prepare food in a "meal prep" kind of way.

2) is something that I sort of thought about but not with as much nuance. I agree that such an infographic would be only useful for people who were looking for an alternate meal preparation strategy or something.

3) if it's true that people want to do meal preppy type things but don't have enough to pay upfront costs, there might be gains from 0-interest microloans, maybe via some MLM-type I loan you money, then once you've saved some money and paid me back, you loan other people money too.

It seems like the bottom 20% of the US spends $2216 per year per income earner or ~$6 per day on food. Given that children themselves don't have an income they might spend less then $5 per person for food per day.

People can drown in a river that's on average 1m deep.

If you have DAI right now, minting on and swapping yTrump for nTrump on is an almost guaranteed 15% profit.

Lesswrong posts that I want someone to write:

  1. Description of pair debugging
  2. Description of value handshakes

Maybe I'll think of more later.

I found a reference to "value handshakes" here:

Your AI doesn't figure out how to do a reasonable "values handshake" with a competitor (where two agents agree to both pursue some appropriate compromise values in order to be Pareto efficient)...

I think it refers to something like this: Imagine that a superintelligent human-friendly AI meets a superintelligent paperclip maximizer, and they both realize their powers are approximately balanced. What should they do?

For humans, "let's fight, and to the victor go the spoils" is the intuitive answer, but the superintelligences can possibly do better. If they fight, they have a 50% chance of achieving nothing, and a 50% chance of winning the universe... minus whatever was sacrificed to Moloch, which could possibly be a lot. If they split the universe to halves, and find out a way how to trust each other, that is better that war. But there is a possibility of even better solution, when both of them would agree on acting as if they were a single superintelligence that values both humans and paperclips equally.

The cooperative solution can be better than a 50% split of the universe, because you could build paperclip factories on places humans care less about, such as uninhabitable planets; or perhaps you could find a way how to introduce paperclips to human environment without reducing the human quality of life. For example, would you mind using paperclips to reinforce the walls of your house? Would you mind if almost all materials used to build stuff for humans contained little paperclips inside? Would you mind living in a simulation implemented on paperclip-shaped circuits? So maybe at the end, humans could get like 70% of the potential utility of the universe, while 70% of potential material would be converted to paperclips.

A weird but not-inaccurate way to think of log(n) is as an answer to "how many digits does n have?"

This suggests that a weird but not-inaccurate way to think of a log-normal distribution is as a distribution where "the number of digits is normally distributed"

There are a bunch of explanations of logarithm as length on Arbital.

Ignorance as evidence

I was answering a bunch of questions from OpenPhill's calibration test of the form "when did <thing> happen?". A lot of the time, I had no knowledge of <thing>, so I gave a fairly large confidence interval as a "maximum ignorance" type prediction (1900-2015, for example).

However, the fact that I have no knowledge of <thing> is actually moderate evidence that it happened "before my time".

Example: "when did <person> die?" If I was alive when <person> died, there's a higher chance of me hearing about their death. Thus not having heard of <person> is evidence that they died some time ago.

Example: "when did <person> die?" If I was alive when <person> died, there's a higher chance of me hearing about their death. Thus not having heard of <person> is evidence that they died some time ago.

Technically, it could also be evidence that you are dead, but your ghost cannot move to afterlife, probably because it is too attached to scoring internet points (a fate that awaits many of us, I am afraid).

(epistemic status: just kidding)

If you're interviewing employees of a company about how good the company is, there's positive bias because people who hate the company will have already left.

Sure. Also, current employees are dis-incented from being truthful about the bad parts. But you're not applying statistics to the results, so that's not terribly important. Such interviews provide limited evidence about goodness of the company. They provide decent evidence about the potential coworkers you're interviewing.

Generally, when you're interviewing employees about a company about whether the company is any good, you're trying to decide whether to work there yourself. And you're evaluating whether any of them seem competent and interesting enough that you can tolerate being near them for any length of time.

Coinfection rates of COVID and normal flu are very low. If you have the set of flu/COVID symptoms, you're basically guaranteed to have one or the other. You can test for the flu pretty easily. Therefore, people can just test for the flu as a proxy for testing for COVID.

Is this just a really obvious chain of reasoning that everyone has missed? Which one of my assumptions is wrong? says coinfection rates are low means we can test for the flu fast

Thus it's either the case that if you have the set of flu/COVID symptoms, you're basically guaranteed to have either flu or COVID.

Maybe the tests are only useful for people who don't have symptoms, but if that's not the case, then the flu test provides a lot of evidence as to whether or not someone has COVID (even if "basically guaranteed" is replaced with "probable").

update the CDC advises testing for the flu and there's a lot of medical things that cause "flu-like" symptoms. Turns out that "flu-like" symptoms is basically "immune system doing things", which is going to happen with most things your body doesn't like.

Not everybody who has a cold has influenza (the flu) or COVID-19. There are many different viruses that cause influenza-like-illnesses.

Low is relativ. 2% of coindience is still a bunch.

Moral uncertainty is a thing that people think about. Do people also think about decision theoretic uncertainty? E.g. how to decide when you're uncertain about which decision theory is correct?

Decision theoretic uncertainty seems easier to deal with, because you’re trying to maximise expected value in each case, just changing what you condition on in calculating the expectation. So you can just take the overall expectation given your different credences in the decision theories