Announcing new reacts!

The reacts palette has 17 new reacts:

Gone are 11 old reacts who were getting little use, are redundant with the new reacts, and/or in my opinion weren't being used well (for example, "I'd bet this is false" almost never turned into a bet as I'd hoped, but was primarily used to disagree with added emphasis).

Reacts are a rich way to interact with a body of text, allowing many people to give information about comments, claims, and subclaims more efficiently than writing full comments, and they're something I love that we have on LessWrong (kudos to Raemon for building them initially!). They're also a way that people learn our culture – I will re-use Jimrandomh's classic meme on this topic.

I encourage you to look out for opportunities to use theses new reacts in the coming weeks, to give them a shot at life! We shall see which of these reacts get good use, and whether the use is good for truthseeking. I expect some of them will not, and will later be replaced by even newer reacts.

Feedback on the new reacts is invited, as are proposals for other reacts potentially worth adding.

P.S. There is also a stub of a wiki page on reacts, I would love it if people filled it o... (read more)

Experimental new reacts coming soon... (image visible on click)

I occasionally get texts from journalists asking to interview me about things around the aspiring rationalist scene. A few notes on my thinking and protocols for this:

• I generally think it is pro-social to share information with serious journalists on topics of clear public interest.
• By-default I speak with them only if their work seems relatively high-integrity. I like journalists whose writing is (a) factually accurate, (b) boring, and (c) do not feel to me to have an undercurrent of hatred for their subjects.
• By default I speak with them off-the-record, and then offer to send them write-ups of the things I said that they want to quote. This has gone quite well. I've felt comfortable speaking in my usual fashion without worrying about nailing each and every phrasing. Then I ask what they're interested in quoting, and I send them (typically a 1-2 page) google doc on those topics (largely re-stating what I already said to them, and making some improvements / additions). Then they tell me which quotes they want to use (typically cutting many sentences or paragraphs half-way). Then I make one or two slight edits and give them explicit permission to quote. I think this has gone quite wel

I have a general belief that internet epistemic hygiene norms should include that, when you quote someone, not only should you link to the source, but you should link to the highlight of that source. In general, if you highlight text on a webpage and right-click, you can "copy link to highlight" which when opened scrolls to and highlights that text. (Random example on Wikipedia.)

Further on this theme, archive.is has the interesting feature of constantly altering the URL to point to the currently highlighted bit of text, making this even easier. (Example, and you can highlight other bits of text to see it change.) Currently I overall don't like it because I constantly highlight text while I'm reading it, and so am v annoyed by the URL constantly changing, but it's plausible I'd get over this in time, and it'd be a good feature to add to LW.

The archive.is feature is also better because the normal "copy link to highlight" can often be unwieldily and long. Also I recall it sometimes not working, probably because the highlight is too short or too long (I don't quite understand the rules). On archive.is it just has a start and end number for where in the text is highlighted, making it al... (read more)

I want to contrast two perspectives on human epistemology I've been thinking about for over a year.

There's one school of thought about how to do reasoning about the future which is about naming a bunch of variables, putting probability distributions over them, multiplying them together, and doing bayesian updates when you get new evidence. This lets you assign probabilities, and also to lots of outcomes. "What probability do I assign that the S&P goes down, and the Ukraine/Russia war continues, and I find a new romantic partner?" I'll call this the "spreadsheets" model of epistemology.

There's another perspective I've been ruminating on which is about visualizing detailed and concrete worlds, in a similar way to if you hold a ball and ask me to visualize how it'll fall if you drop it, that I can see the world in full detail. This is more about loading a full hypothesis into your head, and into your anticipations. It's more related to Privileging the Hypothesis / Believing In / Radical Probabilism^[1]. I'll call this the "cognitive visualization" model of epistemology.

These visualizations hook much more directly to my anticipations and motivations. When I am running after you to r... (read more)

So many people have lived such grand lives. I have certainly lived a greater life than I expected, filled with adventures and curious people. But people will soon not live any lives at all. I believe that we will soon build intelligences more powerful than us who will disempower and kill us all. I will see no children of mine grow to adulthood. No people will walk through mountains and trees. No conscious mind will discover any new laws of physics. My mother will not write all of the novels she wants to write. The greatest films that will be made have probably been made. I have not often viscerally reflected on how much love and excitement I have for all the things I could do in the future, so I didn't viscerally feeling the loss. But now, when it is all lost, I start to think on it. And I just want to weep. I want to scream and smash things. Then I just want to sit quietly and watch the sun set, with people I love.

I Changed My Mind on Prison Sentencing.

I used to have the opinion that prison sentencing should be a disincentive proportional to the upside of the crime. The question I'd ask was "How much of a prison sentence would make the crime not worth it to people?" I tried to estimate the upside to the criminal, and what length of sentence would make the expected utility reliably turn out negative. I would sometimes discuss with people how long of a prison sentence they'd risk in order to get the upside of a crime, and use this as an input into how long I thought sentencing should be. However, I now think this was an error, for two reasons:

1. Many criminals or norm-breakers are not doing something as clear-headed as an explicit expected-value calculation, they are often thinking with part of their mind that is relatively uncivilized (and perhaps desperate or hidden from much of their conscious mind). For instance, it is done impulsively, or it is a crime of passion.
2. An Astral Codex Ten blogpost reiterated the common wisdom that length of sentencing has little effect on crime rate, and instead reliability of punishment has a much larger effect. The conclusion reads "Deterrence effects are so wea

Recently, I told a friend of mine that I'd been to a wedding. They asked how it was, and I said the couple clearly loved each other very much (as they made clear repeatedly in their speeches). My friend made a face like that I read as some kind of displeasure, a bit of a grimace. Since then, I've been wondering why that was.

I think it's a common occurrence, that people feel negatively about others openly expressing their love for something (a person, a piece of art, a place, etc). I'm pretty sure I've had this feeling myself, but I don't know why.

I can thi... (read more)

I've been re-reading tons of old posts for the review to remember them and see if they're worth nominating, and then writing a quick review if yes (and sometimes if no).

I've gotten my list of ~40 down to 3 long ones. If anyone wants to help out, here are some I'd appreciate someone re-reading and giving a quick review of in the next 2 days.

1. To Predict What Happens, Ask What Happens by Zvi
2. A case for AI alignment being difficult by Jessicata
3. Alexander and Yudkowsky on AGI goals by Scott Alexander & Eliezer Yudkowsky

I was chatting with someone, and they said that a particular group of people seemed increasingly like a cult. I thought that was an unhelpful framing, and here's the rough argument I wrote for why:

1. There's lots of group dynamics that lead a group of people to go insane and do unethical things.
2. The dynamics around Bankman-Fried involve a lot of naivety when interfacing with an sociopath who was scamming people for billions of dollars on a massive scale.
3. The dynamics around Leverage Research involved lots of people with extremely little savings and income in a

Two recent changes to LessWrong that I made!

1. I have added two new reacts: "I'd bet this is false" and "I'd bet this is true".
2. You can now sort a given user's comments by 'top' i.e. by karma.

For the first one, if you want to let someone know you'd be willing to take a bet (or if you want to call someone out on their bullshit) you can now highlight the claim they made and use the react. The react is a pair of dice, because we're never certain about propositions we're betting on (and because 'a hand offering money' was too hard to make out at the small scale). Hopefully this will increase people's affordance to take more bets on the site!

These reacts replaced "I checked it's true" and "I checked it's false", which didn't get that much use, but were some of the most abused reacts (often used on opinions or statements-of-positions that were simply not checkable).

For the second, if you go to a user profile and scroll down to the comments, you can now sort by 'top', 'newest', 'oldest', and 'recent replies'. I find that 'top' is a great way to get a sense of a person's thoughts and perspective on the world, and I used to visit greaterwrong a lot for this feature. Now you can do it on LessWrong!

Here's a quick list of 7 things people sometimes do instead of losing arguments, when it would be too personally costly to change their position.

1. They pick a variable in the argument that is hard for the debaters to get closer to, and state that their raw-intuition is simply different to the person they're talking to. ("I just believe that human ingenuity will overcome problems like this, and you don't, and this isn't something we're easily going to be able to resolve." or "I think we're just not going to be able to resolve whether this city-wide intervention works or not without a good RCT.")
2. They undermine your ability to have an opinion in the argument. ("I'm afraid there's very advanced research papers you'd need to read to have an understanding of this" or "You have been wrong so many times on so many issues, I don't think you're worth trusting on this one")
3. They performatively don't understand basic ideas, or weaponize confusion. ("I'm sorry, could you explain your theory of 'ownership', I don't really understand what you mean when you say you own your company" or "I'm sorry, I'm not sure what it means to 'provoke'? I was just saying what I thought.")
4. They get emotionally irate i

Privacy: a tool for thinking for yourself

As part of some recent experiments with debates, today I debated Ronny Fernandez on the topic of whether privacy is good or bad, and I was randomly assigned the “privacy is good” side. I’ve cut a few excerpts together that I think work as a standalone post, and put them below.

At the start I was defending privacy in general, and then we found that our main disagreement was about whether it was helpful for thinking for yourself, so I focus even more on that after the opening statement.

This is an experiment. I'm down for feedback on whether to do more of this sort of thing (it only takes me ~2 hours), how I could make it better for the reader, whether to make it a top-level post, etc.

Epistemic status: soldier mindset. I will here be exaggerating the degree to which I believe my conclusions.

Opening Statement

My core argument is that, in general, the pressures for conformity amongst humans are crazy.

This is true of your immediate circle, your local community, and globally. Each one of these has sufficiently strong pressures that I think it is a good heuristic to actively keep secrets and things you think about and facts about your life separate fr... (read more)

I believe that when people write 'tap out' in comment sections, they are actually supposed to write 'bow out', in almost all cases I've read it.

I regularly see comment sections where someone, to indicate it's going to be their last comment, writes that they're 'tapping out' at this point. They rarely mean they're conceding the point, I'm pretty sure they're just respectfully ending their participation in the conversation. But that's not the standard meaning of the phrase in the place that it comes from.

Here's ChatGPT explaining the two phrases (emphasis added).

The phrase "tap out" originates from the world of combat sports, particularly Brazilian Jiu-Jitsu and mixed martial arts (MMA). In these sports, a competitor signals submission or the desire to end a match by physically tapping their opponent, the mat, or even themselves. This act of tapping indicates that they can no longer continue the fight, either due to exhaustion, pain, or the risk of injury.

and

The phrase "bow out" means to gracefully or politely withdraw from a situation, activity, or role. It often implies a voluntary and dignified exit, often to avoid conflict or because it is the appropriate or respectful thing to do. The term can be used in various contexts, such as resigning from a job, ending participation in a project, or stepping down from a position of responsibility.

So I encourage any such people to change your usage accordingly!

Often I am annoyed when I ask someone (who I believe has more information than me) a question and they say "I don't know". I'm annoyed because I want them to give me some information. Such as:

"How long does it take to drive to the conference venue?" 

"I don't know." 

"But is it more like 10 minutes or more like 2 hours?" 

"Oh it's definitely longer than 2 hours."

But perhaps I am the one making a mistake. For instance, the question "How many countries are there?" can be answered "I'd say between 150 and 400" or it can be answered "195", and the former is called "an estimate" and the latter is called "knowing the answer". There is a folk distinction here and perhaps it is reasonable for people to want to preserve the distinction between "an estimate" and "knowing the answer".

So in the future, to get what I want, I should say "Please can you give me an estimate for how long it takes to drive to the conference venue?".

And personally I should strive, when people ask me a question to which I don't know the answer, to say "I don't know the answer, but I'd estimate between X and Y."

I often wish I had a better way to concisely communicate "X is a hypothesis I am tracking in my hypothesis space". I don't simply mean that X is logically possible, and I don't mean I assign even 1-10% probability to X, I just mean that as a bounded agent I can only track a handful of hypotheses and I am choosing to actively track this one.

• This comes up when a substantially different hypothesis is worth tracking but I've seen no evidence for it. There's a common sentence like "The plumber says it's fixed, though he might be wrong" where I don't want to communicate that I've got much reason to believe he might be wrong, and I'm not giving it even 10% or 20%, but I still think it's worth tracking, because strong evidence is common and the importance is high.
• This comes up in adversarial situations when it's possible that there's an adversarial process selecting on my observations. In such situations I want to say "I think it's worth tracking the hypothesis that the politician wants me to believe that this policy worked in order to pad their reputation, and I will put some effort into checking for evidence of that, but to be clear I haven't seen any positive evidence for that hypothesi

I don't normally just write-up takes, especially about current events, but here's something that I think is potentially crucially relevant to the dynamics involved in the recent actions of the OpenAI board, that I haven't seen anyone talk about:

The four members of the board who did the firing do not know each other very well.

Most boards meet a few times per year, for a couple of hours. Only Sutskever works at OpenAI. D'Angelo works senior roles in tech companies like Facebook and Quora, Toner is in EA/policy, and MacAulay at other tech companies (I'm not aware of any overlap with D'Angelo).

It's plausible to me that MacAulay and Toner have spent more than 50 hours in each others' company, but overall I'd probably be willing to bet at even odds that no other pair of them had spent more than 10 hours together before this crisis.

This is probably a key factor in why they haven't written more publicly about their decision. Decision-by-committee is famously terrible, and it's pretty likely to me that everyone pushes back hard on anything unilateral by the others in this high-tension scenario. So any writing representing them has to get consensus, and they're focused on firefighting and ge... (read more)

I've skimmed more than half of Anthropic's scaling policies doc. Key issues that stood out to me was the lack of incentive for any red-teamers to actually succeed at red-teaming. Perhaps I missed it, but I didn't see anything saying that the red-teamers had to necessarily not also have Anthropic equity. I also didn't see much other financial incentive for them to succeed. I would far prefer a world where Anthropic committed to put out a bounty of increasing magnitude (starting at like $50k, going up to like $2M) for external red-teamers (who signed NDAs) t... (read more)

"Slow takeoff" at this point is simply a misnomer.

Paul's position should be called "Fast Takeoff" and Eliezer's position should be called "Discontinuous Takeoff".

It is said that on this Earth there are two factions, and you must pick one.

1. The Knights Who Arrive at False Conclusions
2. The Knights Who Arrive at True Conclusions, Too Late to Be Useful

(Hat tip: I got these names 2 years ago from Robert Miles who had been playing with GPT-3.)

I think I've been implicitly coming to believe that (a) all people are feeling emotions all the time, but (b) people vary in how self-aware they are of these emotions.

Does anyone want to give me a counter-argument or counter-evidence to this claim?

I am still confused about moral mazes.

I understand that power-seekers can beat out people earnestly trying to do their jobs. In terms of the Gervais Principle, the sociopaths beat out the clueless.

What I don't understand is how the culture comes to reward corrupt and power-seeking behavior.

One reason someone said to me is that it's in the power-seekers interest to reward other power-seekers.

Is that true?

I think it's easier for them to beat out the earnest and gullible clueless people.

However, there's probably lots of ways that their sociopathic underlings ... (read more)

Striking paragraph by a recent ACX commenter (link):

I grew up surrounded by people who believed conspiracy theories, although none of those people were my parents. And I have to say that the fact that so few people know other people who believe conspiracy theories kind of bothers me. It's like their epistemic immune system has never really been at risk of infection. If your mind hasn't been very sick at least sometimes, how can you be sure you've developed decent priors this time?

Sometimes a false belief about a domain can be quite damaging, and a true belief can be quite valuable.

For example, suppose there is a 1000-person company. I tend to think that credit allocation for the success of the company is heavy tailed, and that there's typically 1-3 people who the company just would zombify and die without, and ~20 people who have the key context and understanding that the 1-3 people can work with to do new and live things. (I'm surely oversimplifying because I've not ever been on the inside with a 1000-person company.) In this situation it's very valuable to know who the people are who deserve the credit allocation. Getting the wrong 1-3 people is a bit of a disaster. This means that discussing it, raising hypotheses, bringing up bad arguments, bringing up arguments due to motivated cognition, and so on, can be unusually costly, and conversations about it can feel quite fraught. 

Other fraught topics include breaking up romantically, quitting your job, leaving a community club or movement. I think taboo tradeoffs have a related feeling, like bringing up whether to lie in a situation, whether to cheat in a situation, or when to exchange money for values ... (read more)

Er, Wikipedia has a page on misinformation about Covid, and the first example is Wuhan lab origin. Kinda shocked that Wikipedia is calling this misinformation. Seems like their authoritative sources are abusing their positions. I am scared that I'm going to stop trusting Wikipedia soon enough, which is leaving me feeling pretty shook.

Something I've thought about the existence of for years, but imagined was impossible: this 70s song by Italian Adriano Celentano. It fully registers to my mind as English. But it isn't. It's like skimming the output of GPT-2.