All of AnnaSalamon's Comments + Replies

Is AI Progress Impossible To Predict?

I notice you are noticing confusion.  Any chance either the data, or the code, has a bug?

Good to ask, but I'm not sure what it would be. The code is just a linear regression I did in a spreadsheet, and eyeballing the data points, it doesn't look like there are any patterns that a regression is missing. I tried it several different ways (comparing to different smaller models, comparing to averages of smaller models, excluding extreme values, etc.) and the correlation was always zero. Here's the raw data:

It's hard to know if there is some critical... (read more)

Salvage Epistemology

Can you say more about these for the benefit of folks like me who don't know about them?  What kind of "bad reception" or "controversial" was it?  Was it woo-flavored, or something else?

4TAG20d [] Everett tried to express his ideas as drily as possible, and it didn't entirely work--he was still accused of "theology" by Bohr. But there were and are technical issues as well, notably the basis problem. It can be argued that if you reify the whole formalism, then you have to reify the basis, and that squares the complexity of multiverse -- to every state in every basis. The argument actually was by JS Bell in Modern approaches tend to assume the multiverse has a single "preferred" basis, which has its own problems. Which tells us that it hasn't always been one exact theory.
Salvage Epistemology

I think "you should one-box on Newcomb's problem" is probably an example.  By the time it was as formalized as TDT it was probably not all that woo-y looking, but prior to that I think a lot of people had an intuition along the lines of "yes it would be tempting to one-box but that's woo thinking that has me thinking that."

Salvage Epistemology

I want to state more explicitly where I’m coming from, about LW and woo.

One might think: “LW is one of few places on the internet that specializes in having only scientific materialist thoughts, without the woo.”

My own take is more like: “LW is one of few places on the internet that specializes in trying to have principled, truth-tracking models and practices about epistemics, and on e.g. trying to track that our maps are not the territory, trying to ask what we’d expect to see differently if particular claims are true/not-true, trying to be a “lens that s... (read more)

Are people here mostly materialists? I'm not. In a Cartesian sense, the most authentic experience possible is that of consciousness itself, with matter being something our mind imagines to explain phenomenon that we think might be real outside of our imagination (but we can never really know). In other words, we know that idealism is true, because we experience pure ideas constantly, and we suspect that the images our minds serve up might actually correspond to some reality out there (Kant's things-in-themselves). The map might really be the territory. Like, if you read a book by Tolkein and find that the map doesn't match the text, which is right? And if Tolkein clarified, would he be right, considering the thing he's talking about doesn't even exist? Except it kinda does, in that we're debating real things, and they impact us, etc? I don't think we're anywhere near approaching a meaningful metaphysics, so the confidence of the materialists seems misplaced. I mean, yeah, I've seen matter, so I know it's real. But I've also seen matter in my dreams (including under a microscope, where it continued to be "real"). Sorry to rant on this single word!
Salvage Epistemology

One example, maybe: I think the early 20th century behaviorists mistakenly (to my mind) discarded the idea that e.g. mice are usefully modeled as having something like (beliefs, memories, desires, internal states), because they lumped this in with something like "woo."  (They applied this also to humans, at least sometimes.)

The article Cognition all the way down argues that a similar transition may be useful in biology, where e.g. embryogenesis may be more rapidly modeled if biologists become willing to discuss the "intent" of a given cellular signal ... (read more)

Salvage Epistemology

Thanks.  Are you up for saying more about what algorithm (you in hindsight notice/surmise) you were following internally during that time, and how it did/didn't differ from the algorithm you were following during your "hyper-analytical programmer" times?

Narrative Syncing

Can you say a bit more about why?

Do you agree that the social pressure in the pineapple and nose-picking examples isn't backchained from something like "don't spoil our game, we need everyone in this space to think/speak a certain way about this or our game will break"?

I think it's mostly from thinking about it in terms of means instead of ends. Like, my read of "narrative syncing" is that it is information transfer, but about 'social reality' instead of 'physical reality'; it's intersubjective instead of objective. There's also something going on here where I think most in-the-moment examples of this aren't backchained, and are instead something like 'people mimicking familiar patterns'? That can make it ambiguous whether or not a pattern is backchained, if it's sometimes done for explicitly backchainy reasons and is sometimes others copying that behavior. Suppose some people are allergic to peanuts, and so a space decides to not allow peanuts; I think the "[normatively]: peanuts aren't allowed here" is an example of narrative syncing. Is this backchained from "allowing peanuts will spoil our game"? Ehhh... maybe? Maybe the anti-peanut faction ended up wanting the norm more than the pro-peanut faction, and so it's backchained but not in the way you're pointing at. Maybe being a space that was anti-peanut ended up advantaging this space over adjacent competitive spaces, such that it is connected with "or our game will break" but it's indirect. Also, I predict a lot of this is the result of 'play' or 'cognitive immaturity', or something? Like, there might be high-stakes issues that you need to enforce conformity on or the game breaks, and low-stakes issues that you don't need to enforce conformity on, but which are useful for training how to do conformity-enforcement (and perhaps evade enemy conformity-enforcement). Or it may be that someone is not doing map-territory distinctions or subjective-objective distinctions when it comes to preferences; Alice saying "I like pineapple on pizza" is heard by Bob in a way that doesn't cash it out to "Alice prefers pizza with pineapple" but instead something like "pineapple on pizza is good" or "Bob prefers pizza with pineapple", both of which should be fought.
Narrative Syncing

For example, if you go to a go club and ask the players there how to get stronger at go, and you take their advice, you'll both get stronger and go and become more like the kind of person who hangs out in go clubs.  If you just want to be in sync with the go club narrative and don't care about the game, you'll still ask most of the same questions: the go players will have a hard time telling your real motivation, and it's not clear to me that they have an incentive to try.

This seems right to me about most go clubs, but there’re a lot of other places t... (read more)

Narrative Syncing

I haven't been able to construct any hypotheticals where I'd use it…. tl;dr I think narrative syncing is a natural category but I'm much less confident that “narrative syncing disguised as information sharing” is a problem worth noting,

I’m curious what you think of the examples in the long comment I just made (which was partly in response to this, but which I wrote as its own thing because I also wish I’d added it to the post in general).

I’m now thinking there’re really four concepts:

  1. Narrative syncing.  (Example: “the sand is lava.”)
  2. Narrative sy
... (read more)
Narrative Syncing

I agree with some commenters (e.g. Taran) that the one example I gave isn’t persuasive on its own, and that I can imagine different characters in Alec’s shoes who want and mean different things.  But IMO there is a thing like this that totally happens pretty often in various contexts.  I’m going to try to give more examples, and a description of why I think they are examples, to show why I think this.

Example: I think most startups have a “plan” for success, and a set of “beliefs” about how things are going to go, that the CEO “believes” basically... (read more)

If you are looking for more examples of narrative syncing:

  • "I have read, and accept, the terms and conditions [tick box]". I have not read the terms and conditions. They know I haven't. This is not an information exchange.
  • I was shopping with my Grandma once. I knew bananas were on the list and put them in the trolley. She asked "why didn't you take these bananas" and indicated a different brand. I thought she was asking for information so provided it, saying "they are smaller, cost more, and are wrapped up in plastic.". I got body language that indicated I
... (read more)
Potentially related examples: Group identity statements (pressure not to disagree) 1. A team believes themselves to be the best at what they do and that their training methods etc. are all the best/correct approach. If you suggest a new training method that seems to be yielding good results for other teams, they wouldn't treat it seriously because it's a threat to their identity. However, if the team also takes pride in their ability to continuously refine their training methods, they would be happy to discuss the new method. 2. If a group considers themselves to be "anti-pineapple" people, then saying "I like pineapples on my pizza" would signal that you're not really part of the group. Or maybe they think X is harmful and everyone knows pineapples contain X, then proudly declaring "I like pineapples on pizza" would mark you as an outsider. Self-fulfilling prophecies (coordination + presure not to disagree publicly) 1. It's the first week of school and the different student clubs and societies have set up booths to invite students to join. The president of club X tells you that they are the second largest club in the school. This makes club X seem like an established group and is one of the reasons you register your interest and eventually decide to join the group. Later on, you find out that club X actually had very few members initially. The president was basing his claim on the number of people who had registered their interest, not the actual members. However, since he managed to project the image of club X as a large and established group, many people join and it indeed becomes one of the largest student groups. 2. A captain tells the team before a game that they are going to win. The team is motivated and gives their best, therefore winning the game. (Some people may know the statement is false, which they may reveal to others in private conversations. They won't s
Huh, I would have thought that counted as narrative syncing as well?
I think there’s an alternative explanation for why social pressure may be used to make the dissenter capitulate. This is when an uninformed non-expert is dissenting from the informed judgments of an expert on their core subject. This is done for a few reasons. One is to use mild humiliation to motivate the non-expert to get past their pretentions and ego, do their research, and become more useful. Another is when it would be a bad use of time and resources to educate the non-expert on the spot. A third is when the non-expert is pointing out a real but surmountable challenge to the strategy, which is best dealt with at a later time. And a fourth is when the root of the issue is a values mismatch that is being framed as a disagreement about facts. This maybe is why I’m more inclined to view this as “leadership language” than as “narrative syncing.” Narrative syncing implies that such actions are undertaken mainly for empty propagandistic reasons, to keep the game going. I think that much of the time, the motivations of the people pressuring dissenters to capitulate are as I’m describing here. It’s just that sometimes, the leaders are wrong.
Increasing Demandingness in EA

Some components of my own models, here:

  1. I think most of the better-funded EA organizations would not prefer most LWers working there for $1M/yr, nor for a more typical salary, nor for free.

    (Even though many of these same LW-ers are useful many other places.)
  2. I think many of the better-funded EA organizations would prefer (being able to continue employing at least their most useful staff members) to (receiving an annual donation equal to 30x what that staff member could make in industry).
  3. If a typical LWer somehow really decided, deep in themselves
... (read more)
Increasing Demandingness in EA

I could still be missing something, but I think this doesn't make sense. If the marginal numbers are as you say and if EA organizations started paying everyone 40% of their counterfactual value, the sum of “EA financial capital” would go down, and so the counterfactual value-in-“EA”-dollars of marginal people would also go down, and so the numbers would probably work out with lower valuations per person in dollars. Similarly, if “supply and demand” works for finding good people to work at EA organizations (which it might? I’m honestly unsure), the number... (read more)

nit: I think "but losing its dollars" should be "but keeping its dollars"
3Thomas Kwa25d
Thanks, I agree with this comment.
Increasing Demandingness in EA

Great use of logic to try to force us to have models, and to make those models explicit!

Salvage Epistemology

I don't know; finding a better solution sounds great, but there aren't that many people who talk here, and many of us are fairly reflective and ornery, so if a small group keeps repeatedly requesting this and doing it it'd probably be sufficient to keep "aspiring rationalist" as at least a substantial minority of what's said.

FWIW, I would genuinely use the term 'aspiring rationalist' more if it struck me as more technically correct — in my head 'person aspiring to be rational' ≈ 'rationalist'. So I parse aspiring rationalist as 'person aspiring to be a person aspiring to be rational'.

'Aspiring rationalist' makes sense if I equate 'rationalist' with 'rational', but that's exactly the thing I don't want to do.

Maybe we just need a new word here. E.g., -esce is a root meaning "to become" (as in coalesce, acquiesce, evanesce, convalescent, iridescent, effervescent, quiescent). We c... (read more)

Increasing Demandingness in EA

because EA has much more human capital than financial capital

Is this a typo? It seems in direct contradiction with the OPs claim that EA is people-bottlenecked and not funding-bottlenecked, which I otherwise took you to be agreeing with.

5Thomas Kwa25d
I mean this in a narrow sense (edited to clarify) based on marginal valuations: I'd much rather delete 1% of EA money than 1% of EA human capital. So we can think of human capital as being worth more than money. I think there might be problems with this framing, but the core point applies: even though there are far fewer people than money (when using the conversion ratio implied by industry salary), the counterfactual value of people adds up to more than money. So paying everyone 40% of their counterfactual value would substantially deplete EA financial capital. I think this is equivalent to saying that the marginal trade we're making is much worse than the average trade (where trade = buying labor with money)
Salvage Epistemology

This is a bit off-topic with respect to the OP, but I really wish we’d more often say “aspiring rationalist” rather than “rationalist.” (Thanks to Said for doing this here.) The use of “rationalist” in parts of this comment thread and elsewhere grates on me. I expect most uses of either term are just people using the phrase other people use (which I have no real objection to), but it seems to me that when we say “aspiring rationalist” we at least sometimes remember that to form a map that is a better predictor of the territory requires aspiration, effor... (read more)

Maybe what you're actually looking for is something like "aspiring beisutsuka". Like there's an ideal you're aiming for but can maybe approach only asymptotically. Just don't equate "rationalist" with "beisutsuka" and you're good.

I dutifully tried to say "aspiring rationalist" for awhile, but in addition to the syllable count thing just being too much of a pain, it... feels like it's solving the wrong problem.

An argument that persuaded me to stop caring about it as much: communities of guitarists don't call themselves "aspiring guitarists". You're either doing guitaring, or you're not. (in some sense similar for being a scientist or researcher).

Meanwhile, I know at least some people definitely meet any reasonable bar for "actually a goddamn rationalist". If you intentionally reflec... (read more)

As I see it, "rationalist" already refers to a person who thinks rationality is particularly important, not necessarily a person who is rational, like how "libertarian" refers to a person who thinks freedom is particularly important, not necessarily a person who is free. Then literally speaking "aspiring rationalist" refers to a person who aspires to think rationality is particularly important, not to a person who aspires to be rational. Using "aspiring rationalist" to refer to people who aspire to attain rationality encourages people to misinterpret self-identified rationalists as claiming to have attained rationality. Saying something like "person who aspires to rationality" instead of "aspiring rationalist" is a little more awkward, but it respects the literal meaning of words, and I think that's important.

I agree that "aspiring rationalist" captures the desired meaning better than "rationalist", in most cases, but... I think language has some properties, studied and documented by linguists, which define a set of legal moves, and rationalist->aspiring rationalist is an invalid move. That is: everyone using "aspiring rationalist" is an unstable state from which people will spontaneously drop the word aspiring, and people in a mixed linguistic environemnt will consistently adopt the shorter one. Aspiring Rationalist just doesn't fit within the syllable-count budget, and if we want to displace the unmodified term Rationalist, we need a different solution.

Salvage Epistemology

To elaborate a bit where I'm coming from here: I think the original idea with LessWrong was basically to bypass the usual immune system against reasoning, to expect this to lead to some problems, and to look for principles such as "notice your confusion," "if you have a gut feeling against something, look into it and don't just override it," "expect things to usually add up to normality" that can help us survive losing that immune system. (Advantage of losing it: you can reason!)

My guess is that that (having principles in place of a reflexive or socially ... (read more)

I don't have a complete or principled model of what an epistemic immune system is or ought to be, in the area of woo, but I have some fragments.

One way of looking at it is that we look at a cluster of ideas, form an outside view of how much value and how much crazymaking there is inside it, and decide whether to engage. Part of the epistemic immune system is tracking the cost side of the corresponding cost/benefit. But this cost/benefit analysis doesn't generalize well between people; there's a big difference between a well-grounded well-studied practition... (read more)

Salvage Epistemology

Do you have a principled model of what an "epistemic immune system" is and why/whether we should have one?

To elaborate a bit where I'm coming from here: I think the original idea with LessWrong was basically to bypass the usual immune system against reasoning, to expect this to lead to some problems, and to look for principles such as "notice your confusion," "if you have a gut feeling against something, look into it and don't just override it," "expect things to usually add up to normality" that can help us survive losing that immune system. (Advantage of losing it: you can reason!)

My guess is that that (having principles in place of a reflexive or socially ... (read more)

What an actually pessimistic containment strategy looks like

It seems “taboo” to me. Like, when I go to think about this, I feel … inhibited in some not-very-verbal, not-very-explicit way. Kinda like how I feel if I imagine asking an inane question of a stranger without a socially sensible excuse, or when a clerk asked me why I was buying so many canned goods very early in Covid.

I think we are partly seeing the echoes of a social flinch here, somehow. It bears examining!

Open tolerance of the people involved with status quo and fear of alienating / making enemies of powerful groups is a core part of current EA culture! Steve's top comment on this post is an example of enforcing/reiterating this norm. 

It's an unwritten rule that seems very strongly enforced yet never really explicitly acknowledged, much less discussed. People were shadow blacklisted by CEA from the Covid documentary they funded for being too disrespectful in their speech re: how governments have handled covid.  That fits what I'd consider a taboo,... (read more)

I feel similarly.
I also find it somewhat taboo but not so much that I haven’t wondered about it.

FYI, I thought this sort of idea was an obvious one, and I've been continuously surprised that it didn't have more discussion. I don't feel inhibited and am sort of surprised you are.

(I do think there's a lot of ways to do this badly, with costs on the overall coordination-commons, so, maybe I feel somewhat inhibited from actually going off to do the thing. But I don't feel inhibited from brainstorming potential ways to address the costs and thinking about how to do it)

9Daniel Kokotajlo2mo
OK! Well, I can't speak for everyone's experiences, only my own. I don't think this subject should be taboo and I'm glad people are talking more about it now.
MIRI announces new "Death With Dignity" strategy

Also, MIRI isn't (necessarily) a hive mind, so not sure if Rob, Nate or Abram actually share the same estimate of how doomed we are as Eliezer.

Indeed, I expect that the views of at least some individuals working at MIRI vary considerably.

In some ways, the post would seem more accurate to me if it had the Onion-esque headline: Eliezer announces on MIRI’s behalf that “MIRI adopts new ‘Death with Dignity’ strategy.”

Still, I love the post a lot. Also, Eliezer has always been pivotal in MIRI.

7Rob Bensinger2mo
The five MIRI responses in my AI x-risk survey [] (marked with orange dots) show a lot of variation in P(doom): (Albeit it's still only five people; maybe a lot of MIRI optimists didn't reply, or maybe a lot of pessimists didn't, for some reason.)
MIRI announces new "Death With Dignity" strategy

The spread of opinions seems narrow compared to what I would expect. OP makes some bold predictions in his post. I see more debate over less controversial claims all of the time.

That's fair.

what do aliens have to do with AI?

Sorry, I said it badly/unclearly. What I meant was: most ways to design powerful AI will, on my best guess, be "alien" intelligences, in the sense that they are different from us (think differently, have different goals/values, etc.).

I just want to say I don't think that was unclear at all. It's fair to expect people to know the wider meaning of the word 'alien'.
MIRI announces new "Death With Dignity" strategy

I personally find it relatively gentle and reminding-toward-beauty, but YMMV. Lyrics.

Ah, this was in-person. ("Threads" was more/differently metaphorical than usual)
MIRI announces new "Death With Dignity" strategy

I've had in my head all day Leonard Cohen's song "You got me singing," which he wrote toward the end of his life, watching death approach.

4Ben Pace2mo
Well, I'm sure listening to that is going to be difficult. I'll listen to it sometime when I'm alone and have space to weep.
MIRI announces new "Death With Dignity" strategy

Why is everyone here in agreement that…

We’re not. There’s a spread of perspectives and opinions and lack-of-opinions. If you’re judging from the upvotes, might be worth keeping in mind that some of us think “upvote” should mean “this seems like it helps the conversation access relevant considerations/arguments” rather than “I agree with the conclusions.”

Still, my shortest reply to “Why expect there’s at least some risk if an AI is created that’s way more powerful than humanity?” is something like: “It seems pretty common-sensical to think that alien e... (read more)

The spread of opinions seems narrow compared to what I would expect. OP makes some bold predictions in his post. I see more debate over less controversial claims all of the time. Sorry, but what do aliens have to do with AI?
MIRI announces new "Death With Dignity" strategy


Additionally, the OP seems to me good for communication: Eliezer had a lot of bottled up thoughts, and here put them out in the world, where his thoughts can bump into other people who can in turn bump back into him.

AFAICT, conversation (free, open, "non-consequentialist" conversation, following interests and what seems worth sharing rather than solely backchaining from goals) is one of the places where consciousness and sanity sometimes enter. It's right there next to "free individual thought" in my list of beautiful things that are worth engaging in and safeguarding.

MIRI announces new "Death With Dignity" strategy

"To win any battle, you must fight as if you are already dead.” — Miyamoto Musashi.

I don't in fact personally know we won't make it. This may be because I'm more ignorant than Eliezer, or may be because he (or his April first identity, I guess) is overconfident on a model, relative to me; it's hard to tell.

Regardless, the bit about "don't get psychologically stuck having-to-(believe/pretend)-it's-gonna-work seems really sane and healthy to me. Like falling out of an illusion and noticing your feet on the ground. The ground is a more fun and joyful place... (read more)

I think I have more access to all of my emotional range nearer the ground, but this sentence doesn't ring true to me.

The ground is a more fun and joyful place to live, even when things are higher probability of death than one is used to acting-as-though, in my limited experience. 

On Doing the Improbable

I recently got a chance to interview a couple people about this who'd done product management or similar at bay area tech companies.

They agreed that you can't run projects there unless you project near-certainty the project will succeed. However, they had a trick that had failed to occur to me prior to them saying it, which is to find a mid-scale objective that is all of: a) quite likely to have at least a bit of use in its own right; b) almost certainly do-able; and c) a stepping-stone for getting closer to the (more worthwhile but higher-failure-odds) g... (read more)

Russia has Invaded Ukraine

What are the causes of your prediction?

Russia has Invaded Ukraine

Um, well, gosh. I have been estimating the odds of nuclear exchange as quite a lot below 5%. Why is your estimate so high?

Oh and also, there's potential for this to lead to a coup/domestic upheaval/regime change in Russia [] which would be an exceptionally volatile situation, kind of like having 6000 loose nukes until whoever takes power consolidates control including over the strategic forces again. So factoring that in, it should perhaps be over 5%. But again there should be advance warning for those developments inside Russia.
5% would be by the end of all this. Most of that probability comes from things developing in an unfortunate direction as I said, which would mean it goes against the current indications we have of neither the US nor NATO intervening militarily. This could be either them changing their minds, perhaps due to unexpectedly brutal Russian conduct during the war leading to a decision to impose a no-fly zone or something like that, or a cycle of retaliatory escalation due to unintended spillover of the war like I illustrated. Neither is too likely imo, and both will have advance warning if you're paying any attention luckily. The risk of a sudden nuclear exchange which doesn't even give enough warning for Americans to leave their cities is definitely lower, maybe 2% at most. But it's definitely present as well, due to the misjudgment risks etc. as I mentioned. Also, see the comments [] I just wrote on EA Forum [] .
Russia has Invaded Ukraine

Dominic Cummings's Twitter seems fairly high-information/low-noise re: Ukraine, to me. (ETA: at least for beginners like me who like to see background facts made explicit / sourced.)

Particularly interesting: he retweets someone saying that this morning: "Hard to tell with certainty, but most likely it means that [Russia's] nuclear command and control system received what is known as a preliminary command." (Link)

I'm not overly concerned with the news from this morning. In fact I expected them to raise the nuclear force readiness prior to or simultaneously to commencing the invasion, not now, which is expected going into a time of conflict/high tension from normal peacetime readiness. I had about a 5% chance this will escalate to a nuclear war going into it, and it's not much different now, certainly not above 10% (For context, my odds of escalation to full countervalue exchange in a US intervention in a Taiwan reunification campaign would be about 75%). Virtually all that probability is split between unfavorable developments dragging in NATO and accidents/miscalculation risk, which is elevated during tense times like this (something like, if the Russians had misinterpreted the attack submarine which entered their territorial waters last week [] as being a ballistic missile submarine sneaking up close to launch a first strike, or an early warning radar fluke/misidentification [] being taken seriously when it would've been dismissed during peacetime [], either of which could've caused them to launch on warning). Unintentional nuclear exchange will have no preceding signs, but unfavorable developments will, for example a NATO shootdown of a Russian plane or Russian fire straying over the border killing NATO troops which begins an escalation spiral. If we start seeing such incidents being reported, I would tell all my LW friends to get the fuck out of NATO cities they're living in immediately.
Russia has Invaded Ukraine

Does anyone here have good predictions, or better-operationalized questions, as to the extent to which Russia will/won't find occupying Ukraine to be a huge headache in the way that the US found occupying Afghanistan to be a huge headache?

Here, a senior fellow of the Brookings Institution shares his thoughts about possible parallels with Afghanistan, but more with the Russian occupation of Afghanistan: Could Ukraine be Putin’s Afghanistan? [] He doen't really answer this, but gives a couple of questions on which an answer could hinge: 1. Which state or states will be the frontline sponsor? [Pakistan in the case of Afghanistan] 2. Are they ready to take the heat from Russia? 3. How much support will the United States and NATO provide? 4. Will the insurgency spark a broader conflict, and can it be contained? 5. Are Ukrainians prepared to pay the price?
Russia has Invaded Ukraine

Speaking from ignorance: this prediction failure seems (from my ignorant perspective) similar to forecasting failures in Brexit and in the Trump 2016 election, in that it’s a case where some force whose motives are unlike Western academia/elites was surprising to them/us. If so, the moral might be to study the perspectives, motives, and capabilities of forces outside the Western elite on their own terms / by cobbling together an inside view from primary sources, rather than looking via Western experts/media (though this is much harder).

To what extent are ... (read more)

A LW post predicted a Feb invasion back in Dec: [] Years ago, many politicians and professors of international relations predicted a strong Russian response to talk of admitting Ukraine into NATO: []
In this vein, here's someone calling the situation we're seeing shortly after Russia's annexation of Crimea in 2014. Basic premise being that great power competition is alive and well, and the US and European leaders trying to tell themselves it's dead are consistently confused. Title aside I would not describe his talk as assigning blame - more that the US had a great hand after the collapse of the USSR and has played it poorly. I found a few talks from this guy, and they've all aged remarkably well.
Russia has Invaded Ukraine

One aim I could imagine having in Putin's shoes, that seems better achieved by slow telegraphing of war over Ukraine followed by actual war (vs by a frozen conflict), is gathering information about how the West is likely to respond to any other such wars/similar he might be tempted by.

(I know nothing of geopolitics, so please don't update from my thinking so. I got this idea from this essay)

I sort of get it and I want to believe it. But it makes no actual sense and that's terrifying. The west would barely care if Putin was doing this in the *stans or Georgia. The only other target to go to after Ukraine is Moldova and then the Baltics. If he goes in the Baltics that's war with NATO. Nothing about the reaction to Ukraine makes a difference there. It's black and white NATO vs not NATO. I feel like the most parsimonious explanation is he's not being very rational, rumors about him having terminal cancer are also pushing me towards that belief. It really doesn't seem like anyone on the Russian side saw this coming either, which is extra scary.
In Defense of Attempting Hard Things, and my story of the Leverage ecosystem

I, also, really appreciate Cathleen for writing this piece, and found it worth reading and full of relevant details. I'll try to add more substantive comments in a week or so, but wanted meanwhile to add my vote to those recommending that folks wanting to understand Leverage read this piece.

AnnaSalamon's Shortform

This is one of my bottlenecks on posting, so I'm hoping maybe someone will share thoughts on it that I might find useful:

I keep being torn between trying to write posts about things I have more-or-less understood already (which I therefore more-or-less know how to write up), and posts about things I presently care a lot about coming to a better understanding of (but where my thoughts are not so organized yet, and so trying to write about it involves much much use of the backspace, and ~80% of the time leads to me realizing the concepts are wrong, and going back to the drawing board).

I'm curious how others navigate this, or for general advice.

Not the question you asked, but... is it possible to somehow make your writing easier, and then you perhaps wouldn't have to choose between writing X or Y, because you could just write both? For example, not sure how much time you spend writing and editing, but maybe you could just record yourself talking and writing on blackboard, and then someone else (willing to donate their time) could transcribe it, and then you would just do the final editing and submit the thing? There is no backspace if you talk. What would you do if the same thing happened to you during a lecture? Maybe say "oops, I was wrong about this, because..." -- but this also can be included in the text. The entire wrong part could then be given a heading like "my first (unsuccessful) attempt", which would make the reader less confused.
I post all exploration/babble in LessWrong. For things I am more confident about and I want to push or get more serious feedback on I post it in the Alignment Forum or the Effective Altruism Forum. For example, when I started thinking about forecast aggregation I posted my unpolished thoughts here [1]. Now that I have grown more confident in my understanding I have been posting in the EA Forum [2]. This is not a hard rule, but I found the heuristic useful. My reasoning is something like: * LessWrong is a good place for wild speculation. I feel more likely to be called out on bullshit that does not make sense in a constructive way. * The EA Forum / Alignment Forum is less lenient of jargon and bullshit, plus it has a more consistent quality of content. It forces me to be more precise and rigourous. [1] [] [2] []
4Ben Pace5mo
For me, I only do the former post when I want to really nail something and put loads of work into it (e.g. my common knowledge post). I do the latter kind when I’ve just thought about a thing for a while and I feel like I got somewhere good. I don’t aim to write a perfect piece on it, I aim to write like I would explain my thinking in conversation. I typically can write such posts in ~2hrs (e.g. my environment [] post), and that seems worth publishing to me, and then time to move on with my thoughts.
This is something that I've been thinking about [] . Currently I sense that the overwhelming majority of people are hesitant to write about ideas that are in this exploratory phase. But collaboration at the exploratory phase is important! I suspect that the main way this collaboration currently happens is that people text their friends, but I feel like we can do better than that. I'm not exactly sure how. I think it's largely a social problem. Ie. people need to feel like it is ok to post early stage exploratory thoughts that are likely to have problems. And the way to get to that point is probably to see other (high status) members of the community doing so. There's a chicken-egg problem there, but it could probably be bootstrapped by just convincing [] a critical mass of people to just do it. I should point out that the LessWrong team has tried to solve this problem with the shortform and by making personal blog posts a thing that is very babble-y []. I think that is failing though because the social convention hasn't changed, and the social convention is the crux of the problem. Another possibility is that this type of exploratory conversation just doesn't happen "in public". It needs to happen in small, tight nit groups no larger than, say, four people. In which case it would be an interesting idea for eg. LessWrong to connect people and form such groups, that are limited in size and have the explicit goal of being for discussing exploratory ideas. Edit: A big reason why I'm excited about the possibility of (drastically) improving this exploratory phase is because of how high a level of action [] it is. It should trickle down and have positive effects in many places. In theory.
What would you like from How valuable would it be to you?

I continue to get a lot of value from, just as is. Partly using it myself and partly using it with friends/family who want help evaluating particular actions. Very grateful for this site.

The main additional feature that would be great for me would be help modeling how much of an update to make from Covid tests (e.g., how much does it help if everyone takes a rapid test before a gathering).

The Rationalists of the 1950s (and before) also called themselves “Rationalists”

Thanks! I appreciate knowing this. Do you happen to know if there's a connection between these 1950's rationalists, and the "critical rationalists" (who are a contemporary movement that involves David Deutsch, the "taking children seriously" people, and some larger set of folks who try to practice a certain set of motions and are based out of the UK, I think)?

the “critical rationalists” (who are a contemporary movement that involves David Deutsch, the “taking children seriously” people, and some larger set of folks who try to practice a certain set of motions and are based out of the UK, I think)?

Critical rationalism is basically the scientific philosophy of Karl R. Popper. An Austrian, he relocated to the UK in the 30s for similar reasons to Sigmund Freud's. So CR ended as being a kind of UK thing, despite having its roots in the Vienna Circle. (It also has a following in Oz and NZ, but not so much in the s... (read more)

I think the link in your comment points to something that seems like a one-man show. The man's name is Elliot Temple.

He has a picture on his homepage about the philosophical traditions he builds on, and apparently he makes some money selling his wisdom, but is he actually a part of some larger debate? I mean, other than the forum he owns and writes most of the comments on, with only two or three other active participants.

By the way, he was posting on LW, and got banned. He accuses David Deutch of organizing harassment against him, and generally seems obses... (read more)

Frame Control

But to understand better: if I'd posted a version of this with fully anonymous examples, nothing specifically traceable to Leverage, would that have felt good to you, or would something in it still feel weird?

I'd guess the OP would’ve felt maybe 35% less uneasy-making to me, sans Geoff/Aubrey/“current” examples.

The main thing that bothers me about the post is related to, but not identical to, the post’s use of current examples:

I think the phenomena you’re investigating are interesting and important, but that the framework you present for thinking about ... (read more)

I think I agree with ~everything in your two comments, and yet reading them I want to push back on something, not exactly sure what, but something like: look, there's this thing (or many things with a family resemblance) that happens and it's bad, and somehow it's super hard to describe / see it as it's happening.... and in particular I suspect the easiest, the first way out of it, the way out that's most readily accessible to someone mired in an "oops my internal organs are hooked up to a vampiric force" situation, does not primarily / mainly involve much... (read more)

Frame Control

I expect these topics are hard to write about, and that there’s value in attempting it anyway. I want to note that before I get into my complaints. So, um, thanks for sharing your data and thoughts about this hard-to-write-about (AFAICT) and significant (also AFAICT) topic!

Having acknowledged this, I’d like to share some things about my own perspective about how to have conversations like these “well”, and about why the above post makes me extremely uneasy.

First: there’s a kind of rigor that IMO the post lacks, and IMO the post is additionally in a domain... (read more)

Are you genuinely unsure whether or not there's a bad thing aella is (perhaps suboptimally) pointing at? If yes, then I feel like that's a cause for doom for whatever social communities you're trying to moderate. (By contrast, I'd find it highly understandable if you think aella is onto something, but you're worried she's packing too many ingredients into her description.) If not, then I find it interesting that you're using this pseudo-neutral framing ("whether it's bad") even though you already have at least some agreement with the things aella is trying to say. It's interesting that a post saying "There's this insidious, bad, community-destroying thing" gets mainly reactions like "Careful, this is a weapon that misguided people could use to ostracize innocents" as opposed to ones that acknowledge the really bad thing exists and is really bad. It almost seems like people are saying the bad thing cannot be remotely as bad as the risk that some people get accused of it unfairly, so we should better not talk about it too much. I'm open to being convinced that "unfair convictions" actually are the bigger problem. But I doubt it. My guess is that in instances where a person with benign cognition ends up unfairly ostracized, there's someone with interpersonally incorrigible cognition who had their fingers in the plot somehow. Therefore, the entire risk here (that people illegitimately use what seems like "too easily applicable of a social weapon") is a risk mostly because interpersonally incorrigible cognition / frame distortion exists in the first place. And I suspect that a good step at identifying solutions to the problem is by discussing it head-on and taking seriously the idea that it should be burnt with fire. I'm not saying we should already assume that this is the right answer. I'm just saying, maybe people are shying away from the possibility that it is the right answer. And if so, I want to urgently scream: STOP DOING THAT. Edit: I no longer endorse what I

To try to parse for me here, what I took away from each point:

1. "Where are the concrete claims that allow people to directly check"
2. Discomfort mixing claims about frame control with claims about Geoff, as lots of bad claims or beliefs can get sneaked in through the former while talking about the latter
3. I had a lot of trouble parsing this one, particularly the paragraph starting with "Uncharitable paraphrase/caricature:". I'm gathering something like "unease that I am making arguments that override normal good truth-seeking behavior, with the end goal ... (read more)

Upvoted because Anna articulated a lot of what I wanted to say but didn’t have the energy or clarity to say with such nuance.

Cornell Meetup

I've known Lionel since high school, and can vouch for him if it's somehow helpful. Additional thoughts: He's good at math; he's new enough to AI alignment that having anyone local-to-him (e.g. at Cornell / in Ithaca) who wants to talk about this would probably help, so don't be shy or think you need much background; he cares about this stuff; he enjoys thinking and trying to get at truth, and I tend to find him fun to talk to.

My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

A CFAR board member asked me to clarify what I meant about “corrupt”, also, in addition to this question.

So, um. Some legitimately true facts the board member asked me to share, to reduce confusion on these points:

  • There hasn’t been any embezzlement. No one has taken CFAR’s money and used it to buy themselves personal goods.
  • I think if you took non-profits that were CFAR’s size + duration (or larger and longer-lasting), in the US, and ranked them by “how corrupt is this non-profit according to observers who people think of as reasonable, and who got to
... (read more)

I have strong-upvoted this comment, which is not a sentence I think people usually ought leave as its own reply but which seems relevant given my relationship to Anna and CFAR and so forth.

Self-Integrity and the Drowning Child

Equally importantly IMO, it argues for transfer from a context where the effect of your actions is directly perceptionally obvious to one where it is unclear and filters through political structures (e.g., aid organizations and what they choose to do and to communicate; any governments they might be interacting with; any other players on the ground in the distant country) that will be hard to model accurately.

2Rob Bensinger7mo
My guess is that this has a relatively small effect on most people's moral intuitions (though maybe it should have a larger effect -- I don't think I grok the implicit concern here). I'd be curious if there's research bearing on this, and on the other speculations I tossed out there. (Or maybe Spencer or someone can go test it.)
My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

In the last two years, CFAR hasn't done much outward-facing work at all, due to COVID, and so has neither been a MIRI funnel nor definitively not a MIRI funnel.

Yes, but I would predict that we won't be the same sort of MIRI funnel going forward. This is because MIRI used to have specific research programs that it needed to hire for, and it it was sponsoring AIRCS (covering direct expenses plus loaning us some researchers to help run the thing) in order to recruit for that, and those research programs have been discontinued and so AIRCS won't be so much... (read more)

Zoe Curzi's Experience with Leverage Research

Thanks! I would love follow-up on LW to the twitch stream, if anyone wants to. There were a lot of really interesting things being said in the text chat that we didn’t manage to engage with, for example. Although unfortunately the recording was lost, which is unfortunate because IMO it was a great conversation.

TekhneMakre writes:

This suggests, to me, a (totally conjectural!) story where [Geoff] got into an escalating narrative cold war with the rationality community: first he perceives (possibly correctly) that the community rejects him…

This seems r... (read more)

I have video of the first 22 minutes at the beginning but at the end switched into my password manager (not showing passwords on screens but a series of sides where I'm registered), so I would want to publically post the video but I'm open to share it to individual people if someone wants to write something referencing it.

I wished I would have been more clear about how to do screen recording in a way that only captures one browser window...

A few more half-remembered notes from the conversation: [] If this is true, it does strike me as important and interesting. Speaking from a very abstract viewpoint not strongly grounded in observations, I'll speculate: One contributor, naturally, would be fear of false hope. One is (correctly) afraid of hope because hope somewhat entails investment and commitment. Fear of false hope could actually make hope be genuinely false, even when there could have been true hope. This happens because hope is to some extent a decision, so *expecting* you and others in the future to not collaborate in some way, also *constitutes a decision* to not collaborate in that way. If you will in the future behave in accordance with a plan, then it's probably correct to behave now in accordance with the plan; and if you will not, then it's probably correct to not now. (I tried to meditate on this in the footnotes to my post Hope and False Hope [].) (Obviously most things aren't very subject to this belief-plan mixing, and things where we can separate beliefs from plans are very useful for building foundations, but some non-separable things are important, e.g. open-ended collaboration.) This feels maybe related to a comment you Anna made in the conversation about Geoff seeming somewhat high on a dimension of manic-ness or something, and he said others have said he seems hypomanic. The story being, Geoff is more hopeful and hope-based in general, explaining why he sought collaboration, and caused collective hope in EA, and ended up feeling he had to defend his org's hope against hope-destroyers (which hope he referred to as "morale"). I k
Zoe Curzi's Experience with Leverage Research

Alas, no. I'm pretty bummed about it, because I thought the conversation was rather good, but Geoff pushed the "save recording" button after it was started and that didn't work.

Based on the fact Twitch is counter-intuitive about recording (it's caught me out before too) and the technical issues at the start, I made a backup recording just in case – only audio but hope it helps!:

My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

Thank you. I disagree with "... relishes 'breaking' others", and probably some other points, but a bunch of this seems really right and like content I haven't seen written up elsewhere. Do share more if you have it. I'm also curious where you got this stuff from.

My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

CFAR staff retreats often involve circling. Our last one, a couple weeks ago, had this, though as an optional evening thing that some but not most took part in.

My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

Basically no. Can't say a plain "no," but can say "basically no." I'm not willing to give details on this one. I'm somehow fretting on this one, asking if "basically no" is true from all vantage points (it isn't, but it's true from most), looking for a phrase similar to that but slightly weaker, considering e.g. "mostly no", but something stronger is true. I think this'll be the last thing I say in this thread about this topic.

My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

I think some of it has got to be that it's somehow easier to talk about CFAR/MIRI, rather than a sheer number of people thing. I think Leverage is somehow unusually hard to talk about, such that maybe we should figure out how to be extraordinarily kind/compassionate/gentle to anyone attempting it, or something.

I agree that Leverage has been unusually hard to talk about bluntly or honestly, and I think this has been true for most of its existence.

I also think the people at the periphery of Leverage, are starting to absorb the fact that they systematically had things hidden from them. That may be giving them new pause, before engaging with Leverage as a topic.

(I think that seems potentially fair, and considerate. To me, it doesn't feel like the same concern applies in engaging about CFAR. I also agree that there were probably fewer total people exposed to Leverage... (read more)

The fact that the people involved apparently find it uniquely difficult to talk about is a pretty good indication that Leverage != CFAR/MIRI in terms of cultishness/harms etc.

Load More