All of AnnaSalamon's Comments + Replies

I notice you are noticing confusion.  Any chance either the data, or the code, has a bug?

Can you say more about these for the benefit of folks like me who don't know about them?  What kind of "bad reception" or "controversial" was it?  Was it woo-flavored, or something else?

I think "you should one-box on Newcomb's problem" is probably an example.  By the time it was as formalized as TDT it was probably not all that woo-y looking, but prior to that I think a lot of people had an intuition along the lines of "yes it would be tempting to one-box but that's woo thinking that has me thinking that."

I want to state more explicitly where I’m coming from, about LW and woo.

One might think: “LW is one of few places on the internet that specializes in having only scientific materialist thoughts, without the woo.”

My own take is more like: “LW is one of few places on the internet that specializes in trying to have principled, truth-tracking models and practices about epistemics, and on e.g. trying to track that our maps are not the territory, trying to ask what we’d expect to see differently if particular claims are true/not-true, trying to be a “lens that s... (read more)

One example, maybe: I think the early 20th century behaviorists mistakenly (to my mind) discarded the idea that e.g. mice are usefully modeled as having something like (beliefs, memories, desires, internal states), because they lumped this in with something like "woo."  (They applied this also to humans, at least sometimes.)

The article Cognition all the way down argues that a similar transition may be useful in biology, where e.g. embryogenesis may be more rapidly modeled if biologists become willing to discuss the "intent" of a given cellular signal ... (read more)

Thanks.  Are you up for saying more about what algorithm (you in hindsight notice/surmise) you were following internally during that time, and how it did/didn't differ from the algorithm you were following during your "hyper-analytical programmer" times?

Can you say a bit more about why?

Do you agree that the social pressure in the pineapple and nose-picking examples isn't backchained from something like "don't spoil our game, we need everyone in this space to think/speak a certain way about this or our game will break"?

For example, if you go to a go club and ask the players there how to get stronger at go, and you take their advice, you'll both get stronger and go and become more like the kind of person who hangs out in go clubs.  If you just want to be in sync with the go club narrative and don't care about the game, you'll still ask most of the same questions: the go players will have a hard time telling your real motivation, and it's not clear to me that they have an incentive to try.

This seems right to me about most go clubs, but there’re a lot of other places t... (read more)

I haven't been able to construct any hypotheticals where I'd use it…. tl;dr I think narrative syncing is a natural category but I'm much less confident that “narrative syncing disguised as information sharing” is a problem worth noting,

I’m curious what you think of the examples in the long comment I just made (which was partly in response to this, but which I wrote as its own thing because I also wish I’d added it to the post in general).

I’m now thinking there’re really four concepts:

1. Narrative syncing.  (Example: “the sand is lava.”)
    
2. Narrative sy

I agree with some commenters (e.g. Taran) that the one example I gave isn’t persuasive on its own, and that I can imagine different characters in Alec’s shoes who want and mean different things.  But IMO there is a thing like this that totally happens pretty often in various contexts.  I’m going to try to give more examples, and a description of why I think they are examples, to show why I think this.

Example: I think most startups have a “plan” for success, and a set of “beliefs” about how things are going to go, that the CEO “believes” basically... (read more)

Some components of my own models, here:

1. I think most of the better-funded EA organizations would not prefer most LWers working there for $1M/yr, nor for a more typical salary, nor for free.
   
   (Even though many of these same LW-ers are useful many other places.)
    
2. I think many of the better-funded EA organizations would prefer (being able to continue employing at least their most useful staff members) to (receiving an annual donation equal to 30x what that staff member could make in industry).
    
3. If a typical LWer somehow really decided, deep in themselves

I could still be missing something, but I think this doesn't make sense. If the marginal numbers are as you say and if EA organizations started paying everyone 40% of their counterfactual value, the sum of “EA financial capital” would go down, and so the counterfactual value-in-“EA”-dollars of marginal people would also go down, and so the numbers would probably work out with lower valuations per person in dollars. Similarly, if “supply and demand” works for finding good people to work at EA organizations (which it might? I’m honestly unsure), the number... (read more)

Great use of logic to try to force us to have models, and to make those models explicit!

I don't know; finding a better solution sounds great, but there aren't that many people who talk here, and many of us are fairly reflective and ornery, so if a small group keeps repeatedly requesting this and doing it it'd probably be sufficient to keep "aspiring rationalist" as at least a substantial minority of what's said.

because EA has much more human capital than financial capital

Is this a typo? It seems in direct contradiction with the OPs claim that EA is people-bottlenecked and not funding-bottlenecked, which I otherwise took you to be agreeing with.

This is a bit off-topic with respect to the OP, but I really wish we’d more often say “aspiring rationalist” rather than “rationalist.” (Thanks to Said for doing this here.) The use of “rationalist” in parts of this comment thread and elsewhere grates on me. I expect most uses of either term are just people using the phrase other people use (which I have no real objection to), but it seems to me that when we say “aspiring rationalist” we at least sometimes remember that to form a map that is a better predictor of the territory requires aspiration, effor... (read more)

To elaborate a bit where I'm coming from here: I think the original idea with LessWrong was basically to bypass the usual immune system against reasoning, to expect this to lead to some problems, and to look for principles such as "notice your confusion," "if you have a gut feeling against something, look into it and don't just override it," "expect things to usually add up to normality" that can help us survive losing that immune system. (Advantage of losing it: you can reason!)

My guess is that that (having principles in place of a reflexive or socially ... (read more)

Do you have a principled model of what an "epistemic immune system" is and why/whether we should have one?

It seems “taboo” to me. Like, when I go to think about this, I feel … inhibited in some not-very-verbal, not-very-explicit way. Kinda like how I feel if I imagine asking an inane question of a stranger without a socially sensible excuse, or when a clerk asked me why I was buying so many canned goods very early in Covid.

I think we are partly seeing the echoes of a social flinch here, somehow. It bears examining!

Also, MIRI isn't (necessarily) a hive mind, so not sure if Rob, Nate or Abram actually share the same estimate of how doomed we are as Eliezer.

Indeed, I expect that the views of at least some individuals working at MIRI vary considerably.

In some ways, the post would seem more accurate to me if it had the Onion-esque headline: Eliezer announces on MIRI’s behalf that “MIRI adopts new ‘Death with Dignity’ strategy.”

Still, I love the post a lot. Also, Eliezer has always been pivotal in MIRI.

The spread of opinions seems narrow compared to what I would expect. OP makes some bold predictions in his post. I see more debate over less controversial claims all of the time.

That's fair.

what do aliens have to do with AI?

Sorry, I said it badly/unclearly. What I meant was: most ways to design powerful AI will, on my best guess, be "alien" intelligences, in the sense that they are different from us (think differently, have different goals/values, etc.).

I personally find it relatively gentle and reminding-toward-beauty, but YMMV. Lyrics.

Where do I find these threads?

I've had in my head all day Leonard Cohen's song "You got me singing," which he wrote toward the end of his life, watching death approach.

Why is everyone here in agreement that…

We’re not. There’s a spread of perspectives and opinions and lack-of-opinions. If you’re judging from the upvotes, might be worth keeping in mind that some of us think “upvote” should mean “this seems like it helps the conversation access relevant considerations/arguments” rather than “I agree with the conclusions.”

Still, my shortest reply to “Why expect there’s at least some risk if an AI is created that’s way more powerful than humanity?” is something like: “It seems pretty common-sensical to think that alien e... (read more)

Ditto.

Additionally, the OP seems to me good for communication: Eliezer had a lot of bottled up thoughts, and here put them out in the world, where his thoughts can bump into other people who can in turn bump back into him.

AFAICT, conversation (free, open, "non-consequentialist" conversation, following interests and what seems worth sharing rather than solely backchaining from goals) is one of the places where consciousness and sanity sometimes enter. It's right there next to "free individual thought" in my list of beautiful things that are worth engaging in and safeguarding.

"To win any battle, you must fight as if you are already dead.” — Miyamoto Musashi.

I don't in fact personally know we won't make it. This may be because I'm more ignorant than Eliezer, or may be because he (or his April first identity, I guess) is overconfident on a model, relative to me; it's hard to tell.

Regardless, the bit about "don't get psychologically stuck having-to-(believe/pretend)-it's-gonna-work seems really sane and healthy to me. Like falling out of an illusion and noticing your feet on the ground. The ground is a more fun and joyful place... (read more)

I recently got a chance to interview a couple people about this who'd done product management or similar at bay area tech companies.

They agreed that you can't run projects there unless you project near-certainty the project will succeed. However, they had a trick that had failed to occur to me prior to them saying it, which is to find a mid-scale objective that is all of: a) quite likely to have at least a bit of use in its own right; b) almost certainly do-able; and c) a stepping-stone for getting closer to the (more worthwhile but higher-failure-odds) g... (read more)

What are the causes of your prediction?

Um, well, gosh. I have been estimating the odds of nuclear exchange as quite a lot below 5%. Why is your estimate so high?

Dominic Cummings's Twitter seems fairly high-information/low-noise re: Ukraine, to me. (ETA: at least for beginners like me who like to see background facts made explicit / sourced.)

Particularly interesting: he retweets someone saying that this morning: "Hard to tell with certainty, but most likely it means that [Russia's] nuclear command and control system received what is known as a preliminary command." (Link)

Does anyone here have good predictions, or better-operationalized questions, as to the extent to which Russia will/won't find occupying Ukraine to be a huge headache in the way that the US found occupying Afghanistan to be a huge headache?

Speaking from ignorance: this prediction failure seems (from my ignorant perspective) similar to forecasting failures in Brexit and in the Trump 2016 election, in that it’s a case where some force whose motives are unlike Western academia/elites was surprising to them/us. If so, the moral might be to study the perspectives, motives, and capabilities of forces outside the Western elite on their own terms / by cobbling together an inside view from primary sources, rather than looking via Western experts/media (though this is much harder).

To what extent are ... (read more)

One aim I could imagine having in Putin's shoes, that seems better achieved by slow telegraphing of war over Ukraine followed by actual war (vs by a frozen conflict), is gathering information about how the West is likely to respond to any other such wars/similar he might be tempted by.

(I know nothing of geopolitics, so please don't update from my thinking so. I got this idea from this essay)

I, also, really appreciate Cathleen for writing this piece, and found it worth reading and full of relevant details. I'll try to add more substantive comments in a week or so, but wanted meanwhile to add my vote to those recommending that folks wanting to understand Leverage read this piece.

This is one of my bottlenecks on posting, so I'm hoping maybe someone will share thoughts on it that I might find useful:

I keep being torn between trying to write posts about things I have more-or-less understood already (which I therefore more-or-less know how to write up), and posts about things I presently care a lot about coming to a better understanding of (but where my thoughts are not so organized yet, and so trying to write about it involves much much use of the backspace, and ~80% of the time leads to me realizing the concepts are wrong, and going back to the drawing board).

I'm curious how others navigate this, or for general advice.

I continue to get a lot of value from microcovid.org, just as is. Partly using it myself and partly using it with friends/family who want help evaluating particular actions. Very grateful for this site.

The main additional feature that would be great for me would be help modeling how much of an update to make from Covid tests (e.g., how much does it help if everyone takes a rapid test before a gathering).

Thanks! I appreciate knowing this. Do you happen to know if there's a connection between these 1950's rationalists, and the "critical rationalists" (who are a contemporary movement that involves David Deutsch, the "taking children seriously" people, and some larger set of folks who try to practice a certain set of motions and are based out of the UK, I think)?

But to understand better: if I'd posted a version of this with fully anonymous examples, nothing specifically traceable to Leverage, would that have felt good to you, or would something in it still feel weird?

I'd guess the OP would’ve felt maybe 35% less uneasy-making to me, sans Geoff/Aubrey/“current” examples.

The main thing that bothers me about the post is related to, but not identical to, the post’s use of current examples:

I think the phenomena you’re investigating are interesting and important, but that the framework you present for thinking about ... (read more)

I expect these topics are hard to write about, and that there’s value in attempting it anyway. I want to note that before I get into my complaints. So, um, thanks for sharing your data and thoughts about this hard-to-write-about (AFAICT) and significant (also AFAICT) topic!

Having acknowledged this, I’d like to share some things about my own perspective about how to have conversations like these “well”, and about why the above post makes me extremely uneasy.

First: there’s a kind of rigor that IMO the post lacks, and IMO the post is additionally in a domain... (read more)

I've known Lionel since high school, and can vouch for him if it's somehow helpful. Additional thoughts: He's good at math; he's new enough to AI alignment that having anyone local-to-him (e.g. at Cornell / in Ithaca) who wants to talk about this would probably help, so don't be shy or think you need much background; he cares about this stuff; he enjoys thinking and trying to get at truth, and I tend to find him fun to talk to.

A CFAR board member asked me to clarify what I meant about “corrupt”, also, in addition to this question.

So, um. Some legitimately true facts the board member asked me to share, to reduce confusion on these points:

• There hasn’t been any embezzlement. No one has taken CFAR’s money and used it to buy themselves personal goods.
• I think if you took non-profits that were CFAR’s size + duration (or larger and longer-lasting), in the US, and ranked them by “how corrupt is this non-profit according to observers who people think of as reasonable, and who got to

Equally importantly IMO, it argues for transfer from a context where the effect of your actions is directly perceptionally obvious to one where it is unclear and filters through political structures (e.g., aid organizations and what they choose to do and to communicate; any governments they might be interacting with; any other players on the ground in the distant country) that will be hard to model accurately.

In the last two years, CFAR hasn't done much outward-facing work at all, due to COVID, and so has neither been a MIRI funnel nor definitively not a MIRI funnel.

Yes, but I would predict that we won't be the same sort of MIRI funnel going forward. This is because MIRI used to have specific research programs that it needed to hire for, and it it was sponsoring AIRCS (covering direct expenses plus loaning us some researchers to help run the thing) in order to recruit for that, and those research programs have been discontinued and so AIRCS won't be so much... (read more)

Thanks! I would love follow-up on LW to the twitch stream, if anyone wants to. There were a lot of really interesting things being said in the text chat that we didn’t manage to engage with, for example. Although unfortunately the recording was lost, which is unfortunate because IMO it was a great conversation.

TekhneMakre writes:

This suggests, to me, a (totally conjectural!) story where [Geoff] got into an escalating narrative cold war with the rationality community: first he perceives (possibly correctly) that the community rejects him…

This seems r... (read more)

Alas, no. I'm pretty bummed about it, because I thought the conversation was rather good, but Geoff pushed the "save recording" button after it was started and that didn't work.

Thank you. I disagree with "... relishes 'breaking' others", and probably some other points, but a bunch of this seems really right and like content I haven't seen written up elsewhere. Do share more if you have it. I'm also curious where you got this stuff from.

CFAR staff retreats often involve circling. Our last one, a couple weeks ago, had this, though as an optional evening thing that some but not most took part in.

Basically no. Can't say a plain "no," but can say "basically no." I'm not willing to give details on this one. I'm somehow fretting on this one, asking if "basically no" is true from all vantage points (it isn't, but it's true from most), looking for a phrase similar to that but slightly weaker, considering e.g. "mostly no", but something stronger is true. I think this'll be the last thing I say in this thread about this topic.

I think some of it has got to be that it's somehow easier to talk about CFAR/MIRI, rather than a sheer number of people thing. I think Leverage is somehow unusually hard to talk about, such that maybe we should figure out how to be extraordinarily kind/compassionate/gentle to anyone attempting it, or something.