Can you say more about these for the benefit of folks like me who don't know about them? What kind of "bad reception" or "controversial" was it? Was it woo-flavored, or something else?
I think "you should one-box on Newcomb's problem" is probably an example. By the time it was as formalized as TDT it was probably not all that woo-y looking, but prior to that I think a lot of people had an intuition along the lines of "yes it would be tempting to one-box but that's woo thinking that has me thinking that."
I want to state more explicitly where I’m coming from, about LW and woo.
One might think: “LW is one of few places on the internet that specializes in having only scientific materialist thoughts, without the woo.”
My own take is more like: “LW is one of few places on the internet that specializes in trying to have principled, truth-tracking models and practices about epistemics, and on e.g. trying to track that our maps are not the territory, trying to ask what we’d expect to see differently if particular claims are true/not-true, trying to be a “lens that s... (read more)
One example, maybe: I think the early 20th century behaviorists mistakenly (to my mind) discarded the idea that e.g. mice are usefully modeled as having something like (beliefs, memories, desires, internal states), because they lumped this in with something like "woo." (They applied this also to humans, at least sometimes.)
The article Cognition all the way down argues that a similar transition may be useful in biology, where e.g. embryogenesis may be more rapidly modeled if biologists become willing to discuss the "intent" of a given cellular signal ... (read more)
Thanks. Are you up for saying more about what algorithm (you in hindsight notice/surmise) you were following internally during that time, and how it did/didn't differ from the algorithm you were following during your "hyper-analytical programmer" times?
Can you say a bit more about why?
Do you agree that the social pressure in the pineapple and nose-picking examples isn't backchained from something like "don't spoil our game, we need everyone in this space to think/speak a certain way about this or our game will break"?
For example, if you go to a go club and ask the players there how to get stronger at go, and you take their advice, you'll both get stronger and go and become more like the kind of person who hangs out in go clubs. If you just want to be in sync with the go club narrative and don't care about the game, you'll still ask most of the same questions: the go players will have a hard time telling your real motivation, and it's not clear to me that they have an incentive to try.
This seems right to me about most go clubs, but there’re a lot of other places t... (read more)
I haven't been able to construct any hypotheticals where I'd use it…. tl;dr I think narrative syncing is a natural category but I'm much less confident that “narrative syncing disguised as information sharing” is a problem worth noting,
I’m curious what you think of the examples in the long comment I just made (which was partly in response to this, but which I wrote as its own thing because I also wish I’d added it to the post in general).
I’m now thinking there’re really four concepts:
I agree with some commenters (e.g. Taran) that the one example I gave isn’t persuasive on its own, and that I can imagine different characters in Alec’s shoes who want and mean different things. But IMO there is a thing like this that totally happens pretty often in various contexts. I’m going to try to give more examples, and a description of why I think they are examples, to show why I think this.
Example: I think most startups have a “plan” for success, and a set of “beliefs” about how things are going to go, that the CEO “believes” basically... (read more)
If you are looking for more examples of narrative syncing:
Some components of my own models, here:
I could still be missing something, but I think this doesn't make sense. If the marginal numbers are as you say and if EA organizations started paying everyone 40% of their counterfactual value, the sum of “EA financial capital” would go down, and so the counterfactual value-in-“EA”-dollars of marginal people would also go down, and so the numbers would probably work out with lower valuations per person in dollars. Similarly, if “supply and demand” works for finding good people to work at EA organizations (which it might? I’m honestly unsure), the number... (read more)
Great use of logic to try to force us to have models, and to make those models explicit!
I don't know; finding a better solution sounds great, but there aren't that many people who talk here, and many of us are fairly reflective and ornery, so if a small group keeps repeatedly requesting this and doing it it'd probably be sufficient to keep "aspiring rationalist" as at least a substantial minority of what's said.
FWIW, I would genuinely use the term 'aspiring rationalist' more if it struck me as more technically correct — in my head 'person aspiring to be rational' ≈ 'rationalist'. So I parse aspiring rationalist as 'person aspiring to be a person aspiring to be rational'.
'Aspiring rationalist' makes sense if I equate 'rationalist' with 'rational', but that's exactly the thing I don't want to do.
Maybe we just need a new word here. E.g., -esce is a root meaning "to become" (as in coalesce, acquiesce, evanesce, convalescent, iridescent, effervescent, quiescent). We c... (read more)
because EA has much more human capital than financial capital
Is this a typo? It seems in direct contradiction with the OPs claim that EA is people-bottlenecked and not funding-bottlenecked, which I otherwise took you to be agreeing with.
This is a bit off-topic with respect to the OP, but I really wish we’d more often say “aspiring rationalist” rather than “rationalist.” (Thanks to Said for doing this here.) The use of “rationalist” in parts of this comment thread and elsewhere grates on me. I expect most uses of either term are just people using the phrase other people use (which I have no real objection to), but it seems to me that when we say “aspiring rationalist” we at least sometimes remember that to form a map that is a better predictor of the territory requires aspiration, effor... (read more)
I dutifully tried to say "aspiring rationalist" for awhile, but in addition to the syllable count thing just being too much of a pain, it... feels like it's solving the wrong problem.
An argument that persuaded me to stop caring about it as much: communities of guitarists don't call themselves "aspiring guitarists". You're either doing guitaring, or you're not. (in some sense similar for being a scientist or researcher).
Meanwhile, I know at least some people definitely meet any reasonable bar for "actually a goddamn rationalist". If you intentionally reflec... (read more)
I agree that "aspiring rationalist" captures the desired meaning better than "rationalist", in most cases, but... I think language has some properties, studied and documented by linguists, which define a set of legal moves, and rationalist->aspiring rationalist is an invalid move. That is: everyone using "aspiring rationalist" is an unstable state from which people will spontaneously drop the word aspiring, and people in a mixed linguistic environemnt will consistently adopt the shorter one. Aspiring Rationalist just doesn't fit within the syllable-count budget, and if we want to displace the unmodified term Rationalist, we need a different solution.
To elaborate a bit where I'm coming from here: I think the original idea with LessWrong was basically to bypass the usual immune system against reasoning, to expect this to lead to some problems, and to look for principles such as "notice your confusion," "if you have a gut feeling against something, look into it and don't just override it," "expect things to usually add up to normality" that can help us survive losing that immune system. (Advantage of losing it: you can reason!)
My guess is that that (having principles in place of a reflexive or socially ... (read more)
I don't have a complete or principled model of what an epistemic immune system is or ought to be, in the area of woo, but I have some fragments.
One way of looking at it is that we look at a cluster of ideas, form an outside view of how much value and how much crazymaking there is inside it, and decide whether to engage. Part of the epistemic immune system is tracking the cost side of the corresponding cost/benefit. But this cost/benefit analysis doesn't generalize well between people; there's a big difference between a well-grounded well-studied practition... (read more)
Do you have a principled model of what an "epistemic immune system" is and why/whether we should have one?
To elaborate a bit where I'm coming from here: I think the original idea with LessWrong was basically to bypass the usual immune system against reasoning, to expect this to lead to some problems, and to look for principles such as "notice your confusion," "if you have a gut feeling against something, look into it and don't just override it," "expect things to usually add up to normality" that can help us survive losing that immune system. (Advantage of losing it: you can reason!)
My guess is that that (having principles in place of a reflexive or socially ... (read more)
It seems “taboo” to me. Like, when I go to think about this, I feel … inhibited in some not-very-verbal, not-very-explicit way. Kinda like how I feel if I imagine asking an inane question of a stranger without a socially sensible excuse, or when a clerk asked me why I was buying so many canned goods very early in Covid.
I think we are partly seeing the echoes of a social flinch here, somehow. It bears examining!
Open tolerance of the people involved with status quo and fear of alienating / making enemies of powerful groups is a core part of current EA culture! Steve's top comment on this post is an example of enforcing/reiterating this norm.
It's an unwritten rule that seems very strongly enforced yet never really explicitly acknowledged, much less discussed. People were shadow blacklisted by CEA from the Covid documentary they funded for being too disrespectful in their speech re: how governments have handled covid. That fits what I'd consider a taboo,... (read more)
FYI, I thought this sort of idea was an obvious one, and I've been continuously surprised that it didn't have more discussion. I don't feel inhibited and am sort of surprised you are.
(I do think there's a lot of ways to do this badly, with costs on the overall coordination-commons, so, maybe I feel somewhat inhibited from actually going off to do the thing. But I don't feel inhibited from brainstorming potential ways to address the costs and thinking about how to do it)
Also, MIRI isn't (necessarily) a hive mind, so not sure if Rob, Nate or Abram actually share the same estimate of how doomed we are as Eliezer.
Indeed, I expect that the views of at least some individuals working at MIRI vary considerably.
In some ways, the post would seem more accurate to me if it had the Onion-esque headline: Eliezer announces on MIRI’s behalf that “MIRI adopts new ‘Death with Dignity’ strategy.”
Still, I love the post a lot. Also, Eliezer has always been pivotal in MIRI.
The spread of opinions seems narrow compared to what I would expect. OP makes some bold predictions in his post. I see more debate over less controversial claims all of the time.
That's fair.
what do aliens have to do with AI?
Sorry, I said it badly/unclearly. What I meant was: most ways to design powerful AI will, on my best guess, be "alien" intelligences, in the sense that they are different from us (think differently, have different goals/values, etc.).
I personally find it relatively gentle and reminding-toward-beauty, but YMMV. Lyrics.
Where do I find these threads?
I've had in my head all day Leonard Cohen's song "You got me singing," which he wrote toward the end of his life, watching death approach.
Why is everyone here in agreement that…
We’re not. There’s a spread of perspectives and opinions and lack-of-opinions. If you’re judging from the upvotes, might be worth keeping in mind that some of us think “upvote” should mean “this seems like it helps the conversation access relevant considerations/arguments” rather than “I agree with the conclusions.”
Still, my shortest reply to “Why expect there’s at least some risk if an AI is created that’s way more powerful than humanity?” is something like: “It seems pretty common-sensical to think that alien e... (read more)
Ditto.
Additionally, the OP seems to me good for communication: Eliezer had a lot of bottled up thoughts, and here put them out in the world, where his thoughts can bump into other people who can in turn bump back into him.
AFAICT, conversation (free, open, "non-consequentialist" conversation, following interests and what seems worth sharing rather than solely backchaining from goals) is one of the places where consciousness and sanity sometimes enter. It's right there next to "free individual thought" in my list of beautiful things that are worth engaging in and safeguarding.
"To win any battle, you must fight as if you are already dead.” — Miyamoto Musashi.
I don't in fact personally know we won't make it. This may be because I'm more ignorant than Eliezer, or may be because he (or his April first identity, I guess) is overconfident on a model, relative to me; it's hard to tell.
Regardless, the bit about "don't get psychologically stuck having-to-(believe/pretend)-it's-gonna-work seems really sane and healthy to me. Like falling out of an illusion and noticing your feet on the ground. The ground is a more fun and joyful place... (read more)
I think I have more access to all of my emotional range nearer the ground, but this sentence doesn't ring true to me.
The ground is a more fun and joyful place to live, even when things are higher probability of death than one is used to acting-as-though, in my limited experience.
I recently got a chance to interview a couple people about this who'd done product management or similar at bay area tech companies.
They agreed that you can't run projects there unless you project near-certainty the project will succeed. However, they had a trick that had failed to occur to me prior to them saying it, which is to find a mid-scale objective that is all of: a) quite likely to have at least a bit of use in its own right; b) almost certainly do-able; and c) a stepping-stone for getting closer to the (more worthwhile but higher-failure-odds) g... (read more)
What are the causes of your prediction?
Um, well, gosh. I have been estimating the odds of nuclear exchange as quite a lot below 5%. Why is your estimate so high?
Dominic Cummings's Twitter seems fairly high-information/low-noise re: Ukraine, to me. (ETA: at least for beginners like me who like to see background facts made explicit / sourced.)
Particularly interesting: he retweets someone saying that this morning: "Hard to tell with certainty, but most likely it means that [Russia's] nuclear command and control system received what is known as a preliminary command." (Link)
Does anyone here have good predictions, or better-operationalized questions, as to the extent to which Russia will/won't find occupying Ukraine to be a huge headache in the way that the US found occupying Afghanistan to be a huge headache?
Speaking from ignorance: this prediction failure seems (from my ignorant perspective) similar to forecasting failures in Brexit and in the Trump 2016 election, in that it’s a case where some force whose motives are unlike Western academia/elites was surprising to them/us. If so, the moral might be to study the perspectives, motives, and capabilities of forces outside the Western elite on their own terms / by cobbling together an inside view from primary sources, rather than looking via Western experts/media (though this is much harder).
To what extent are ... (read more)
One aim I could imagine having in Putin's shoes, that seems better achieved by slow telegraphing of war over Ukraine followed by actual war (vs by a frozen conflict), is gathering information about how the West is likely to respond to any other such wars/similar he might be tempted by.
(I know nothing of geopolitics, so please don't update from my thinking so. I got this idea from this essay)
I, also, really appreciate Cathleen for writing this piece, and found it worth reading and full of relevant details. I'll try to add more substantive comments in a week or so, but wanted meanwhile to add my vote to those recommending that folks wanting to understand Leverage read this piece.
This is one of my bottlenecks on posting, so I'm hoping maybe someone will share thoughts on it that I might find useful:
I keep being torn between trying to write posts about things I have more-or-less understood already (which I therefore more-or-less know how to write up), and posts about things I presently care a lot about coming to a better understanding of (but where my thoughts are not so organized yet, and so trying to write about it involves much much use of the backspace, and ~80% of the time leads to me realizing the concepts are wrong, and going back to the drawing board).
I'm curious how others navigate this, or for general advice.
I continue to get a lot of value from microcovid.org, just as is. Partly using it myself and partly using it with friends/family who want help evaluating particular actions. Very grateful for this site.
The main additional feature that would be great for me would be help modeling how much of an update to make from Covid tests (e.g., how much does it help if everyone takes a rapid test before a gathering).
Thanks! I appreciate knowing this. Do you happen to know if there's a connection between these 1950's rationalists, and the "critical rationalists" (who are a contemporary movement that involves David Deutsch, the "taking children seriously" people, and some larger set of folks who try to practice a certain set of motions and are based out of the UK, I think)?
the “critical rationalists” (who are a contemporary movement that involves David Deutsch, the “taking children seriously” people, and some larger set of folks who try to practice a certain set of motions and are based out of the UK, I think)?
Critical rationalism is basically the scientific philosophy of Karl R. Popper. An Austrian, he relocated to the UK in the 30s for similar reasons to Sigmund Freud's. So CR ended as being a kind of UK thing, despite having its roots in the Vienna Circle. (It also has a following in Oz and NZ, but not so much in the s... (read more)
I think the link in your comment points to something that seems like a one-man show. The man's name is Elliot Temple.
He has a picture on his homepage about the philosophical traditions he builds on, and apparently he makes some money selling his wisdom, but is he actually a part of some larger debate? I mean, other than the forum he owns and writes most of the comments on, with only two or three other active participants.
By the way, he was posting on LW, and got banned. He accuses David Deutch of organizing harassment against him, and generally seems obses... (read more)
But to understand better: if I'd posted a version of this with fully anonymous examples, nothing specifically traceable to Leverage, would that have felt good to you, or would something in it still feel weird?
I'd guess the OP would’ve felt maybe 35% less uneasy-making to me, sans Geoff/Aubrey/“current” examples.
The main thing that bothers me about the post is related to, but not identical to, the post’s use of current examples:
I think the phenomena you’re investigating are interesting and important, but that the framework you present for thinking about ... (read more)
I think I agree with ~everything in your two comments, and yet reading them I want to push back on something, not exactly sure what, but something like: look, there's this thing (or many things with a family resemblance) that happens and it's bad, and somehow it's super hard to describe / see it as it's happening.... and in particular I suspect the easiest, the first way out of it, the way out that's most readily accessible to someone mired in an "oops my internal organs are hooked up to a vampiric force" situation, does not primarily / mainly involve much... (read more)
I expect these topics are hard to write about, and that there’s value in attempting it anyway. I want to note that before I get into my complaints. So, um, thanks for sharing your data and thoughts about this hard-to-write-about (AFAICT) and significant (also AFAICT) topic!
Having acknowledged this, I’d like to share some things about my own perspective about how to have conversations like these “well”, and about why the above post makes me extremely uneasy.
First: there’s a kind of rigor that IMO the post lacks, and IMO the post is additionally in a domain... (read more)
To try to parse for me here, what I took away from each point:
1. "Where are the concrete claims that allow people to directly check"
2. Discomfort mixing claims about frame control with claims about Geoff, as lots of bad claims or beliefs can get sneaked in through the former while talking about the latter
3. I had a lot of trouble parsing this one, particularly the paragraph starting with "Uncharitable paraphrase/caricature:". I'm gathering something like "unease that I am making arguments that override normal good truth-seeking behavior, with the end goal ... (read more)
Upvoted because Anna articulated a lot of what I wanted to say but didn’t have the energy or clarity to say with such nuance.
I've known Lionel since high school, and can vouch for him if it's somehow helpful. Additional thoughts: He's good at math; he's new enough to AI alignment that having anyone local-to-him (e.g. at Cornell / in Ithaca) who wants to talk about this would probably help, so don't be shy or think you need much background; he cares about this stuff; he enjoys thinking and trying to get at truth, and I tend to find him fun to talk to.
A CFAR board member asked me to clarify what I meant about “corrupt”, also, in addition to this question.
So, um. Some legitimately true facts the board member asked me to share, to reduce confusion on these points:
I have strong-upvoted this comment, which is not a sentence I think people usually ought leave as its own reply but which seems relevant given my relationship to Anna and CFAR and so forth.
Equally importantly IMO, it argues for transfer from a context where the effect of your actions is directly perceptionally obvious to one where it is unclear and filters through political structures (e.g., aid organizations and what they choose to do and to communicate; any governments they might be interacting with; any other players on the ground in the distant country) that will be hard to model accurately.
In the last two years, CFAR hasn't done much outward-facing work at all, due to COVID, and so has neither been a MIRI funnel nor definitively not a MIRI funnel.
Yes, but I would predict that we won't be the same sort of MIRI funnel going forward. This is because MIRI used to have specific research programs that it needed to hire for, and it it was sponsoring AIRCS (covering direct expenses plus loaning us some researchers to help run the thing) in order to recruit for that, and those research programs have been discontinued and so AIRCS won't be so much... (read more)
Thanks! I would love follow-up on LW to the twitch stream, if anyone wants to. There were a lot of really interesting things being said in the text chat that we didn’t manage to engage with, for example. Although unfortunately the recording was lost, which is unfortunate because IMO it was a great conversation.
TekhneMakre writes:
This suggests, to me, a (totally conjectural!) story where [Geoff] got into an escalating narrative cold war with the rationality community: first he perceives (possibly correctly) that the community rejects him…
This seems r... (read more)
I have video of the first 22 minutes at the beginning but at the end switched into my password manager (not showing passwords on screens but a series of sides where I'm registered), so I would want to publically post the video but I'm open to share it to individual people if someone wants to write something referencing it.
I wished I would have been more clear about how to do screen recording in a way that only captures one browser window...
Alas, no. I'm pretty bummed about it, because I thought the conversation was rather good, but Geoff pushed the "save recording" button after it was started and that didn't work.
Based on the fact Twitch is counter-intuitive about recording (it's caught me out before too) and the technical issues at the start, I made a backup recording just in case – only audio but hope it helps!:
https://drive.google.com/file/d/1Af1dl-v7Q7uJhdX8Al9FsrJDBc4BqM_f/view?usp=sharing
Thank you. I disagree with "... relishes 'breaking' others", and probably some other points, but a bunch of this seems really right and like content I haven't seen written up elsewhere. Do share more if you have it. I'm also curious where you got this stuff from.
CFAR staff retreats often involve circling. Our last one, a couple weeks ago, had this, though as an optional evening thing that some but not most took part in.
Basically no. Can't say a plain "no," but can say "basically no." I'm not willing to give details on this one. I'm somehow fretting on this one, asking if "basically no" is true from all vantage points (it isn't, but it's true from most), looking for a phrase similar to that but slightly weaker, considering e.g. "mostly no", but something stronger is true. I think this'll be the last thing I say in this thread about this topic.
I think some of it has got to be that it's somehow easier to talk about CFAR/MIRI, rather than a sheer number of people thing. I think Leverage is somehow unusually hard to talk about, such that maybe we should figure out how to be extraordinarily kind/compassionate/gentle to anyone attempting it, or something.
I agree that Leverage has been unusually hard to talk about bluntly or honestly, and I think this has been true for most of its existence.
I also think the people at the periphery of Leverage, are starting to absorb the fact that they systematically had things hidden from them. That may be giving them new pause, before engaging with Leverage as a topic.
(I think that seems potentially fair, and considerate. To me, it doesn't feel like the same concern applies in engaging about CFAR. I also agree that there were probably fewer total people exposed to Leverage... (read more)
The fact that the people involved apparently find it uniquely difficult to talk about is a pretty good indication that Leverage != CFAR/MIRI in terms of cultishness/harms etc.
I notice you are noticing confusion. Any chance either the data, or the code, has a bug?
Good to ask, but I'm not sure what it would be. The code is just a linear regression I did in a spreadsheet, and eyeballing the data points, it doesn't look like there are any patterns that a regression is missing. I tried it several different ways (comparing to different smaller models, comparing to averages of smaller models, excluding extreme values, etc.) and the correlation was always zero. Here's the raw data:
https://docs.google.com/spreadsheets/d/1Y_00UcsYZeOwRuwXWD5_nQWAJp4A0aNoySW0EOhnp0Y/edit?usp=sharing
It's hard to know if there is some critical... (read more)