All of AnnaSalamon's Comments + Replies

The public early Covid-19 conversation (in like Feb-April 2020) seemed pretty hopeful to me -- decent arguments, slow but asymmetrically correct updating on some of those arguments, etc.  Later everything became politicized and stupid re: covid.

Right now I think there's some opportunity for real conversation re: AI.  I don't know what useful thing follows from that, but I do think it may not last, and that it's pretty cool.  I care more about the "an opening for real conversation" thing than for the changing overton window as such, although I think the former probably follows from the latter (first encounters are often more real somehow).

This seems like a very off-distribution move from Eliezer—which I suspect is in large part the point: when your model predicts doom by default, you go off-distribution in search of higher-variance regions of outcome space.

That's not how I read it.  To me it's an attempt at the simple, obvious strategy of telling people ~all the truth he can about a subject they care a lot about and where he and they have common interests.  This doesn't seem like an attempt to be clever or explore high-variance tails.  More like an attempt to explore the obvious strategy, or to follow the obvious bits of common-sense ethics, now that lots of allegedly clever 4-dimensional chess has turned out stupid.

Thanks for the suggestion.  I haven't read it.  I'd thought from hearsay that it is rather lacking in "light" -- a bunch of people who're kinda bored and can't remember the meaning of life -- is that true?  Could be worth it anyway.

I did not know this; thanks!

Not sure where you're going with this.  It seems to me that political methods (such as petitions, public pressure, threat of legislation) can be used to restrain the actions of large/mainstream companies, and that training models one or two OOM larger than GPT4 will be quite expensive and may well be done mostly or exclusively within large companies of the sort that can be restrained in this sort of way.

I was figuring GPT4 was already trained on a sizable fraction of the internet, and GPT5 would be trained on basically all the text (plus maybe some not-text, not sure).  Is this wrong?

It may not be possible to prevent GPT4-sized models, but it probably is possible to prevent GPT-5-sized models, if the large companies sign on and don't want it to be public knowledge that they did it.  Right?

Oh no.  Apparently also Yann LeCun didn't really sign this.

https://twitter.com/ylecun/status/1640910484030255109

As a personal datapoint: I think the OPs descriptions have a lot in common with how I used to be operating, and that I think this would have been tremendously good advice for me personally, both in terms of its impact on my personal wellness and in terms of its impact on whether I did good-for-the-world things or harmful things.

(If it matters, I still think AI risk is a decent pointer at a thingy in the world that may kill everyone, and that this matters.  The "get sober" thing is a good idea both in relation to that and broadly AFAICT.)

Nope, haven't changed it since publication.

I like this observation.  As a random note, I've sometimes heard people justifying "leave poor working conditions in place for others, rather than spending managerial time improving them" based on how AI risk is an emergency, though whether this checks out on a local consequentialist level is not actually analyzed by the model above, since it partly involves tradeoffs between people and I didn't try to get into that.

I sorta also think that "people acting on a promise of community and support that they later [find] [isn't] there" is sometimes done semi... (read more)

I can think of five easily who spontaneously said something like this to me and who I recall specific names and details about.  And like 20 more who I'm inclined to project it onto but there was some guesswork involved on my part (e.g., they told me about trouble having hobbies and about feeling kinda haunted by whether it's okay to be "wasting" their time, and it seemed to me these factors were connected, but they didn't connect them aloud for me; or I said I thought there was a pattern like this and they nodded and discussed experiences of theirs bu... (read more)

If you get covid (which many of my friends seem to be doing lately), and your sole goal is to minimize risk of long-term symptoms, is it best to take paxlovid right away, or with a delay?

My current low-confidence guess is that it is best with a delay of ~2 days post symptoms.  Would love critique/comments, since many here will face this sometime this year.

Basic reasoning: anecdotally, "covid rebound" seems extremely common among those who get paxlovid right away, probably also worse among those who get paxlovid right away.  Paxlovid prevents vira... (read more)

Maybe.  But a person following up on threads in their leisure time, and letting the threads slowly congeal until they turn out to turn into a hobby, is usually letting their interests lead them initially without worrying too much about "whether it's going anywhere," whereas when people try to "found" something they're often trying to make it big, trying to make it something that will be scalable and defensible.  I like that this post is giving credit to the first process, which IMO has been historically pretty useful pretty often.  I'd also ... (read more)

I appreciate this comment a lot.  Thank you.  I appreciate that it’s sharing an inside view, and your actual best guess, despite these things being the sort of thing that might get social push-back!

My own take is that people depleting their long-term resources and capacities is rarely optimal in the present context around AI safety.

My attempt to share my reasoning is pretty long, sorry; I tried to use bolding to make it skimmable.

In terms of my inside-view disagreement, if I try to reason about people as mere means to an end (e.g. “labor”):

0. &nb... (read more)

I think this is a solid point, and that pointing out the asymmetry in evolutionary gradients is important; I would also expect different statistical distributions for men and women here.  At the same time, my naive ev psych guess about how all this is likely to work out would also take into account that men and women share genes, and that creating gender-specific adaptations is actually tricky.  As evidence: men have nipples, and those nipples sometimes produce drops of milk.

Once, awhile ago and outside this community, a female friend swore me to... (read more)

A bunch of people have told me they got worse at having serious/effortful intellectual hobbies, and at "hanging out", after getting worried about AI.  I did, for many long years.  Doesn't mean it's not an "excuse"; I agree it would be good to try to get detailed pictures of the causal structure if we can.

In fairness, a lot of these things (clothes, hairstyles, how "hard core" we can think we are based on working hours and such) have effects on our future self-image, and on any future actions that're mediated by our future self-image.  Maybe they're protecting their psyches from getting eaten by corporate memes, by refusing to cut their hair and go work there.

I suspect we need to somehow have things less based in self-image if we are to do things that're rooted in fresh perceptions etc. in the way e.g. science needs, but it's a terrifying transition.

I may have been primed to interpret this post in those terms too much, because I perceived it to be a reaction to Eliezer's recent doomy-sounding blog posts (and people worrying about AI more than usual recently because of that, plus ML news, plus various complicated social dynamics), trying to prevent the community from 'going too far' in certain directions. ... But it sounds like I may be imposing context on the post that isn't the way you were thinking about it while writing it.

Oh, yeah, maybe.  I was not consciously responding to that.  I was... (read more)

I make this point not to argue against finding love or starting a family, but to argue against a mindset that treats AGI and daily life as more or less two different magisteria….

It still doesn't feel to me like it's fully speaking as though the two worlds as one world

The situation is tricky, IMO.  There is, of course, at the end of the day only one world.  If we want to have kids who can grow up to adulthood, and who can have progeny of their own, this will require that there be a piece of universe hospitable to human life where they can do... (read more)

A friend emailed me a comment I found helpful, which I am copying here with their permission: 

"To me [your post] sounded a bit like a lot of people are experiencing symptoms similar to ADHD: both becoming hyperfocused on a specific thing and having a lot of habits falling apart. Makes sense conceptually if things labeled as emergencies damage our attention systems. I think it might have to do with a more general state of stress/future-shock where people have to go into exception-handling mode more often. As exceptions become normalized the s... (read more)

But it still feels to me like it's a post trying to push the pendulum in a particular direction, rather than trying to fully and openly embody the optimal-by-your-lights Balancing Point.

AFAICT, I am trying to fully and openly embody the way of reasoning that actually makes sense to me in this domain, which… isn’t really a “balancing point.”  It’s more like the anarchist saying “the means are the ends.”  Or it’s more like Szilard’s “ten commandments,” (which I highly recommend reading for anyone who hasn’t; they’re short).  Or more like... (read more)

I want to have a dialog about what’s true, at the level of piece-by-piece reasoning and piece-by-piece causes.  I appreciate that you Rob are trying to do this; “pedantry” as you put it is great, and seems to me to be a huge chunk of why LW is a better place to sort some things out than is most of the internet.

I’m a bit confused that you call it “pedantry”, and that you talk of my post as trying to push the pendulum in a particular way, and “people trying to counter burnout,” and whether this style of post “works” for others.  The guess I’m formi... (read more)

Yes!  I am really interested in this sort of dynamic; for me things in this vicinity were a big deal I think.  I have a couple half-written blog posts that relate to this that I may manage to post over the next week or two; I'd also be really curious for any detail about how this seemed to be working psychologically in you or others (what gears, etc.).  

I have been using the term "narrative addiction" to describe the thing that in hindsight I think was going on with me here -- I was running a whole lot of my actions off of a backchain from a... (read more)

I agree that many of those who decide to drop everything to work on AI expect AI sooner than that.  (Though far from all.)

It seems to me though that even if AI is in fact coming fairly soon, e.g. in 5 years, this is probably still not-helpful for reducing AI risk in most cases, compared to continuing to have hobbies and to not eat one's long-term deep interests and spiritual health and ability to make new sense of things.

Am I missing what you're saying?

One substantive issue I didn’t manage to work into the OP, but am interested in, is a set of questions about memetics and whether memetics is one of the causes of how urgent so many people seem to find so many causes.

A section I cut from the OP, basically  because it's lower-epistemic-quality and I'm not sure how relevant it is or isn't to the dynamics I kept in the OP, but that I'd like to throw into the comments section for discussion:
--

Memetics sometimes leads to the amplification of false “emergencies”

Once upon a time, my former housemate Steve Ra... (read more)

I wrote: 
>... Also, I am excited about people trying to follow paths to all of their long-term goals/flourishing, including their romantic and reproductive goals, and I am actively not excited about people deciding to shelve that because they think AI risk demands it. 

Justis (amid a bunch of useful copy-editing comments) said he does not know what "actively not excited" is supposed to mean, and suggested that maybe I meant "worried."  I do not mean "worried", and do mean "actively not excited": when people do this, it makes me less excited by and hopeful about the AI risk movement; it makes me think we're eating our seedcorn and have less here to be excited about.

There’s a lot I want to try to tell LessWrong about.  A lot of models, perceptions, thoughts, patterns of thinking.  It’s been growing and growing for me over the last several years.

A lot of the barrier to me posting it has been that I am (mostly unendorsedly) averse to publishing drafts that’re worse than my existing blog posts, or that may not make sense to people, or that talk about some things without having yet talked about other things that I care more about, or etc.  This aversion seems basically mistaken to me because “trial and erro... (read more)

Justis and Ruby made a bunch of good substantive comments on my draft, and also Justis made a bunch of very helpful typo-fixes/copy-editing comments on my draft.

I fixed the copy-editing ones but mostly did not respond to the substantive ones, though I liked them; I am hoping some of that discussion makes its way here, where it can happen in public.

> I've thought a bit about ideas like this, and talked to much smarter people than myself about such ideas - and they usually dismiss them, which I take as a strong signal this may be a misguided idea. 

I honestly don’t know whether slowing down AI progress in these ways is/isn’t a good idea.  It seems plausibly good to me.  I do think I disagree about whether the “much smarter people”s dismissal of these ideas is a strong signal.

Why I disagree about the strong signal thing:

I had to push through some fear as I wrote the sentence about it s... (read more)

I didn't follow CFAR that closely, so I don't know how transparent you were that this was a MIX of rationality improvement AND AI-Safety evangelism.

How transparent we were about this varied by year.  Also how much different ones of us were trying to do different mixes of this by different programs varied by year, which changed the ground truth we would've been being transparent about.  In the initial 2012 minicamps, we were part of MIRI still legally and included a class or two on AI safety.  Then we kinda dropped it from the official stuff,... (read more)

Teaching rationality looks more similar to AI capabilities research than AI alignment research to me.
 

I love this question.  Mostly because your model seems pretty natural and clear, and yet I disagree with it.

To me it looks more like AI alignment research, in that one is often trying to align internal processes with e.g. truth-seeking, so that a person ends up doing reasoning instead of rationalization.  Or, on the group level, so that people can work together to form accurate maps and build good things, instead of working to trick each oth... (read more)

I mean, that kind of is the idea in Eliezer's post "Schools proliferating without evidence," from two years before CFAR was founded.

(Minus the "so we stop" part.)

"Anti-crux" is where the two parties who're disagreeing about X take the time to map out the "common ground" that they both already believe, and expect to keep believing, regardless of whether X is true or not.  It's a list of the things that "X or not X?" is not a crux of.  Often best done before double-cruxing, or in the middle, as a break, when the double-cruxing gets triggering/disorienting for one or both parties, or for a listener, or for the relationship between the parties.

A common partial example that may get at something of the spirit o... (read more)

Thanks for weighing in; I trust these conversations a lot more when they have multiple people from current or former CFAR.  (For anyone not tracking, Unreal worked at CFAR for awhile.)  (And, sorry, I know you said you're mainly writing this to not-me, but I want to engage anyhow.)
 

The hypotheses listed mostly focus on the internal aspects of CFAR.

This may be somewhat misleading to a naive reader. (I am speaking mainly to this hypothetical naive reader, not to Anna, who is non-naive.) 

.... It's good FOR CFAR to consider what the org cou

Also:

• The egregores that are dominating mainstream culture and the global world situation are not just sitting passively around while people try to train themselves to break free of their deeply ingrained patterns of mind. I think people don't appreciate just how hard it is to uninstall the malware most of us are born with / educated into (and which block people from original thinking). These egregores have been functioning for hundreds of years. Is the ground fertile for the art of rationality? My sense is that the ground is dry and salted, and yet we stil

I suspect we need to engage with politics, or with noticing the details of how rationality (on group-relevant/political topics) ends up in-practice prevented in many groups, if we want to succeed at doing something real and difficult in groups (such as AI safety).

Is this what you mean?

One of the big modeling errors that I think was implicit in CFAR through most of its history, was that rationality was basically about making sure individuals have enough skill for individual reasoning, rather than modeling it as having a large component that is about resisti... (read more)

CFAR, to really succeed at what I see as its mission (bring rationality to the masses), needed...

IMO (and the opinions of Davis and Vaniver, who I was just chatting with), CFAR doesn't and didn't have this as much of its mission.

We were and are (from our founding in 2012 through the present) more focused on rationality education for fairly small sets of people who we thought might strongly benefit the world, e.g. by contributing to AI safety or other high-impact things, or by adding enrichment to a community that included such people.  (Though with th... (read more)

and relatedly at some point I got a doomy sense about CFAR after inquiring with various people and not being able to get a sense of a theory of change or a process that could converge to a theory of change for being able to diagnose this and other obstacles.

Can you say a bit more about what kind of a "theory of change" you'd want to see at CFAR, and why/how?  I still don't quite follow this point.

Weirdly, we encountered "behaviors consistent with wanting fancy indirect excuses to not change" less than I might've expected, though still some.  This... (read more)

I mean... "are you making progress on how to understand what intelligence is, or other basic foundational issues to thinking about AI" does have somewhat accessible feedback loops sometimes, and did seem to me to feed back in on the rationality curriculum in useful ways.

I suspect that if we keep can our motives pure (can avoid Goodhardting on power/control/persuasion, or on "appearance of progress" of various other sorts), AI alignment research and rationality research are a great combination.  One is thinking about how to build aligned intelligence i... (read more)

I wish we had.  Unfortunately, I don't think we did much in the way of pre-portems on our long-term goals, unless I'm forgetting something.  (We discussed "what if CFAR doesn't manage to become financially able to keep existing at all", and "what if particular workshops can't be made to work," but those are shorter term.)  Eliezer's sequence "The craft and the community" was written before CFAR, but after he wanted an independent rationality community that included rationality training, so we could try to compare what happened against that.

Mostly wanted to say that even though CFAR got maybe "less far" than hoped for, in my view it actually got quite far.

I agree CFAR accomplished some real, good things.  I'd be curious to compare our lists (and the list of whoever else wants to weigh in) as to where CFAR got.

On my best guess, CFAR's positive accomplishments include:

• Learning to run workshops where people often "wake up" and are more conscious/alive/able-to-reflect-and-choose, for at least ~4 days or so and often also for a several-month aftermath to a lesser extent;
• Helping a bunch of peo

The even broader context is Bay area is currently the best place in the world for production of memeplexes, influence-seeking patterns, getting money for persuasion, etc., which implies it is likely a great place where world would benefit from someone teaching rationality, but maybe not the best place for developing the skills.

Thanks for mentioning this.  I think this had a big effect.

Fair enough.  FWIW, I found the movie good / full of useful anecdata for piecing together a puzzle that I personally care a lot about, and so found it rewarded my four hours, but our interests are probably pretty different and I know plenty who would find it empty and annoying.

On reflection, I shouldn’t have written my paragraph the way I did in my parent comment; I am not sure what trouble something-like-every self-help thingy has run into, I just suspect there’re threads in common based on how things look.  I might be wrong about it.

Still, I wr... (read more)

I'd like to upvote reading Val's linked post, if someone's wondering whether to bother reading it and likes my opinions on things.

Sorry.  I don't have a good short description of the problem, and so did not try to say explicitly what I meant.  Instead I tried to refer to a 4-hour film, "Century of the self," as trying to describe the same problem.

I may come back later with an attempted description, probably not a good one.

And a second afterthought:

I think for a long time CFAR was trying, though maybe not in a very smart/calibrated/wise/accurate way, to have public relationship with "the rationality community" along the lines of "we will attempt this project that you guys care about; and you guys may want to collaborate with us on that."  (Details varied by year; I think at the beginning something like this was more intended, accurate, and sincere, but after awhile it was more like accumulated branding we didn't mean but didn't update.)

I think at the moment we are not t... (read more)

I think the conclusion I take from it is ~"There's a bunch of individual people who were involved with CFAR still doing interesting stuff, but there is no such public organisation anymore in a meaningful sense (although shards of the organisation still help with AIRCS workshops); so you have to follow these individual people to find out what they're up to. Also, there is no concentration of force working towards a public accessible rationality curriculum anymore."

This seems about right to me personally, although as noted there is some network / concentrati... (read more)