Thanks for writing this piece; I think your argument is an interesting one.
One observation I've made is that MIRI, despite its first-mover advantage in AI safety, no longer leads the conversation in a substantial way. I do attribute this somewhat to their lack of significant publications in the AI field since the mid-2010s, and their diminished reputation within the field itself. I feel like this serves as one data point that supports your claim.
I feel like you've done a good job laying out potential failure modes of the current strategy, but it's not a slam dunk (not that I think it was your intention to write a slam dunk as much as it was to inject additional nuance to the debate). So I want to ask, have you put any thought into what a more effective strategy for maximizing work on AI safety might be?
Thanks for writing this up! We tried this out in our group today and it went pretty well :-)
Because our venue didn't have internet I ended up designing and printing out question sheets for us to use (google docs link). Being able to compare so many responses easily, we were able to partner up first and find disagreements second, which I think was overall a better experience for complete beginners. The takes that you were most polarized on with any random person weren't actually that likely to be the ones that you feel the most strongly about, and there were generally a few options to choose from. So we got a lot of practice in with cruxing without getting particularly heated. I'd like to find a way to add that spice back for a level 2 double crux workshop, though!
We repurposed using the showing fingers for agreement/disagreement for coming up with custom questions; we had quite a few suggestions but only wrote down the ones that got a decent spread in opinion. This took a while to do, but was worth it, because I was actually really bad at choosing takes that would be controversial in the group, and people were like "wtf Jenn how can we practice cruxing if we all agree that everything here is a bunch of 3s." (slightly exaggerated for effect)
I didn't realize this until I was running the event, but this write-up was really vague on what was supposed to happen after step 3! I ended up referencing this section of the double crux post a lot, and we ended up with this structure:
We did two rounds in total. People unfortunately did not report that the second round was generally easier than the first, but seemed to overall find the workshop a valuable experience! One person commented that it led to much more interesting conversation than most readings-based meetups, and I'm inclined to agree.
The question is rather, what qualities do EAs want themselves and the EA movement to have a reputation for?
Yes, I think this is a pretty central question. To cross the streams a little, I did talk about this a bit more in the EA Forums comments section: https://forum.effectivealtruism.org/posts/5oTr4ExwpvhjrSgFi/things-i-learned-by-spending-five-thousand-hours-in-non-ea?commentId=KNCg8LHn7sPpQPcR2
I get a sense that the org is probably between 15 and 50 years old
Yep, close to the top end of that.
It's probably been through a bunch of CEOs, or whatever equivalent it has, in that time. Those CEOs probably weren't selected on the basis of "who will pick the best successor to themselves". Why has no one decided "we can help people better like this, even if that means breaking some (implicit?) promises we've made" and then oops, no one really trusts them any more?
That's a really great observation. Samaritans has chosen to elide this problem simply by having no change in leadership throughout the entire run of the organization so far. They'll have to deal with a transition soon as the founders are nearing retirement age, but I think they'll be okay; there are lots of well aligned people in the org who have worked there for decades.
Have they had any major fuck ups? If so, did that cost them reputationally? How did they regain trust?If not, how did they avoid them? Luck? Tending to hire the sorts of people who don't gamble with reputation? (Which might be easier because that sort of person will instead play the power game in a for-profit company?) Just not being old enough yet for that to be a serious concern?
Have they had any major fuck ups? If so, did that cost them reputationally? How did they regain trust?
If not, how did they avoid them? Luck? Tending to hire the sorts of people who don't gamble with reputation? (Which might be easier because that sort of person will instead play the power game in a for-profit company?) Just not being old enough yet for that to be a serious concern?
They haven't had any major fuck ups, and there's two main reasons for that imo:
[reputation and popularity] probably have overlapping causes and effects, but they're not the same.
I'm inclined to think that this is a distinction without a difference, but I'm open to having my mind changed on this. Can you expand on this point further? I'm struggling to model what an organization that has a good reputation but is unpopular, or vice versa, might look like.
If EA as a whole is unpopular, that's also going to cause problems for well-reputed EA orgs.
Yes, I think that's the important part, even though you're right that we can't do much about individual orgs choosing to associate itself with EA branding.
I share your sense that EAs should be thinking about reputation a lot more. A lot of the current thinking has also been very reactive/defensive, and I think that's due both to external factors and to the fact that the community doesn't realize how valuable an actually good reputation can be - thought Nathan is right that it's not literally priceless. Still, I'd love to see the discourse develop in a more proactive position.
Thanks for your super thought out response! I agree with all of it, especially the final paragraph about making EA more human-compatible. Also, I really love this passage:
We can absolutely continue our borg-like utilitarianism and coldhearted cost-benefit analysis while projecting hospitality, building reputation, conserving slack, and promoting inter-institutional cooperation!
Yes. You get me :')
I don't think the answer is super mysterious; a lot of people are in the field for the fuzzies and it weirds them out that there's some weirdos that seem to be in the field, but missing "heart".
It is definitely a serious problem because it gates a lot of resources that could otherwise come to EA, but I think this might be a case where the cure could be worse than the disease if we're not careful - how much funding needs to be dangled before you're willing to risk EA's assimilation into the current nonprofit industrial complex?
The meeting rooms are in the basement! If you come in through the main entrance, do a U turn to the left of the vestibule and go down the stairs. It'll be the first door to your right
Sort of related, everything studies wrote this essay in 2017 and now "wamb" is a term that my friends and I use all the time.