[Epistemic status: Low confidence speculations]
I notice some comments are suggesting that the rationalist hide their weirdness. I want to point out that this is only possible if you separate the rationalist into a separate co-working space away from the professionals, and only have the tow groups interact occasionally. This may or may not be the correct trade-off.
Anecdote: LISA (the main London AI Safety coworking space) used to be at WeWork for a while, before they got their own space. I was there a few times during this era. The LISA organizers got constant complains from WeWork because the AI Safety people didn't behave "professional" enough. And this was a situation where we didn't even try to work together, or interact with other groups renting desks in the same building. The main complaints I remember was that the rationalist would not ware shoes, and take napps in the common sofas.
A lot of rationalists are autistic, and/or gender-non-conforming, or just don't want to where shoes all day. Some of us are never going to appear normal. Some of us can mask for a day, but not all day every day. And a few (probably) could adopt and fit in anywhere.
So far I've only talked about physical appearance. But this also apply to interest, communication style, etc.
So maybe the solution is not to have one space, but to have two spaces, but visit each other? And for everyone to be mindful that when your visiting you're a guest in a different culture?
The theory is that it easier to learn to appreciate a florigen culture, if you don't have to put up with it too much. Because being out of your element constantly is draining, and leave you with fewer spoons.
What you have now is (maybe?) that the tow groups are constantly exposed to each other, are constantly low grade fed up with each other, and therefore mostly self segregating, while in the same office. But maybe it would be better to have actual separation most of the time, and regular deliberate deep engagement?
Suggestion: If your space have more than one floor, or more than one room, you can have part of the space being rat space and part be policy professional space. The rules are that you can visit, but if you do so, you should try to conform to the space you are visiting. Expect on Fridays, when everyone is encourage to mingle, or something like that.
1I'm 63 years old, an ex-Google-Senior-Staff software engineer who's now an independent AI safety researcher, and I've also been reading/involved in EA/MIRI for about 15 years. So I'm solidly in both camps, and I don't really see the conflict.
On the making EA assumptions and terminology comprehensible to professionals side, I'd suggest a weekly reading-and-discussion group focusing on some of the more famous/influential/thought-provoking LW posts over the past 5 years or so, and on how things have moved on since they were written. (Note that a lot of the stuff written before ChatGPT/the rise of LLMs as the most likely AGI technology need to be contextualized a bit.)
some people said things like "I feel existential terror at the thought that maybe everyone I love and also our entire species is entirely wiped out in ten years' time" and other people said things like "I am feeling extremely inspired by the amount of driven and intelligent people in the room"
It's entirely possible for both of these things to be true at the same time: that we're facing a huge, terrifying existential risk that many ordinary people are unaware of, yet that we're happy that at least a lot of smart, motivated people are working on trying to reduce it.
2One of my current projects is to try to x-risk-pill a bunch of academic mathematicians, I think you're pointing to a relevant dynamic that I'm just figuring out and trying to navigate. In particular out of the ~10 people who I have successfully pilled in person, there's a visible inclination with almost all of them to move in what you call the "professional" direction.
I've been told explicitly, for example, that "for this project to go bigger you should avoid the LessWrong brand like the plague." I'm currently confused if this is an incorrect personal bias from that person, a narrowly correct statement about messaging to academics, or a broadly correct statement about messaging to the general public.
1Fortunately for AI safety, the smart policy person who wants to work on compute governance or export controls isn't proposing the AI-safety equivalent of a donkey sanctuary.
This seems obviously false; as two examples, accelerating race dynamics and ignoring the plausibility of a need to stop AI development entirely are typical.
2I feel like a good frame for me has been "just point out when people are wrong very frequently" rather than thinking about it in terms of a culture thing. If people have wrong beliefs or don't take things seriously that should be taken seriously (e.g. space property rights), just tell them. Reasonable people update on reasonable arguments.
2I find that in general, if one does not put in the work of establishing rapport, building common understanding, and demonstrating understanding the other party's frameworks, there is no reason for the other party to take one's disagreements seriously. This makes sense; even in very tightly connected sub-fields of professional industries, established frameworks/conventions/world models are different enough that it's often reasonable to say "yeah that might make sense inside your community of practice, but it is outside the overton window/will not have any traction/is fnord shaped over here".
I agree that reasonable people update on reasonable arguments, and I think professionals tend to be reasonable people. I do not think there is a short[1], clear argument for why space property rights (or, indeed, GCR from AI) should be taken seriously.
- ^
as in, you can explain it in two sentences in a casual conversation.
1I'm probably more of a rationalist, but I suspect one thing rationalist types can do that might help somewhat is tone down the extent to which we broadcast the the unorthodox aspects of the culture which aren't directly linked to AI safety.
E.g. don't be too eager to discuss stuff like polyamory/meditation/board games/drugs/meal replacement/overt analysis of social status+signalling games etc.
1I think your "in the closet" framing is wrong.
In some sense all of us are somewhat "in the closet" all the time - there is almost never a context in which you interact with other people without the need to heavily regulate your behavior according to a complex web of norms and taboos. There's almost never a context where you shouldn't be putting effort into modifying what you say and do in service of being perceived more positively. Even when you're hanging out with rationalists.
Imagine you and I were both in the same group social interaction and I started openly sharing any piece of personal information about myself or broadcasting every thought that crossed my mind.
This is an exageration, but to get the point across: imagine we were in a semi-professional setting and I started asking people if I can eat their dog's corpse after it passes away, or start talking about the effectiveness of different suicide methods unprompted, or show people pictures I took of my last bowel movement.
I'm very confident you would instinctivly find my behavior repulsive, and instinctively try to socially punish that behavior and enforce norms/taboos. As you should!
I'm even confident you'd instinctively act to socially punish me (or anyone else) for minor things like laughing too loud or chewing with my mouth open, or even for talking to much about topics which you weren't interested in.
If I responded to that by complaining that you want me to be "in the closet" or "not have my own culture" that would be very lame objection.
In my view there's a common mistake people make where they make valid observations like:
"people should on the margin be more tolerant of differences"
"it's not good when someone is treated like a pariah because of their race/sexuality"
"bulying is bad"
And incorrectly generalize that to some notion like "Groups of people don't need to regulate each other's behavior in any way and I shouldn't have to put any work into managing the way others percieve me"
That's just not how people work! (And don't mean "that's not how normies work", it's not how you work either!)
It's completely unavoidable that you have to modify your words and actions based on who you're communicating with, and you already do it intuitively to a massive extent
1beg your pardon, how did board games got on this list?
(that was an actual question, do you have a story how your/someone's love for board games burned you in some community? ...are we talking DnD or Settlers of Catan? am I too sheltered when I am part of some local software and queer communities as opposed to other groups?)
1I agree it's probably less of an obvious "unusual especially rationalist" thing than other stuff in the list.
Not saying it's necessarily a bad thing to discuss in general either! There's nothing wrong with mentioning that you enjoy playing Catan with your friends or whatever. I suppose in my head I'm more imagining like... someone being mildly put off when someone makes an analogy to the Magic cards color wheel when talking about different attitudes of AI labs or something.
And again it's not like it's that bad to be someone who's really into board games either! If the only obvious difference between rationalists and non-rationalists was a slight tendency towards more complex board games this wouldn't matter. But it's amplified by the large number of other eccentricities.
So it's not that board games are especially aversive to anyone or anything like that, it's just maybe it's sometimes better to de-ephasize the ways in which the professionals are different to the rationalists - and from cost/benefit angle it's better to de-ephasize the trivial differences (like an appreciation for tungsten cubes) than to de-emphasize weird stuff that might have material importance (like a willingness to discuss abstract moral philosophy or assign probabilities to beliefs)
1The responsibility probably rests on the rationalists here. People become professionals all the time (and indeed, maximizing impact often requires being a professional), but well under 1% of the population has the personality to ever become a rationalist.
Also, it is not necessary to be a rationalist; the 80/20 which many professionals can do is to understand x-risk and take it seriously, which is getting easier and easier.
4Communication and understanding can never rest on one side, because that will just fail.
People become professionals all the time
A lot of professional culture is neurotypical culture, and a lot of rat culture is autism culture, and people don't just change neurotype. That's obliviously not all of it, there are learned parts too. But for some people it is easy to become a rationalist and hard to become a passing professional.
Also, it is not necessary to be a rationalist; the 80/20 which many professionals can do is to understand x-risk and take it seriously, which is getting easier and easier.
I think this line misses the point of the post? I agree that it is not necessary for everyone to become a rationalist. And I agree that understanding x-risk is important. But that is not all it takes to work together. That also required mutual understanding and respect. If every conversation with the other group feels weird and off-putting, that is a problem. I
1Appreciate the corrections! Taking notes for when I start conducting field research :p
I agree that it would be weird to bag every single profession together and speak of them in general terms. To clarify where I'm coming from, I find that the professionals who are transferring into AI safety largely come from a few subfields that all seem interrelated: consulting, public service policy, procurement, nonprofit advocacy. While these subfields all have their own unique norms as well, I think it's reasonable to look at this ~PMCish collection and speak of it in a collective.
...I sort of consider programmers to already be fairly rationalist by default, and find bridging inferential gaps with them to be generally less challenging.
2I think "professionals" here is acting as a polite euphemism for "normies". AI safety is increasingly attracting normies.
1I think the word normie emits more heat than light, and also is underspecified here. The cluster that I'm interested in engaging with are university educated, generally non-STEM white collar professionals who are used to working in offices, and decently far along in their careers.
I acknowledge that many rationalists think of that group as the prototypical "normies"! However they actually make up not that large a percentage of the total population, even in developed nations.
1
At the local AI safety co-working space, there are ~two kinds of regulars.
There's the kind of regular who's been thinking seriously about AI safety and alignment since pre-2022, who have passing to intimate familiarity with the funding ecosystem, the Sequences, and various conferences that happen at Lighthaven. Let's call them rationalists.
Then there's the kind of regular who comes in with many years of impressive industry or government experience, who realized in the last few years that it is important and worthwhile to pivot their career towards making sure that this AI thing is handled competently by the people in power, and who have many valuable skills, insights, and connections that are lacking in rationalist culture. Let's call them professionals.
There are, of course, many people who are somewhere in between - bright undergrads born this millennium who have been involved in EA since stumbling upon 80k hours in high school, professionals who previously identified as EA but drifted out of the scene a few years ago, founders who have idly read some Scott Alexander. But let's call it a dichotomy for now.
There's a large culture gap between the rationalists and the professionals. Robust mutual understanding seems important if we want to work together on this project of not having AI blow up our civilization, and it is only by working together that there is a chance that we might succeed.
Unfortunately, there seems to be an underlying assumption that all you really need to do is stick the two groups in the same room for long enough and the acculturation will happen by default. This seems wrong to me; my experience so far is that there are many unspoken assumptions that are held by the two groups that sort of never come up in conversation, except in weird awkward eruptions when a fundamental assumption is violated. When that happens, the risk is that it pulls the two groups further apart, instead of closer together.
Some examples from recent events and discussions:
But also, like, timelines, so it seems useful to ask: what might deliberate acculturation look like?
Here are some ideas for how to more deliberately bridge fundamental culture gaps:
Having written this list out, I am dissatisfied with it because I'm not actually that confident that doing all those things will result in the degree of trust and understanding that I think is necessary for us to do good work together.
What's the actual tension? Perhaps it's that it seems to me like the culturally rationalist are the gatekeepers of the funding resources for work on catastrophic risks, and this feels like a thing that is unsayable and somewhat anti-inductive (in that to explain it is to give an answer key to what funders want to hear, which is not ideal).
Recently a big tent animal welfare conference happened in town and I hosted a mixer for the EAs in attendance (around a third of the ~500-person conference attendees were EA). One of them, a long-established EA, gave me an interesting rundown of the wider animal welfare funding ecosystem. They said that EAs used to not be that accepted in the wider animal welfare movement, but as it became common knowledge that EAs control a large portion of the funding, the community began to embrace them more and adopt more EA frameworks and ways of thinking - but not without some amount of grumbling and resentment and bad feeling in at least some contingents. If you're not EA pilled you will simply resent the fact that the EAs are fundamentally not interested in funding the obviously morally important work of running your local donkey sanctuary, and there's literally nothing you can do about it. And if you are running a local donkey sanctuary you are sort of by definition not EA pilled.
Fortunately for AI safety, the smart policy person who wants to work on compute governance or export controls isn't proposing the AI-safety equivalent of a donkey sanctuary. And unlike the donkey sanctuary owner they have leverage in things rationalists can't buy at any price - things like institutional legibility, standing relationships, tacit knowledge about how to make key organisations do things. And perhaps the problem will also partially solve itself through funder pluralism; the rationalists are not literally the only people on the planet interested in funding alignment research and policy.
Still, I don't want to overstate the case for optimism.
Funder pluralism might result in parallel communities that don't collaborate in the same rooms at all. Animal welfare maybe only avoided this outcome because EA funding was a very significant portion of all funding, but it seems like what the rationalists are gatekeeping is one specific flavour of alignment funding.
Plenty of things professionals consider serious AI work (bias, misinformation, labor displacement, near-term harms) are donkey sanctuaries to at least some parts of the rationalist funding ecosystem, and perhaps it would be useful to make that common knowledge, but that legibility will come at the cost of resentment.
In animal welfare, EAs were newcomers who bought influence into an existing movement. In AI safety, rationalists are more likely to be ~founders, which tilts the balance even more in their favour, which brings with it a deeper (more resentment-building/polarizing) form of gatekeeping.
Bad near-term equilibria that might result (/maybe are already starting to appear) include:
I'm squarely in the rationalist camp, which means that when I see a problem my default response is to write a post and publish it publicly. A skilled professional, on the other hand, would solve problems like these in a way that never produces a public artifact, and because I can't read about how they did it, I can't model what they'd do.
I'm cognizant of the fact that I'm holding only half the puzzle pieces here. If you identify more as a professional than as a rationalist, and reading this post made you feel some type of way, please come find me and give me the other half of the puzzle.