Common mistakes in AI safety group organizing

Nikola Jurkovic

Common mistakes in AI safety group organizing — LessWrong

177 Common mistakes in AI safety group organizing

19th Sep 2026

2 min read

177

Back in the day, I was a very active AI safety group organizer. I commonly notice people making the same mistakes across many clubs. I have written down a list of some of these mistakes hoping people will avoid them in the future:

Reading groups often require that people read things before meetings. This is a mistake. People often don't do the readings. And the lack of common knowledge that everyone has read the reading degrades the conversation quality.
- Instead, have longer meetings, serve food (so, lunch/dinner meeting slots), and read during the actual meeting.
Reading groups often don't sort people into cohorts properly. Mainly, they fail at clustering people into clusters of roughly equal ML knowledge and age. Grad students don't want to discuss a paper with freshmen. People with lots of ML knowledge don't want to discuss a paper with people with no ML knowledge.
- Instead, group people with people similar to them in ML knowledge and age.
Reading groups often rely on digital materials instead of physical printouts. Screens are distracting and there is no common knowledge that people are paying attention.
- Neatly print every reading ahead of time instead.
Clubs don't gatekeep enough. There exist many people whose presence is net-negative to events. Those people should not be accepted to the club or invited to its events.
- Instead, get signal on people through application forms and word-of-mouth. If people seem like their presence wouldn't make the club better, don't include them. Also, invest in people (through helping them upskill and expecting them to contribute to high-context conversations) such that they can become the types of people who can contribute. A lot of what it means to build a community is knowing how to gatekeep it properly.
Organizers aren't knowledgeable enough. People can notice if you're well-read and have good takes. If you're not knowledgeable, they will take you less seriously and you will be worse at understanding the world and helping others understand it.
- Instead, foster a culture where organizers (especially the main ones) need to be worthy of being role models for other students. They should be doing quality work and thinking about and talking about AI safety.
Organizers don't have serious vibes. Unserious vibes drive people away and make people think that the organizers don't take the actual issues seriously.
- AI safety is not just another thing to form a club around. It's a serious topic and people who are teaching others about it should act like it. It should feel much more like a pre-professional club (e.g. an ML reading group, or a consulting club) and less like a social club (e.g. a board game club).
Organizers aren't close enough friends. If the organizers don't know each other well enough and don't get along well enough, the club will function much worse because of friction, and outside people will find the club less attractive.
- Instead, treat "the organizers are friends" as one of the main goals of the club.

Thank you, group organizers, for your work on helping others contribute to the field. I hope that you will meet the current moment with the seriousness it deserves.

Frontpage

177

Common mistakes in AI safety group organizing

New Comment

8 comments, sorted by

top scoring

Click to highlight new comments since: Today at 11:03 AM

[-]carolanne1mo*1310

some related thoughts:

re: Clubs don't gatekeep enough: I think this applies especially to organizers / "board members". It is very costly to remove an organizer whose presence is net-negative to the club & I think people should be very thoughtful about on boarding new board members etc.

Related: it feels natural to model an AIS club after all other types of uni-clubs, and in my observation this pulls people toward recruiting for specialized roles ("graphic design," "outreach," etc). I think this is often a mistake. I think the standard club frame treats a filled org chart as the goal (/a large part of the goal) , which should not be the goal of your AIS group. I think its important to internalize the idea of the club as a single unit with one ultimate goal (eg. get more ppl into useful roles in AIS or whatever) and to understand all other tasks strictly as means to that end, and to be ready to cut off any task / role / person that is no longer in service of that goal. (I also feel that habryka's post Do not hand off what you cannot pick up is very relevant to the context of running uni groups; I think the organizers should have the skills to do all of the subtasks of the club)

- also i think its good to internalize that a large club is not intrinsically valuable & having more board members comes with costs (eg. more management overhead, potential for internal politics, meetings with more people are often less productive / harder to run, clubs with more board members are often harder to steer, etc)

re: organizers aren't knowledgeable enough: I think running a bad reading group (eg. due to insufficient organizer bandwidth, low context organizer, bad curriculum even, etc) is possibly worse than no reading group, as the pool of people who'll attend an AIS reading group in a given population is finite, and if they have a bad experience they're not coming back again.

also, re "People can notice if you're well-read and have good takes." I think its important to not accidentally select for people do not notice that you're not well read & don't have good takes / people that are satisfied & 'convinced' by a much simplified / bad explanations

[-]JenniferRM1mo51

It seems that you're in favor of groups that have physical proximity so they can all sit in a shared space and eat and read-from-paper together... or at least that you're proposing that a group that already has the proximity to do this, should take advantage of their proximity?

What about groups that are more spread out, that are already meeting in a video chat with a shared text channel, who look at screens and can research things as they talk?

[-]Nikola Jurkovic1mo51

To be clear, my post is mostly about student groups at universities.

I'm not super sure what to do in virtual groups. I think that BlueDot has a long track record of organizing high-quality online reading groups, but the selection effects there are pretty different. People often join student groups without the intent or executive function to actually do readings ahead of time. I'm not sure what BlueDot's approach is to making sure people actually do the readings and stay focused during the meetings. When I was doing online reading groups I remember I would only have the call on my screen so that I could focus on the discussion. I think encouraging people to keep their cameras on is good as it helps establish common knowledge that people are paying attention

[-]Matan Shtepel1mo30

Thanks for this Nikola! Overall, we at CMU agree that these seem like good principles to strive for and are making some changes (e.g. sorting in RG).

I think we are a little hesitant about gatekeeping because we are not confident we can distinguish who will become a positive presence for AI Safety. Historically some promising folks have faded (e.g. into random startups) and quiet randos flourished into some of our best output students.

We are also hesitant about starting with X-risk "seriousness" because it seems to push away folks who need more time to warm up the idea. Many good people's models of the world respond better to many gentler nudges over time rather than decisive and (often somewhat overly confident?) arguments.

[-]MieszkoP1mo1-2

I’m not so sure. I think the requirement to read the material before the group meeting is a strong motivator to actually do it, and it allows time to reflect on the text. Reading during the meeting wastes the time of those who have already read it, and if the text is long, it could mean hours of reading. Reading on your own also allows for a much deeper analysis of the content compared to listening to an audiobook. I also disagree with the idea of imposing age or knowledge-level restrictions. If someone has read the text with understanding, it implies they looked up any concepts they encountered along the way. On the other hand, if the text is very introductory, it stands to reason that the participants won't be experienced individuals, as they wouldn't want to go over the basics again.

[-]Nikola Jurkovic1mo30

I think the requirement to read the material before the group meeting is a strong motivator to actually do it

This seems wrong in my experience organizing reading groups with undergrads. Also, reading in front of others is an even stronger motivator. It's not just about whether people are motivated to read the reading or not. It's also about whether every person in the group knows that every other person in the group has also read the thing. This is impossible if they have to read in advance, but it's what happens by default if they read during the meeting.

Reading during the meeting wastes the time of those who have already read it

Yes, this is a sad consequence of this, but most of the time, when people organize reading groups for AI safety, people have not read the thing already. If they have, why would they be attending the reading group?

if the text is long, it could mean hours of reading

I think that reading groups, especially for undergrads and grad students, should definitely not be assigning much more than an hour of reading per week.

I also disagree with the idea of imposing age or knowledge-level restrictions. If someone has read the text with understanding, it implies they looked up any concepts they encountered along the way. On the other hand, if the text is very introductory, it stands to reason that the participants won't be experienced individuals, as they wouldn't want to go over the basics again.

There are levels to understanding. An ML researcher's understanding of a paper will be much higher than that of the undergrad, even if they've both read it and feel like they understand it. Also, I'm not only making a claim that the understanding will be different, but also that people do not want to be in reading groups where other people have obviously different levels of knowledge or are much younger than them. Most grad students have a pretty strong preference not to be surrounded by undergrads.

[-]alackles1mo10

Overall this sounds to me like good general community organization level feedback; thanks for putting together your thoughts.

I was wondering if you could expand on two things in this post:

The "age" part of clustering folks; roughly equal ML knowledge makes sense to me, but I'm not sure I agree about age, only because I'm unsure what your reasoning is.
The statement that some people's presence is net-negative to events. I think it would be helpful for people considering organizing if you have some examples of what you would consider signs someone's presence is or would be net-negative at such events.

[-]Nikola Jurkovic1mo30

The "age" part of clustering folks; roughly equal ML knowledge makes sense to me, but I'm not sure I agree about age, only because I'm unsure what your reasoning is.

I mostly speak from experience in talking to grad students and observing what kinds of events they go to. Grad students are expected and prefer to hang out with grad students, undergrads are expected and prefer to hang out with undergrads.

The statement that some people's presence is net-negative to events. I think it would be helpful for people considering organizing if you have some examples of what you would consider signs someone's presence is or would be net-negative at such events.

If someone is consistently being disruptive, not engaging with ideas seriously, etc.

Moderation Log