This is a special post for quick takes by Marko Katavic. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I attended an AI Safety event in London last night, and it was well attended by both AI Safety practitioners and enthusiasts. In a lot of way, it was structured as a recruitment drive.
Epistemic status on what I'm about to claim: I have very low confidence in this, it's incidental from one event and inferring across organisational dynamics, but it's a hypothesis that seems structurally sound. I have low confidence, but high concern which is likely biasing me in many ways. I am also aware that I'm a newcomer to safety, which cuts against what I'll say below (but I have felt on my own skin how hard it is to maintain epistemic humility against a field with a rich history of deep thought).
From conversing to the people around the room, sometimes practitioners as well, but especially enthusiasts - there's been a notable amount of people with low epistemic humility and high conviction.
I hypothesise - AI Safety is getting its big television moment. It is a cause that is ethically clean, that provides purpose in life, that is well renumerated and that doesn't have institutional educational pathways into. The likely outcome is that it will draw a higher-than average amount of people with a hero-complex (which pushes for ego-feeding satisfaction) and sub-par epistemics (compared to the current AI safety baseline), purely due to these factors.
I worry that in the short term this causes a bulge in the AI Safety community that adds coordination tax, decreases epistemic baseline, and adds marginal idea value. Net, it decreases the ability of the ecosystem to produce safety reduction.
I am not saying this because I think we shouldn't recruit people into a field that is largely underfunded and underpopulated - but to illustrate that I think without dedicated ecosystem steering, coordinating, growing, pollinating and management pathways - we'll likely end up with mess as organisational complexity increases, rather than with net gain. A predictable outcome of unmanaged organisational complexity growth is flattening of signal and explosion of noise.
Is anyone working on this? I didn't manage to catch the ecosystem coordinator yesterday as she was in high demand - but it made me worry.
Absolutely dismantle my worry if you can.
The medical field has dealt with this problem with pre-med people for a long time. There are ways to determine authenticity and goodness of fit such as asking what their backup plans are if their chosen role doesn't work out for them, asking them adjacent questions that reveal their values and motivations, etc.
A similar failure mode that comes to mind is how Engineers Without Borders has at times in the past had overconfident engineering teams build complex solutions dependent on hard-to-replace parts rather than focusing on long-term sustainability and ha... (read more)