When I finished the 2024 AI safety fundamentals course by BlueDot Impact, there was a small tick box saying something like 'I want to start an org in this space'.
I was just coming to the end of my artificial intelligence MSc, and wanted to start an AI safety research organisation. I had tried once and done various other entrepreneurial things, and was looking for precisely this opportunity. Turns out the tick box was put there by then alignment course lead, Adam Jones, who invited me to co-work with him on starting a new AI safety organisation on something related to AI-enabled oligarchies.
We brainstormed for a bit, and I ended up focusing on lock-in, which seemed highly neglected as a problem, and I was excited to explore what an org could do in the space. BlueDot offered me a small grant to co-work with them in an incubator-style arrangement at LISA. I spent 8 weeks working with them on what the org should do, and making longer-term funding applications to start Formation Research.
Fast-forward to now, and I am recruiting a founding team to work on an empirical research agenda for secret loyalties, aiming to answer foundational science of deep learning questions that will help us build defences in AI labs against secretly loyal AI systems. While the org has not yet scaled or built a founding team, I have learned a lot about what is important when starting a venture like this, which I'm writing up here for other people looking to do the same thing
What I think is Important
1. Do things as in-person as you can.
I spent some time working on the org remotely. Compared with living in London, working out of LISA full-time, and attending conferences and events, I think I made much less progress being remote.
Feedback loops were possible when working remotely, but I was more isolated from the community. Even if I read blog posts and chatted with Adam, this only happened every now and then and when it was deliberately planned.
In LISA, talking to people all the time, I constantly rehearsed 'my story': who I am, what I'm doing, why, etc. This made me think through, out loud, my reasoning, and practice it often.
This doesn't naturally happen remotely. Sure you could schedule a chat in the mirror every lunchtime about what you're doing and why, and that might do something, but if you can be as in-person as possible, I believe this will help make you more effective at doing the thing
2. Talk to people in your specific subject area.
I made more progress on my thinking when talking to people who had already thought about the same thing.
Talking to people at Forethought helped me solidify my ideas around lock-in
Talking to ML researchers helped me figure out how to build, measure, and monitor the thing I cared about
I might have been able to derive this from first principles over many weeks. But it was much more efficient for me to learn from others and stand on their shoulders.
There are ways you can do this systematically. I used the VFWPA (Vision / Framing / Weakness / Pedestal / Ask) template from The Mom Test when reaching out to people, and I found people by going to events and talking to people at LISA about what I do.
I email people with the VFWPA template and talk to them using Mom Test questions where applicable.
I often ask for commitments at the end of meetings. This is usually something small like, 'where can I reach you, is email okay?', it may escalate as I get to know the person or if they indicate they are keen to help, e.g., 'please spend 5 mins commenting on this document by next Friday, there are feedback guidelines at the top of the doc'. Commitments came more naturally the more I got to know people, e.g., we would set up a meeting notes doc, a regular meeting, or establish some collaboration norms.
3. Get career capital and make it public.
This advice is already given at least by 80,000 Hours and Cal Newport, but I think it's important.
The more artefacts you can show people which indicate that you can do things and can think, the more they might want to do things and think about things with you.
Giving people solid evidence you can do things lets them trust the investment of their time rather than make an uncertain bet. Making it public by posting it online makes it much easier to point to.
4. Get help with the legal stuff.
I didn't have a traditional org-starting path. Orgs are often founded through incubators, or existing collaborations, or spinouts. This one started with Adam and me brainstorming about things that an org could do.
But something that was so helpful in setting up the organisation was the fact that BlueDot Impact helped incubate me at the start, then intro'd me to Impact Ops, who helped navigate the fiscal sponsorship and subsequent spinout, grant administration, and handle contracts.
If I had to do this stuff by myself it would have taken up a lot of my time, but it only took up about ~5% of my working time thanks to BlueDot and Impact Ops.
I think this is very important because it took me a long time (~9 months) to make solid progress on the org while working full-time on just that. If I had to also handle all the legal stuff that might have taken much longer.
Despite not going through a more typical route, I recommend fiscal sponsors, incubators, and residencies to people wanting to start an org. Yes, you may be entrepreneurial, but if you can automate or delegate the sweaty stuff it lets you focus on what you think is important as a founder. Especially when there are now established orgs offering this explicitly. Don't take on everything just because you can.
5. Set up a board of trustees, even in a low-effort, synthetic way.
This one is thanks to Michael Aird. He suggested I do exactly this back in August.
It was fairly easy to do. I wrote a document outlining what I would expect from unofficial trustees (~hour-long quarterly meetings, occasional doc review, and talking in a Slack channel), and gave it to people in my network whom I felt would have useful inputs (and who I just get along with).
They accepted and we started having the meetings, in which I just write a summary of what's going on in the org in some meeting notes, and then a list of questions for the board. I found this very helpful because I had a group of people who I could ask for advice when I was otherwise solo.
After we had done this for a while and I was appointing legal org members, I asked the unofficial trustees if they would like to become official trustees or advisers, again explaining what this meant for them, and some accepted. Now I have a real board of trustees composed of people whom I trust and with whom I have a rapport.
7. Make a website as soon as you can
When I was making an early funding application, Adam suggested that to get around the word limit I just make a website and put more information there and link it.
I made a simple static website in Next.js where I could put all the information I wanted to tell funders but didn't have the space. The website is still Next.js and it is hosted on Vercel (which is free), and Godaddy (which is relatively cheap)
Fast forward to now (about 1.5 years later), I am constantly pulling up the landing page when introducing people to the org. It takes me about two clicks and four keyboard presses to have my landing page up, and when meeting people frequently, this has proved very useful in getting the gist across.
The landing page is supposed to say the most important things as simply as possible, the about page then has more info, and the research page has the research outputs we can point to (also serves as quick-access career capital to gesture at).
I later updated the website using the guidance in How to Launch a High-Impact Nonprofit, and completely rely on it as a generally-accessible source of truth for the org. I think websites are really helpful.
Thanks to Adam Jones and Ben R Smith for suggesting I make this.
These are my opinionated takes on things that matter when starting a new organisation. It may also apply to orgs that are not AI safety research nonprofits. Andrew Draganov and Erin Robertson also recently wrote a post which contains useful thoughts on building an AI safety research team.
How and Why it Started
When I finished the 2024 AI safety fundamentals course by BlueDot Impact, there was a small tick box saying something like 'I want to start an org in this space'.
I was just coming to the end of my artificial intelligence MSc, and wanted to start an AI safety research organisation. I had tried once and done various other entrepreneurial things, and was looking for precisely this opportunity. Turns out the tick box was put there by then alignment course lead, Adam Jones, who invited me to co-work with him on starting a new AI safety organisation on something related to AI-enabled oligarchies.
We brainstormed for a bit, and I ended up focusing on lock-in, which seemed highly neglected as a problem, and I was excited to explore what an org could do in the space. BlueDot offered me a small grant to co-work with them in an incubator-style arrangement at LISA. I spent 8 weeks working with them on what the org should do, and making longer-term funding applications to start Formation Research.
Fast-forward to now, and I am recruiting a founding team to work on an empirical research agenda for secret loyalties, aiming to answer foundational science of deep learning questions that will help us build defences in AI labs against secretly loyal AI systems. While the org has not yet scaled or built a founding team, I have learned a lot about what is important when starting a venture like this, which I'm writing up here for other people looking to do the same thing
What I think is Important
1. Do things as in-person as you can.
I spent some time working on the org remotely. Compared with living in London, working out of LISA full-time, and attending conferences and events, I think I made much less progress being remote.
Feedback loops were possible when working remotely, but I was more isolated from the community. Even if I read blog posts and chatted with Adam, this only happened every now and then and when it was deliberately planned.
In LISA, talking to people all the time, I constantly rehearsed 'my story': who I am, what I'm doing, why, etc. This made me think through, out loud, my reasoning, and practice it often.
This doesn't naturally happen remotely. Sure you could schedule a chat in the mirror every lunchtime about what you're doing and why, and that might do something, but if you can be as in-person as possible, I believe this will help make you more effective at doing the thing
2. Talk to people in your specific subject area.
I made more progress on my thinking when talking to people who had already thought about the same thing.
I might have been able to derive this from first principles over many weeks. But it was much more efficient for me to learn from others and stand on their shoulders.
There are ways you can do this systematically. I used the VFWPA (Vision / Framing / Weakness / Pedestal / Ask) template from The Mom Test when reaching out to people, and I found people by going to events and talking to people at LISA about what I do.
I email people with the VFWPA template and talk to them using Mom Test questions where applicable.
I often ask for commitments at the end of meetings. This is usually something small like, 'where can I reach you, is email okay?', it may escalate as I get to know the person or if they indicate they are keen to help, e.g., 'please spend 5 mins commenting on this document by next Friday, there are feedback guidelines at the top of the doc'. Commitments came more naturally the more I got to know people, e.g., we would set up a meeting notes doc, a regular meeting, or establish some collaboration norms.
3. Get career capital and make it public.
This advice is already given at least by 80,000 Hours and Cal Newport, but I think it's important.
The more artefacts you can show people which indicate that you can do things and can think, the more they might want to do things and think about things with you.
Giving people solid evidence you can do things lets them trust the investment of their time rather than make an uncertain bet. Making it public by posting it online makes it much easier to point to.
4. Get help with the legal stuff.
I didn't have a traditional org-starting path. Orgs are often founded through incubators, or existing collaborations, or spinouts. This one started with Adam and me brainstorming about things that an org could do.
But something that was so helpful in setting up the organisation was the fact that BlueDot Impact helped incubate me at the start, then intro'd me to Impact Ops, who helped navigate the fiscal sponsorship and subsequent spinout, grant administration, and handle contracts.
If I had to do this stuff by myself it would have taken up a lot of my time, but it only took up about ~5% of my working time thanks to BlueDot and Impact Ops.
I think this is very important because it took me a long time (~9 months) to make solid progress on the org while working full-time on just that. If I had to also handle all the legal stuff that might have taken much longer.
Despite not going through a more typical route, I recommend fiscal sponsors, incubators, and residencies to people wanting to start an org. Yes, you may be entrepreneurial, but if you can automate or delegate the sweaty stuff it lets you focus on what you think is important as a founder. Especially when there are now established orgs offering this explicitly. Don't take on everything just because you can.
5. Set up a board of trustees, even in a low-effort, synthetic way.
This one is thanks to Michael Aird. He suggested I do exactly this back in August.
It was fairly easy to do. I wrote a document outlining what I would expect from unofficial trustees (~hour-long quarterly meetings, occasional doc review, and talking in a Slack channel), and gave it to people in my network whom I felt would have useful inputs (and who I just get along with).
They accepted and we started having the meetings, in which I just write a summary of what's going on in the org in some meeting notes, and then a list of questions for the board. I found this very helpful because I had a group of people who I could ask for advice when I was otherwise solo.
After we had done this for a while and I was appointing legal org members, I asked the unofficial trustees if they would like to become official trustees or advisers, again explaining what this meant for them, and some accepted. Now I have a real board of trustees composed of people whom I trust and with whom I have a rapport.
7. Make a website as soon as you can
When I was making an early funding application, Adam suggested that to get around the word limit I just make a website and put more information there and link it.
I made a simple static website in Next.js where I could put all the information I wanted to tell funders but didn't have the space. The website is still Next.js and it is hosted on Vercel (which is free), and Godaddy (which is relatively cheap)
Fast forward to now (about 1.5 years later), I am constantly pulling up the landing page when introducing people to the org. It takes me about two clicks and four keyboard presses to have my landing page up, and when meeting people frequently, this has proved very useful in getting the gist across.
The landing page is supposed to say the most important things as simply as possible, the about page then has more info, and the research page has the research outputs we can point to (also serves as quick-access career capital to gesture at).
I later updated the website using the guidance in How to Launch a High-Impact Nonprofit, and completely rely on it as a generally-accessible source of truth for the org. I think websites are really helpful.