[Crossposted from: https://www.catalyze-impact.org/blog]

We are excited to introduce the eleven organizations that participated in the London-based Winter 2024/25 Catalyze AI Safety Incubation Program. 

Through the support of members of our Seed Funding Network, a number of these young organizations have already received initial funding. This program was open to both for-profit and non-profit founders from all over the world, allowing them to choose the structure that best serves their mission and approach. We extend our heartfelt gratitude to our mentors, funders, advisors, the LISA offices staff, and all participants who helped make this pilot incubation program successful.

This post provides an overview of these organizations and their missions to improve AI safety. If you're interested in supporting these organizations with additional funding or would like to get involved with them in other ways, you'll find details in the sections below. 

To stay up to date on our activities, or to be involved in future iterations of the program, please consider subscribing to our newsletter and/or filling out our expression of interest form.

Group picture in front of the LISA offices.

The AI Safety Organizations

• Wiser Human advances AI control by modelling risks, stress-testing safeguards, and developing subversion-robust control protocols.
• [Stealth] is bringing the Industrial Revolution to science, starting with systematizing AI alignment with AI agents.
• TamperSec develops hardware security for AI hardware, enabling international AI governance.
• Netholabs is a Whole Brain Emulation (WBE) research organization dedicated to preserving and enhancing human agency.
• More Light protects insiders and evidence against bad actors in frontier AI labs, and helps apply strategic legal pressure to achieve reforms.
• Lyra Research is leveraging existing model capabilities to directly automate safety research with agentic systems, today.
• Luthien is building on AI Control research to provide production-ready AI Control solutions as quickly as possible.
• Live Theory is building out a novel research methodology and infrastructure for contextually-adapting AI safety.
• Anchor Research develops evaluations to help understand whether AI systems actually stay safe and aligned when running for days or weeks at a time.
• Aintelope’s mission is to solve AGI alignment via biologically inspired mechanisms of cooperation and social behavior.
• AI Leadership Collective runs fellowship programs that empower frontier lab employees to implement AI safety initiatives in their organizations.

1. Wiser Human

Co-founders: Francesca Gomez, Seb Ben M'Barek
Website: www.wiserhuman.ai
Email: info@wiserhuman.ai
Based in: London, UK
Structure: Most likely non-profit, exploring for-profit opportunities.

Background & Vision

Wiser Human was founded by Francesca Gomez and Sebastien Ben M'Barek, who bring expertise in risk management, software engineering of complex systems and AI safety research. We aim to map the limits of AI control, evaluating all available options, from deployment restrictions to technical mitigations, so we can understand, extend, and clearly communicate the frontier of control safeguards for advanced AI.

As AI systems become more capable and are granted greater autonomy, existing safety measures are likely to prove insufficient, particularly if advanced models learn to subvert safeguards. Our approach combines comprehensive threat modelling, honeypots to elicit unsafe behaviours, and control evaluations to stress-test AI’s ability to bypass protections. By rigorously testing and developing layered safeguards, we work to identify where control remains viable and where risks become unmanageable.

By shifting AI safety from theoretical assurances to verifiable proof, we aim to establish new standards for AI control. Our work equips decision-makers with a clear, empirical understanding of the boundaries of AI control, ensuring that policies and safety frameworks remain grounded in what is realistically achievable as AI surpasses human intelligence.

Near-term Plans

Over the next year, we will develop systematic threat modelling techniques to identify high-risk AI failure scenarios. Our first priority is to create a repeatable methodology for assessing how misaligned, scheming AI agents could cause catastrophic harm by exploiting their capabilities, tool access, and scalability.

In parallel, we will evaluate which safeguards are vulnerable to AI subversion, running control evaluations to design more robust control protocols for untrusted models.

Finally, we will engage senior risk leaders with targeted demos, showcasing real examples of AI models subverting safeguards currently relied upon by policymakers and labs. This will bridge the gap between AI safety research and real-world decision-making, helping stakeholders understand both the potential strengths and limitations of AI control.

Room for Additional Funding

We are seeking $400K to accelerate the development of threat modelling techniques and subversion-robust control protocols. Additional funding would enable us to expand our research team, scale empirical evaluations, and develop practical AI control frameworks.

Our priorities are: (1) building systematic AI threat assessments, (2) testing and refining subversion-resistant safeguards, and (3) engaging senior risk and corporate leaders through targeted demos. With full funding, we could deliver empirical validation of AI control strategies to decision-makers who can drive change.

2. [Stealth]

Founder: Jacques Thibodeau
Website: in stealth (email or DM jacquesthibs for beta access)
Email: thibo.jacques@gmail.com
Based in: Bay Area
Structure: Public Benefit Corporation (For-Profit)

Background & Vision

[Stealth] was founded by Jacques Thibodeau, an AI safety researcher with experience building the alignment literature dataset and doing research in areas like mechanistic interpretability and behavioural evaluations. For phase 1, we are developing infrastructure to progressively automate technical AI safety research, with the expectation that this work will be crucial within short timelines.

Our platform helps researchers rapidly iterate on experiments by providing agents and tools integrated into a centralized alignment research ecosystem. This includes access to curated codebases, research tools, and up-to-date research findings. We expect that our work will soon accelerate entire research agendas.

In response to dual-use risks and to preserve our moat, we will keep some of our infrastructure private (shared with vetted researchers and organizations). We chose to go for-profit because we believe it will allow us to quickly scale our impact.

We intend to repurpose our internal infrastructure for potentially high-stakes R&D domains that will differentially accelerate safety.

We invite prosaic alignment researchers who want to accelerate their agenda and funders who share our sense of urgency to get in touch.

Near-term Plans 

We'll soon start beta testing with selected AI safety researchers, focusing on automated evaluations and interpretability research. Our immediate priority is improving our platform through direct feedback from alignment researchers. Over the next year, we aim to significantly accelerate alignment research through our infrastructure, setting up the foundation for accelerating other domains of high-stakes R&D and leveraging swarms of AI safety agents at scale.

Room for Additional Funding

We seek $400,000 for three technical co-founders and additional startup expenses to make immediate progress on our platform. We’re open to both donations and investments at this point in time. Beyond that point, we will need additional funding to hire a machine learning engineer and full-stack developer. We expect all of our staff to be able to scale productively when leveraging a swarm of AIs and give them a generous budget to do so. Over 99% of our employees will be AIs.

3. TamperSec

Founder: Jonathan Happel
Website: tampersec.com  
Email: contact@tampersec.com
Based in: Currently building in Germany, likely relocating to the Bay Area soon.
Structure: Delaware C Corp (for-profit), likely restructuring to a Public Benefit Corporation.

Background & Vision

Adversaries can manipulate AI hardware at the nanometer scale, allowing them to extract confidential data, steal AI model weights, or disable on-chip governance mechanisms. These attacks can be executed at scale and, at present, remain both undetectable and unpreventable.

Such vulnerabilities risk the unintended proliferation of AI capabilities. Moreover, they undermine efforts by governments to establish international treaties that rely on on-chip verification mechanisms. The need to counteract this with novel hardware security has been highlighted by multiple papers in recent months.

At TamperSec, we are developing a secure box for AI hardware. This box is retrofittable at existing servers, capable of identifying nanometer-scale modifications, and deletes a secret attestation key upon tampering.

This is an emerging and largely unexplored field with no traction outside our organization. Since September 2024, we have been working full-time on this mission, supported by Impact Academy, the Survival and Flourishing Fund, Entrepreneur First's def/acc, and Catalyze Impact.

TamperSec’s founder, Jonathan, brings a decade of experience in product development and expertise in hardware security and entrepreneurship, positioning him well to lead this initiative. 

Near-term Plans

We have developed two iterations of our hardware and will continue refining our product based on recent insights. Additionally, we plan to conduct both internal and external red-teaming to rigorously test its security, identify vulnerabilities, and eliminate potential flaws.

Our goal is to launch a market-ready product and drive early industry adoption by 2025.

We are actively hiring Electronic Engineers, Embedded Systems Engineers and a Business Development Lead - if you have relevant expertise or recommendations, we’d love to hear from you.

Room for Additional Funding

Additional funding would allow us to expand our R&D efforts. With up to $1.5 million in additional investment, we could scale our engineering team and accelerate product development.

Beyond this, an additional $500,000 would be allocated to enhancing transparency in hardware security. We plan to pilot an information-sharing protocol that enables querying third-party security assessment results without exposing underlying attack vectors. This approach would provide decision-makers with critical insights while minimizing the risk of information hazards.

4. Netholabs

Co-founders: Catalin Mitelut, Christian Larsen, Jamie Boyd
Website: netholabs.com
Email: info@netholabs.com 
Based in: London, UK
Structure: Delaware C Corp (for-profit), with UK subsidiary. 

Background & Vision

Netholabs is a Whole Brain Emulation (WBE) research organization dedicated to preserving and enhancing human agency.

Our mission is to develop and validate large-scale, biologically and ethologically faithful models of the brain, an ambitious goal that demands extensive neuro-behavioral datasets.

Central to our work is the creation of cutting-edge neurotechnology, including the world's most advanced behavioral arenas and the highest-channel-count wireless neural loggers for mice. Our integrated systems enable continuous, longitudinal recordings across tens to hundreds of brain regions in freely behaving animals, generating multimodal datasets at unprecedented scales.

This unique capability allows us to explore neural dynamics during natural behaviors and rigorously test whether deep neural networks can accurately emulate complex biological systems. Our datasets will further support future parameterization for connectomic approaches to WBE.

Near-term Plans

Over the coming year, Netholabs will:

• Release our first curated datasets for functional WBE, providing essential benchmarks for researchers worldwide.

• Expand our existing behavioral arenas tenfold and fully automate our neural logging hardware to significantly accelerate data collection.

• Publish new neural scaling laws derived from these extensive, high-resolution recordings.

• Establish a third animal research facility with high performance computing capacity at or near a prominent university in the London region.

Room for Additional Funding

We are currently seeking investors for our upcoming funding round, with capital directed toward scaling our advanced data collection infrastructure following the successful deployment of our MVP systems.

Additionally, we are experienced in managing targeted non-profit grants and welcome sponsorship opportunities for specific projects. For example, you can support our forthcoming seminar on WBE and its implications for AI safety.

5. More Light

Founder: Jay Ready
Website: https://www.morelight.ai/
Email: JayReady@proton.me
Based in: London, UK and San Francisco, USA.
Structure: Non-profit 

Background & Vision 

More Light emerged from direct experience with frontier AI lab accountability. We're currently protecting someone going against a major AI frontier lab and witnesses supporting them, who have faced intimidation, surveillance, and cyberattacks. We're building practical knowledge of what works.

Our team knows what it's like to face threats from AI labs. We've been there ourselves. We combine this real-world experience with deep knowledge of AI governance, understanding what good oversight looks like. This gives us a clear view of what insiders actually need when speaking up and how to turn individual cases into meaningful reform.

Individual action can drive meaningful change, but those taking action need protection and strategic support. We provide this through coordinated legal, security, and professional networks that we're building and testing. Our AI governance advisors help ensure these actions lead to meaningful changes.

The future of AI development requires reliable accountability channels. More Light exists to make these channels work effectively while protecting those who use them. By supporting courageous individuals, we aim to make frontier AI development more responsible and transparent.

Near-term Plans

Our work has two priorities:

First, we're building protection systems. This means expanding our network of legal, security, and cybersecurity experts who can respond immediately to threats. We’ve done some research on Suchir Balaji’s case to figure out likely scenarios of what happened, to use in our threat models to protect other insiders.

Second, we're identifying the most impactful changes we can achieve through legal channels. We work with AI governance experts to determine which specific reforms will improve safety the most. This ensures that when insiders take action, their efforts create meaningful improvements in AI lab practices.

Success means 1) no more harm comes to lab insiders 2) legal pressure leads to meaningful change. 

Room for Additional Funding

We seek around $700,000 for our first year of operations. This covers core infrastructure, expert staff, and an emergency fund for protecting industry insiders. With additional funding, we can maintain a legal action fund and conduct more advocacy for transparency and accountability of frontier AI labs. 

6. Lyra Research

Founder: Ronak Mehta
Email: ronakrm@gmail.com
Based in: San Francisco, CA, USA
Structure: Most likely non-profit, exploring for-profit options.

Background & Vision

Models are capable enough today to replace software engineers, conduct literature reviews, and generate novel research ideas. Given that the fundamental nature of the workforce is likely to change drastically in the coming years, and that these technologies are being used to increase automation of capabilities research, Lyra wants to close this gap sooner rather than later, before it grows wider.

With the right scaffolding, frontier models can be used to accelerate AI safety research agendas. There are now many academic papers and projects as well as for-profit, capabilities-focused or business-driven startups building out agentic and autonomous systems. We want to build this scaffold to specifically target core technical AI safety problems.

My (Ronak's) experience with traditional machine learning research during graduate school, with software engineering at a recent startup, with modern AI safety research, and with modern SWE development tools lends itself strongly to making this happen.

Near-term Plans

The immediate term plan is building upon proofs of concept to meaningfully take on low-hanging fruit alignment work, e.g., training typical SAEs on a new model release, automated eval creation, and rough exploration around SOTA technical results.

Within 3 months we aim to leverage hundreds of orchestrated agents to more deeply tackle and explore harder problems, including for example, replicating a more complex recent paper without access to its original code.

Lyra believes timelines are quite short, and that within 6 months it will largely be possible to fully automate most of the engineering and research work currently being done. Our goal is to make sure that technical safety agendas keep up with this prediction.

Room for Additional Funding

Lyra is currently looking for $100,000 to build fast within the next 6 months, largely leveraging inference time compute for automated safety research.

7. Luthien

Founder: Jai Dhyani
Website: https://luthienresearch.org
Email: contact@luthienresearch.org
Based in: Seattle, WA
Structure: Non-profit

Background & Vision

Luthien was founded by Jai Dhyani, a senior FAANG ML engineer with over a decade of experience in training and deploying AI models at scale of billions of MAU. Building on pioneering AI Control research from Redwood Research, Luthien's mission is to develop and deploy production-ready AI Control solutions to mitigate the risks posed by unpredictable AI.

AI Control differs from traditional approaches to AI Safety by assuming that models may act adversarially. Working from that assumption, AI Control ensures safe outcomes by building robust oversight and management systems that demonstrably prevent misaligned systems from achieving misaligned goals. Luthien aims to take this from research into widespread practice by developing automated frameworks for stress-testing AI systems, creating tooling for easy deployment under Control measures, and building clear feedback loops to gather real-world data on system effectiveness.

Luthien's ultimate goal is to ensure that when highly dangerous AI is deployed, it will be under an effective AI Control system that drastically reduces the risk of catastrophic outcomes.

Near-term Plans

Luthien is currently iterating on its proof-of-concept system and coordinating with AI Control experts at research and frontier labs to develop a production-ready solution.

Room for Additional Funding

Luthien is currently targeting $500,000 to grow the Seattle-based team to include several SDEs and at least one PM to manage coordination with partners. We have received a seed grant from the AI Safety Tactical Opportunities Fund that covers part of these costs.

8. Live Theory

Founder: Sahil
Email: fufstsahil@gmail.com
Based in: London, UK
Structure: Non-profit at current stage, open to for-profit soon.

Background & Vision

We aim to open up new research *methodology* that can handle the hydra of the sociotechnical risks of AGI, using AI-based adaptive research artefacts.

(By "adaptive" we mean the ability generative AI will soon have to produce research artefacts like publications, tools, even theories, that can adapt to context with some prompts+data.)

We are working on an AI safety research methodology and design philosophy that exploits this upcoming adaptivity to enable scaling of alignment techniques across the variety of challenges/evasiveness that AI risk poses. In particular, we expect to better handle important slippery concepts (eg. “deception”, "power-seeking") that resist static abstraction, using contextually adapting theories (viz. "live theory").

We’re creating a space (community + platform) for the exchange of such artefacts.

Near-term Plans

We’re working on interface design experiments within the same design philosophy, for the sake of fast feedback loops. In parallel, we’re charting out AI risk mitigation applications for them.

Room for Additional Funding

In the range of $100k-500k, to be able to financially support ~4 FTE researchers and developers who are already dedicating a major amount of time on projects under these directions.

9. Anchor Research

Founder: Diogo Cruz
Website: N/A - more details on Anchor Research can be found in this document.
Email: diogo.abc.cruz@gmail.com
Based in: US (fiscal sponsor)
Structure: Most likely non-profit

Background & Vision

As AI systems become more autonomous, we'll run into a problem: how do they behave when running for extended periods? Right now, there's very limited research examining what happens when AI agents operate for days or weeks at a time. An AI might pass all our short safety tests but then develop concerning behaviors over time - like drifting away from its original goals or finding ways around its constraints.

Anchor Research grew from Diogo Cruz's work in AI safety research, where he noticed this gap in our understanding. We're building tools to study long-term AI behavior systematically, looking for patterns that might indicate future problems. Our goal is simple: catch potentially dangerous behaviors before they show up in real-world deployments.

We're already exploring this research agenda, and we're building open frameworks that anyone can use to test their AI systems. By gathering solid evidence about long-term behavioral risks, we hope to help both AI developers and policymakers make better decisions about deploying autonomous systems.

Near-term Plans

We've built an initial framework for testing how well AI agents stick to their intended behaviors over time. Now we're expanding it to capture more types of behavioral changes and make it easier for others to use. We're working closely with research partners to develop clear standards for what "stable" AI behavior looks like and creating tools that help spot potential problems early.

Our research focuses on two key questions: how do AI systems gradually drift away from their original goals, and do they accidentally break free from constraints over time? Through extended experiments running agents for days or weeks, we're gathering evidence about these behaviors and developing better ways to detect them before they become dangerous.

Room for Additional Funding

With more funding ($50k - $200k), we could bring on additional researchers to run more extensive experiments and develop better testing tools. We're especially keen to work with AI labs and help them implement long-term safety testing in their development process. Both funding and research partnerships would help us expand this important work faster.

10. Aintelope

Co-founders: Angie Normandale, Joel Pyykkö
Website: www.aintelope.net 
Email: angie@aintelope.net, pyykkojoel@gmail.com 
Based in: Remote team with legal entity in Germany, team members across UK and EU
Structure: Non-profit

Background & Vision

Our team is a multi-disciplinary set of experts from neuroscience to computer science, whose dream has been to simulate brain-like mechanisms to study AI safety. Our project is at the most opportune moment to ride the growing agentic wave, as the simulations we’re planning are meant to test whether we can replicate any of the safety related behaviors humans exhibit. 

Although there are a few other projects around the same topic, ours differs by our approach to nurture simulated agents with biologically inspired developmental heuristics, which allows us to fail fast and keep exploring a broad neurological literature. 

Our aim is to find out if any of the aligned behaviors we humans exhibit can also scale to AGIs, and hope to bring these lessons and mechanisms into controlling any agentic solutions that have started to emerge in recent years. These include, but are not limited to, robotics, virtual assistants, self-driving vehicles, AI scientists and so on. 

Near-term Plans

Our next step is to simulate and benchmark safety relevant behavior as seen in nature, such as safe exploration (fear response), sustainability (homeostasis), treacherous turn (social norms, loyalty), avoiding instrumental convergence (maternal bonding) and more. The planned proxies allow us to develop a dynamic platform with an ensemble of proxy mechanisms, leading to rapid experimentation and publishing 2-3 papers by the end of 2025. These publications underlie the work that will enable us to experiment for a holistic control scheme for agents. We will discuss the results with our AI safety advisors to plan the best way to utilize these results within the growing agentic field. 

Room for Additional Funding

We’re seeking 475k for a year to finish simulating and benchmarking biologically inspired mechanisms (affective and homeostatic), which would be used next year to build an affective control framework for agents. With more we can parallelize and start the work sooner, as we could hire programmers for further experiments, as the work branches after the initial platform is ready. This work would include more mechanisms and related experiments, as well as formalization and product development.

11. AI Leadership Collective

Co-founders: Rupi Sureshkumar, Cecilia Callas
Website: aileadershipcollective.com  
Email: aileadershipcollective@gmail.com
Based in: San Francisco
Structure: Non-profit

Background & Vision

The AI Leadership Collective was founded by Cecilia Callas and Rupi Sureshkumar, who bring two decades of combined industry experience, including in Trust & Safety and AI community building. We aim to reduce catastrophic AI risk by empowering safety-conscious employees at frontier AI labs to implement organizational safety practices. This work is crucial because while critical AI development decisions are made within these companies, frontier labs consistently score poorly on safety benchmarks, with no organization rating above a "C" grade.

Our unique approach combines structured implementation support with community building to address a critical gap in AI safety: the disconnect between external safety expertise and internal decision-makers. Through our 3-month fellowship program, we select motivated professionals and provide them with education, mentorship, and practical support to execute safety initiatives within their organizations. We already have interest from employees at major AI companies like Anthropic and Microsoft, with promising project ideas including cross-company model review boards and industry-wide risk benchmarks.

Near-term Plans

Over the next six months, we plan to launch our pilot fellowship program with 25 Fellows from frontier AI labs. We’re finalizing our curriculum and building out our mentor network, drawing on expertise from both the AI safety community and organizational change specialists. The program will kick off with a three-day retreat in the San Francisco Bay Area, where fellows will develop concrete implementation plans for their safety initiatives and connect with other safety-minded AI practitioners.

Following the retreat, we'll provide three months of structured support, including weekly learning groups, expert mentorship, and implementation resources. We'll measure success through both project completion and improvement in organizational safety metrics. Based on pilot outcomes, we'll refine our program design for a second cohort while building our always-on community platform to sustain long-term impact.

Room for Additional Funding

We are seeking $264k to launch our pilot fellowship program over six months, covering program development, a 25-person cohort, and core operational costs. Additional funding would enable us to extend to a full year ($532k) and expand our impact through a second fellowship cohort, cross-company working groups for safety frameworks, and enhanced programming. With full funding, we could support 50 fellows across two cohorts, creating a robust community of practice for organizational safety while establishing AILC as an enduring organization.