Four hands-on Jupyter notebooks offer a practical introduction to building AI evaluations with Inspect AI.
As a four-person project team for the Spring 2026 Technical AI Safety Evals Course, we revised and expanded notebooks, originally written as course materials, that teach Inspect AI - from your first evaluation through benchmarks, LLM judges, and a small agent. As a natural continuation of the project, we revised the official Inspect AI tutorial and filled the gaps we found in it. We added a new Custom Solvers section and fixed several inconsistencies.
This post covers:
what we built,
how we worked on it,
why we think teaching people to build evals well matters more for safety than for capabilities.
We’d appreciate any feedback on our project. What problems with the idea or its implementation do you see? What parts do you find valuable? Do you want to contribute yourself? Feel free to write here, on discord, via email, or directly in the github.
What We Built
The notebooks cover real-world use cases and essential theory.
They take you from your first Inspect evaluation to more advanced work with benchmarks, LLM judges, and agents. Here's what each notebook covers:
The basics of Inspect AI, including how to build your first evaluation
The statistical elements of evaluations
The LLM-as-a-Judge method for evaluation
How to evaluate autonomous LLM agents
How We Worked on This
We started with four Jupyter notebooks that had been supplementary materials for the course, feedback from course participants, suggestions from Inspect AI developers on Slack, and our own ideas for improvements.
Our workflow looked like this:
Turning all this input into a list of tickets and grouping related changes together;
Prioritizing the tickets;
Following a standard team development workflow: a separate Git branch for each change, PRs requiring two approvals before merging, weekly syncs, and async work in between.
As an example of our work, we rewrote the assignment on calculating LLM-as-a-judge metrics to make it much clearer. We also added exercises on Inspect Scout. As part of QA, we fixed an incorrect validation test in one of the assignments. Beyond these examples, we made many other improvements to make the notebooks fully standalone.
After improving our notebooks, we looked at the official Inspect AI tutorial. We worked on it from an educational point of view:
We analyzed the desired learning outcomes;
Identified the gaps;
Designed improvements that would supplement the tutorial while keeping it minimalistic;
Implemented those changes (added a new standalone Custom Solvers section and fixed several inconsistencies).
AI evaluations are an important component of the AI governance toolkit, underlying current approaches to safety cases for preventing catastrophic risks. Our paper examines what these evaluations can and cannot tell us. Evaluations can establish lower bounds on AI capabilities and assess certain misuse risks given sufficient effort from evaluators.
Unfortunately, evaluations face fundamental limitations that cannot be overcome within the current paradigm. These include an inability to establish upper bounds on capabilities, reliably forecast future model capabilities, or robustly assess risks from autonomous AI systems. This means that while evaluations are valuable tools, we should not rely on them as our main way of ensuring AI systems are safe. We conclude with recommendations for incremental improvements to frontier AI safety, while acknowledging these fundamental limitations remain unsolved.
So we need safety evals to have feedback signals. For now, capabilities development has rich feedback signals. Training loss, RL reward, internal testing, etc. provide a lot of information. But AI safety often has no comparable signal, so it must instead rely on evaluations of how systems behave in certain situations or conditions. This makes evals unusually important specifically for AI safety.
Producing trustworthy safety evaluations is a pretty difficult task, though! Results depend on particular details such as scaffolding, time budgets, prompting, etc. Moreover, models may recognize evaluation settings, conceal capabilities, or change their behavior depending on whether they believe they are being monitored or trained (alignment faking).
Together, that's why we think work on AI evaluations - including teaching people to do it well - matters more for safety than for capabilities, and it's also why the specific tooling matters, not just the general practice.
We think Inspect AI is that tool. It's a general-purpose evaluation framework whose roots and ecosystem are closely connected to AI safety. It was developed by the UK AI Security Institute and is used by safety-focused organizations such as Apollo Research and METR, as well as by a wider community of independent researchers. This framework provides an intuitive interface to develop understandable, easy-to-reproduce evals.
Inspect AI is powerful and well documented, but it still can be difficult to move from reading the documentation to constructing useful evaluations yourself. So these notebooks help bridge that gap.
What You Need to Get Started
Required: Python coding at a basic level.
May be helpful, but not expected: experience with NumPy, Pandas, and basic statistics.
Not required: experience with Inspect AI, Jupyter notebooks, AI safety research. The first notebook includes setup instructions and begins with an introduction. Later notebooks build toward more challenging and engaging exercises.
You also don't need access to a paid model API: the notebooks explain how to use free cloud models or run an open model locally with Ollama.
The full set of notebooks to run locally is on GitHub. Also all four notebooks are on Google Colab: 1, 2, 3, and 4. There is also a guide via Google NotebookLM.
We'd like to hear:
what worked,
what was confusing,
where you got stuck,
whether you work through the whole sequence or just one notebook.
Questions, suggestions, bug reports, and contributions are all welcome.
Thanks also go to Alexandra Rybakova, Jamilya Erkenova, and Sergei Kudriashov for running interactive workshops using the notebooks. These workshops were especially valuable because they helped gather feedback from more than a hundred course participants.
Special thanks also go to JJ Allaire, creator of Inspect AI, and Alexandra Abbas and Charles Teague from Meridian Labs for their feedback and review at different stages of the project.
TL;DR
Four hands-on Jupyter notebooks offer a practical introduction to building AI evaluations with Inspect AI.
As a four-person project team for the Spring 2026 Technical AI Safety Evals Course, we revised and expanded notebooks, originally written as course materials, that teach Inspect AI - from your first evaluation through benchmarks, LLM judges, and a small agent. As a natural continuation of the project, we revised the official Inspect AI tutorial and filled the gaps we found in it. We added a new Custom Solvers section and fixed several inconsistencies.
This post covers:
All four notebooks are on Google Colab: Notebook 1, Notebook 2, Notebook 3, Notebook 4. The full set to run locally is on GitHub. There is also a guide via Google NotebookLM.
We’d appreciate any feedback on our project. What problems with the idea or its implementation do you see? What parts do you find valuable? Do you want to contribute yourself? Feel free to write here, on discord, via email, or directly in the github.
What We Built
The notebooks cover real-world use cases and essential theory.
They take you from your first Inspect evaluation to more advanced work with benchmarks, LLM judges, and agents. Here's what each notebook covers:
How We Worked on This
We started with four Jupyter notebooks that had been supplementary materials for the course, feedback from course participants, suggestions from Inspect AI developers on Slack, and our own ideas for improvements.
Our workflow looked like this:
As an example of our work, we rewrote the assignment on calculating LLM-as-a-judge metrics to make it much clearer. We also added exercises on Inspect Scout. As part of QA, we fixed an incorrect validation test in one of the assignments. Beyond these examples, we made many other improvements to make the notebooks fully standalone.
After improving our notebooks, we looked at the official Inspect AI tutorial. We worked on it from an educational point of view:
Why Evals - and Why Inspect AI?
MIRI’s article "What AI evaluations for preventing catastrophic risk can and cannot do" begins with:
So we need safety evals to have feedback signals. For now, capabilities development has rich feedback signals. Training loss, RL reward, internal testing, etc. provide a lot of information. But AI safety often has no comparable signal, so it must instead rely on evaluations of how systems behave in certain situations or conditions. This makes evals unusually important specifically for AI safety.
Producing trustworthy safety evaluations is a pretty difficult task, though! Results depend on particular details such as scaffolding, time budgets, prompting, etc. Moreover, models may recognize evaluation settings, conceal capabilities, or change their behavior depending on whether they believe they are being monitored or trained (alignment faking).
Together, that's why we think work on AI evaluations - including teaching people to do it well - matters more for safety than for capabilities, and it's also why the specific tooling matters, not just the general practice.
We think Inspect AI is that tool. It's a general-purpose evaluation framework whose roots and ecosystem are closely connected to AI safety. It was developed by the UK AI Security Institute and is used by safety-focused organizations such as Apollo Research and METR, as well as by a wider community of independent researchers. This framework provides an intuitive interface to develop understandable, easy-to-reproduce evals.
Inspect AI is powerful and well documented, but it still can be difficult to move from reading the documentation to constructing useful evaluations yourself. So these notebooks help bridge that gap.
What You Need to Get Started
Required: Python coding at a basic level.
May be helpful, but not expected: experience with NumPy, Pandas, and basic statistics.
Not required: experience with Inspect AI, Jupyter notebooks, AI safety research. The first notebook includes setup instructions and begins with an introduction. Later notebooks build toward more challenging and engaging exercises.
You also don't need access to a paid model API: the notebooks explain how to use free cloud models or run an open model locally with Ollama.
The full set of notebooks to run locally is on GitHub. Also all four notebooks are on Google Colab: 1, 2, 3, and 4. There is also a guide via Google NotebookLM.
We'd like to hear:
Questions, suggestions, bug reports, and contributions are all welcome.
Credits
The notebooks were initially developed by Alex Gorbachev and Alexandra Rybakova as supporting materials for the Spring 2026 Technical AI Safety Evals Course, organized by the Monoid AI Safety Hub.
They were later expanded and adapted into standalone learning materials through a volunteer project carried out as part of the course. The team consisted of Alex Gorbachev, Vladislav Meshkorudnyi, Ivan Belashkin, and Denis Shmurak, with Alex as project lead.
Special thanks go to Elena Ericheva, Igor Ivanov, and Slava Meriton for their ideas and valuable feedback on the tasks. Check out Elena’s Russian-language Telegram channel on AI safety.
Thanks also go to Alexandra Rybakova, Jamilya Erkenova, and Sergei Kudriashov for running interactive workshops using the notebooks. These workshops were especially valuable because they helped gather feedback from more than a hundred course participants.
Special thanks also go to JJ Allaire, creator of Inspect AI, and Alexandra Abbas and Charles Teague from Meridian Labs for their feedback and review at different stages of the project.