This work was supported through the MARS (Mentorship for Alignment Research Students) program at the Cambridge AI Safety Hub (caish.org/mars).

Why We Built `inspect_wandb`

Evaluating frontier AI models can be messy; each benchmark has its own quirks, scripts, and formats, and even after a successful run, the results usually remain buried in local JSON files. That’s fine for a solo experiment, but painful when you need to compare models, audit results, or reproduce someone else’s work.

Inspect is an open-source framework for running evaluations which runs locally on one's machine. Weights & Biases (WandB) are tools for sharing, tracking, and visualizing ML experiments and LLM evals in the cloud. Our integration—inspect_wandb—is the missing bridge: it lets Inspect stream evaluations, metrics, and traces directly into WandB, providing a single source of truth, searchable dashboards, and reproducible runs.

Core Features

Connecting Models and Weave

The integration leverages two complementary WandB tools, each serving distinct but interconnected purposes.

When running LLM evals, WandB Models is primarily useful for reproducibility and tracking. Every evaluation becomes a fully logged Run complete with system state, git information, command history, and configurable file uploads. When you need to reproduce a result months later, all the context is preserved.

Overview of an evaluation run, easy to reproduce with the context preserved.

WandB Weave, delivers analysis and comparison capabilities. It transforms individual evaluation tasks into a searchable, filterable, comparable dataset. Rather than looking through local log files or sharing log files via Google Drive, teams can query their entire evaluation history through an intuitive, dedicated interface.

Evals tab in Weave shows all the tasks with status, metadata and metrics if available.

This dual approach lets each tool handle its own tasks while connected through a unified run ID. In the Models UI, clicking a run’s trace opens the Weave traces for the associated evaluation tasks—and you can jump back from Weave to Models by searching for the run ID.

Cross-Evaluation Analysis

Cross-evaluation analysis has been a frequent pain-point for Inspect users, typically requiring custom scripts and database queries. inspect_wandb addresses this pain point with the following features.

The filtering system supports combined queries on ...

Posts

Wikitag Contributions

Comments