Alexander Gietelink Oldenziel

Towards Developmental Interpretability

Developmental interpretability is a research agenda that has grown out of a meeting of the Singular Learning Theory (SLT) and AI alignment communities. To mark the completion of the first SLT & AI alignment summit we have prepared this document as an outline of the key ideas. As the name suggests, developmental interpretability (or "devinterp") is inspired by recent progress in the field of mechanistic interpretability, specifically work on phase transitions in neural networks and their relation to internal structure. Our two main motivating examples are the work by Olsson et al. on In-context Learning and Induction Heads and the work by Elhage et al. on Toy Models of Superposition. Developmental interpretability studies how structure incrementally emerges through phase transitions during training. Mechanistic interpretability emphasizes features and circuits as the fundamental units of analysis and usually aims at understanding a fully trained neural network. In contrast, developmental interpretability: * is organized around phases and phase transitions as defined mathematically in SLT, and * aims at an incremental understanding of the development of internal structure in neural networks, one phase transition at a time. The hope is that an understanding of phase transitions, integrated over the course of training, will provide a new way of looking at the computational and logical structure of the final trained network. We term this developmental interpretability because of the parallel with developmental biology, which aims to understand the final state of a different class of complex self-assembling systems (living organisms) by analyzing the key steps in development from an embryonic state.[1] In the rest of this post, we explain why we focus on phase transitions, the relevance of SLT, and how we see developmental interpretability contributing to AI alignment. > Thank you to @DanielFilan, @bilalchughtai, @Liam Carroll for reviewing early drafts of this

194Jul 12, 2023

Alexander Gietelink Oldenziel

Message

" (...) the term technical is a red flag for me, as it is many times used not for the routine business of implementing ideas but for the parts, ideas and all, which are just hard to understand and many times contain the main novelties."

...

6472

198

1054

Announcement: Iliad Intensive + Iliad Fellowship

Iliad is proud to announce that applications are now open for the Iliad Intensive and the Iliad Fellowship! These programs, taken together, are our evolution of the PIBBSS × Iliad Research Residency pilot. The Iliad Intensive will cover taught coursework, serving as a widely comprehensive introduction to the field of...

Feb 2068

Announcing: Agent Foundations 2026 at CMU

Iliad is now opening up applications to attend Agent Foundations 2026 at CMU! Agent Foundations 2026 will be a 5-day conference (of ~35 attendees) on fundamental, mathematical research into agency. It will take place March 2–6, 2026 at Carnegie Mellon University in Pittsburgh, Pennsylvania, and will be the third conference...

Dec 5, 202560

NATO is dangerously unaware that its military edge is slipping

"(...) war comes not when the forces of freedom are strong, but when they are weak. It is then that tyrants are tempted." - Ronald Reagan NATO faces its gravest military disadvantage since 1949, as the balance of power has shifted decisively toward its adversaries in the Era of Drone...

Nov 24, 2025113

Proceedings of ILIAD: Lessons and Progress

tl;dr This post is an update on the Proceedings of ILIAD, a conference journal for AI alignment research intended to bridge the gap between the Alignment Forum and academia. Following our successful first issue with 9 workshop papers from last year's ILIAD conference, we're launching a second issue in association...

Apr 28, 202578

Announcing ILIAD2: ODYSSEY

We are pleased to announce ILIAD2: ODYSSEY—a 5-day conference bringing together 100+ researchers to build scientific foundations for AI alignment. This is the 2nd iteration of ILIAD, which was first held in the summer of 2024. ***Apply to attend by June 1!*** * When: Aug 25-29, 2025 * Where: @Lighthaven...

Apr 3, 202580

Timaeus in 2024

> TLDR: We made substantial progress in 2024: > > * We published a series of papers that verify key predictions of Singular Learning Theory (SLT) [1, 2, 3, 4, 5, 6]. > * We scaled key SLT-derived techniques to models with billions of parameters, eliminating our main concerns around...

Feb 20, 2025100

Agent Foundations 2025 at CMU

We are opening applications to attend a 5 day agent foundations conference at Carnegie Mellon University. The program will include talks, breakout sessions, and other activities. Endlessly debate your favored decision theory, precommit to precommit, bargain with(in) yourselves, make friends across the multiverse, and remember: never give in to acausal...

Jan 19, 202590

Load More (7/39)

LESSWRONG
LW

LESSWRONG
LW

Alexander Gietelink Oldenziel

Alexander Gietelink Oldenziel

Alexander Gietelink Oldenziel

Towards Developmental Interpretability

Announcing Timaeus

Timaeus's First Four Months

Announcing ILIAD — Theoretical AI Alignment Conference

Alexander Gietelink Oldenziel

Announcement: Iliad Intensive + Iliad Fellowship

Announcing: Agent Foundations 2026 at CMU

NATO is dangerously unaware that its military edge is slipping

Proceedings of ILIAD: Lessons and Progress

Announcing ILIAD2: ODYSSEY

Timaeus in 2024

Agent Foundations 2025 at CMU

Towards Developmental Interpretability

Announcing Timaeus

Timaeus's First Four Months

Announcing ILIAD — Theoretical AI Alignment Conference

Announcement: Iliad Intensive + Iliad Fellowship

Announcing: Agent Foundations 2026 at CMU

NATO is dangerously unaware that its military edge is slipping

Proceedings of ILIAD: Lessons and Progress

Announcing ILIAD2: ODYSSEY

Timaeus in 2024

Agent Foundations 2025 at CMU