Paul Colognese

Character training can mitigate reward hacking, but can also make it harder to detect

Thanks to Johannes Treutlein, Jan Betley, Lennie Wells, Joey Yudelson, Arun Jose, Anna Marešová, Asvin Gothandaraman, and Clément Dumas for discussions and feedback. Edit (9th Oct 2026): I've shared the codebase along with some checkpoints. Edit (14th Oct 2026): We found and fixed two bugs after publication; the numbers hardly...

Sep 28140

Explaining the AI Alignment Problem to Tibetan Buddhist Monks

Introduction As part of an exchange being facilitated between religion and science, a group of academics has been asked to compile a short description of their greatest scientific achievement/discovery that will be translated into Tibetan and presented to Tibetan Buddhist scholars/monks.[1] I was also invited to contribute, but I sort...

Mar 7, 202420

Anomalous Concept Detection for Detecting Hidden Cognition

Thanks to Johannes Treutlein, Erik Jenner, Joseph Bloom, and Arun Jose for their discussions and feedback. Summary Monitoring an AI’s internals for features/concepts unrelated to the task the AI appears to be performing may help detect when the AI is performing hidden cognition. For example, it would be very suspicious...

Mar 4, 202424

Hidden Cognition Detection Methods and Benchmarks

Thanks to Johannes Treutlein for discussions and feedback. Introduction An AI may be able to hide cognition that leads to negative outcomes from certain oversight processes (such as deceptive alignment/scheming). Without being able to detect this hidden cognition, an overseer may not be able to prevent the associated negative outcomes...

Feb 26, 202422

Notes on Internal Objectives in Toy Models of Agents

Thanks to Jeremy Gillen and Arun Jose for discussions related to these ideas. Summary WARNING: The quality of this post is low. It was sitting in my drafts folder for a while, yet I decided to post it because some people found these examples and analyses helpful in conversations. I...

Feb 22, 202416

Internal Target Information for AI Oversight

Thanks to Arun Jose for discussions and feedback. Summary In this short post, we discuss the concept of Internal Target Information within agentic AI systems, arguing that agentic systems possess internal information about their targets. This information, we propose, can potentially be detected and interpreted by an overseer before the...

Oct 20, 202315

Potential alignment targets for a sovereign superintelligent AI

I'd like to compile a list of potential alignment targets for a sovereign superintelligent AI. By an alignment target, I mean something like what goals/values/utility function we might want to instill in a sovereign superintelligent AI (assuming we've solved the alignment problem). Here are some alignment targets I've come across:...

Oct 3, 202329

Paul Colognese

Paul Colognese

Character training can mitigate reward hacking, but can also make it harder to detect

Decision Transformer Interpretability

High-level interpretability: detecting an AI's objectives

Deception?! I ain’t got time for that!

Paul Colognese

Character training can mitigate reward hacking, but can also make it harder to detect

Decision Transformer Interpretability

High-level interpretability: detecting an AI's objectives

Deception?! I ain’t got time for that!

Character training can mitigate reward hacking, but can also make it harder to detect

Explaining the AI Alignment Problem to Tibetan Buddhist Monks

Anomalous Concept Detection for Detecting Hidden Cognition

Hidden Cognition Detection Methods and Benchmarks

Notes on Internal Objectives in Toy Models of Agents

Internal Target Information for AI Oversight

Potential alignment targets for a sovereign superintelligent AI