I think 1 is true. This is only a single, quite obscure, factual recall eval. It's certainly possible to have regressions on some evals across model versions if you don't optimize for those evals at all.

Wrt point 2 -> here is the plot of how often the models guess, versus say they do not know, on the same dataset. My understanding is that the theory in point 2 would have predicted a much more dramatic drop in GPT-5.2?

Daniel Paleka16d

Here's on the Sonnet size class for now, nothing very interesting...

Daniel Paleka17d

Based on the plot below, my intuition is that GPT-5 and GPT-5.1 had some pretraining data added to the original base pretraining dataset (or base model) dating back to GPT-4o; and GPT-5.2 is something different. I did this experiment a while back.

Daniel Paleka3moQuick Take

On the apocalyptic residual, or: the secret third way to save the world.

Bostrom's Vulnerable World Hypothesis says our world is vulnerable. The most commonly discussed threat model are his "Type-1 vulnerabilities": destructive technologies such as bioengineered pandemics.

There are three steps required for our world to be destroyed in this way:

The dangerous technology exists and is available to many people;
There are no defensive technologies to protect us;
A number of people want to destroy the world; we call those people the apocalyptic residual.

If all three are satisfied at any point in time, we die.

Most work on AI safety has been focused on (1) and (2). To be precise, here is a non-exhaustive list of... (read 442 more words →)

Daniel Paleka3moQuick Take

Four views of AI automation: model, researcher, lab, economy

Every serious AI lab wants to automate themselves. I believe this sentence to hold predictive power over AI timelines and all other predictions about the future. In particular, I believe taking the AI-lab-centric view is the right way to think about automation.

In this post, I want to present the different levels of abstraction at which AI automation can be thought of:

Model: The model is being optimized by the lab. The right measure of acceleration is how much a model can autonomously do on tasks of a given purpose; ideally, research on future versions of itself. Examples of metrics are the METR time horizons study

... (read 498 more words →)

Daniel Paleka3mo*Quick Take

AIs being bad at AI research says nothing about acceleration

Lots of people are trying to make AI good at AI research. How are they doing?

One way to measure this is to assume AIs are gradually doing more and more complex tasks independently. Eventually it would grow to doing whole research projects. Something like this, but for software engineering instead of research, is captured in the METR "time horizons" benchmark.

I think extending this line of thinking to forecasting progress in AI research is wrong. Instead, a better way to accelerate AI research for the time being is combining AI and people to do research together, in a way that uses the complementary strengths... (read 467 more words →)

Daniel Paleka3moQuick Take

Slow takeoff for AI R&D, fast takeoff for everything else

Why is AI progress so much more apparent in coding than everywhere else?

Among people who have "AGI timelines", most do not set their timelines based on data, but rather update them based on their own day-to-day experiences and social signals.

As of 2025, my guess is that individual perception of AI progress correlates with how closely someone's daily activities resemble how an AI researcher spends their time. The reason why users of coding agents feel a higher rate of automation in their bones, whereas people in most other occupations don't, is because automating engineering has been the focus of the industry for a while... (read 427 more words →)

102

•••

A/B testing could lead LLMs to retain users instead of helping them

Daniel Paleka

3mo

OpenAI’s updates of GPT-4o in April 2025 famously induced absurd levels of sycophancy: the model would agree with everything users would say, no matter how outrageous. After they fixed it, OpenAI released a postmortem; and while widely discussed, I find it curious that this sentence received little attention:

Similarly, the A/B tests seemed to indicate that the small number of users who tried the model liked it.

In this post, I argue that A/B testing will implicitly optimize models for user retention; and propose ways to measure whether AIs try to retain the user in ways other than just being helpful to the user.

The labs use A/B testing to decide which updates to roll

... (read 947 more words →)

Daniel Paleka's Shortform

Daniel Paleka

10mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Daniel Paleka10mo*Quick Take

GPT-4o's drawings of itself as a person are remarkably consistent: it's more or less always a similar-looking white male in his late 20s with brown hair, often sporting facial hair and glasses, unless you specify otherwise. All the men it generates might as well be brothers. I reproduced this on two ChatGPT accounts with clean memory.

On the contrary, its drawings of itself when it does not depict itself as a person are far more diverse: a wide range of robot designs and abstract humanoids, often featuring OpenAI logo as a head or on the word "GPT" on the chest.

Replying toWhat are the strongest arguments for very short timelines?

Daniel Paleka11mo

What are the strongest arguments for very short timelines?

I think the labs might well be rational in focusing on this sort of "handheld automation", just to enable their researchers to code experiments faster and in smaller teams.

My mental model of AI R&D is that it can be bottlenecked roughly by three things: compute, engineering time, and the "dark matter" of taste and feedback loops on messy research results. I can certainly imagine a model of lab productivity where the best way to accelerate is improving handheld automation for the entirety of 2025. Say, the core paradigm is fixed; but inside that paradigm, the research team has more promising ideas than they have time to implement and try out on smaller-scale... (read more)

You should delay engineering-heavy research in light of R&D automation

Daniel Paleka

tl;dr: LLMs rapidly improving at software engineering and math means lots of projects are better off as Google Docs until your AI agent intern can implement them.

Implementation keeps getting cheaper

Writing research code has gotten a lot faster over the past few years. Since 2021 and OpenAI Codex, new models and tools such as Cursor built around them have saved myself more and more time on coding every year.

This trend is accelerating fast: AI agents using Claude-3.5-Sonnet and o1-preview can do tasks that take ML researchers up to 2 hours of coding. This is without considering newer models such as o3, which do 70% on SWE-bench out of the box.
Yet this progress remains... (read 1388 more words →)

Replying toUsing an LLM perplexity filter to detect weight exfiltration

Daniel Paleka1y

Using an LLM perplexity filter to detect weight exfiltration

N = #params, D = #data

Training compute = const .* N * D

Forward pass cost (R bits) = c * N, and assume R = Ω(1) on average

Now, thinking purely information-theoretically:
Model stealing compute = C * fp16 * N / R ~ const. * c * N^2

If compute-optimal training and α = β in Chinchilla scaling law:
Model stealing compute ~ Training compute

For significantly overtrained models:
Model stealing << Training compute

Typically:
Total inference compute ~ Training compute
=> Model stealing << Total inference compute

Caveats:
- Prior on weights reduces stealing compute, same if you only want to recover some information about the model (e.g. to create an equally capable one)
- Of course, if the model is producing much fewer than 1 token per forward pass, then model stealing compute is very large

Evaluating Superhuman Models with Consistency Checks

Daniel Paleka

Daniel Paleka, Lukas Fluri

Consider the following two questions:

In both cases, the ground truth is not known to us humans. Furthermore, in both cases there either already exist superhuman AI systems (as in the case of chess), or researchers are actively working to get to a superhuman level (as in the case of forecasting / world modeling). The important question is of course:

How can we evaluate decisions made by superhuman models?

In a new paper, we propose consistency / metamorphic testing as a first step towards extending the evaluation frontier. We test:

superhuman chess engine (Leela)
GPT-4 forecasting future events
LLMs making legal decisions

The first part of this post gives a short overview of... (read 2447 more words →)

My SERI MATS Application

Daniel Paleka

Thanks to Flo Dorner for feedback on the technical content in this post.

I have recently been accepted to the SERI ML Alignment Theory Scholars Program program, where applicants are paired with mentors. The application process was nonstandard: each mentor has a set of open-ended questions on safety research you need to answer in short essay form.
I am posting my application below. Reasons:

It might help future applicants to similar programs overcome writing anxiety;
I want to share the ideas from my answers.

I decided against editing my application post-hoc, to preserve a real "example application" for reference. There were some minor writing corrections. ^[1] Note: the answers were written in a single pass in a short amount... (read 2388 more words →)

LESSWRONG
LW

LESSWRONG
LW

Daniel Paleka

You should delay engineering-heavy research in light of R&D automation

A/B testing could lead LLMs to retain users instead of helping them

Evaluating Superhuman Models with Consistency Checks

My SERI MATS Application

Daniel Paleka

Daniel Paleka

A/B testing could lead LLMs to retain users instead of helping them

Daniel Paleka's Shortform

You should delay engineering-heavy research in light of R&D automation

Evaluating Superhuman Models with Consistency Checks

My SERI MATS Application

Daniel Paleka

You should delay engineering-heavy research in light of R&D automation

A/B testing could lead LLMs to retain users instead of helping them

Evaluating Superhuman Models with Consistency Checks

My SERI MATS Application

Daniel Paleka

Daniel Paleka

A/B testing could lead LLMs to retain users instead of helping them

Daniel Paleka's Shortform

You should delay engineering-heavy research in light of R&D automation

Evaluating Superhuman Models with Consistency Checks

My SERI MATS Application

On the apocalyptic residual, or: the secret third way to save the world.

Four views of AI automation: model, researcher, lab, economy

AIs being bad at AI research says nothing about acceleration

Slow takeoff for AI R&D, fast takeoff for everything else

The labs use A/B testing to decide which updates to roll

Implementation keeps getting cheaper