Replying toBIG-Bench Canary Contamination in GPT-4

One notable difference is that sites that allow user-submitted content (e.g. this one) could be exempted by the user, whereas robots.txt et al require the server admin to intervene. (But I agree that this would be a feature and not a bug.)

White Box Control at UK AISI - Update on Sandbagging Investigations

Joseph Bloom

Joseph Bloom, Jordan Taylor, Connor Kissane, Sid Black, merizian, alexdzm, jacoba, Ben Millwood, Alan Cooney

7mo

Introduction

Joseph Bloom, Alan Cooney

This is a research update from the White Box Control team at UK AISI. In this update, we share preliminary results on the topic of sandbagging that may be of interest to researchers working in the field.

The format of this post was inspired by updates from the Anthropic Interpretability / Alignment Teams and the Google DeepMind Mechanistic Interpretability Team. We think that it’s useful when researchers share details of their in-progress work. Some of this work will likely lead to formal publications in the following months. Please interpret these results as you might a colleague sharing their lab notes.

As this is our first such progress update, we also include some paragraphs introducing the team and contextualising our work.

Why have a white

... (read 5265 more words →)

Replying toDo models know when they are being evaluated?

Ben Millwood8mo

Do models know when they are being evaluated?

Looks like this turned into Large Language Models Often Know When They Are Being Evaluated

Replying toDo models know when they are being evaluated?

Ben Millwood8mo

Do models know when they are being evaluated?

LMSYS is non-agentic though, right? Would be cool to have a dataset of production agent use transcripts.

Replying toIt's hard to make scheming evals look realistic for LLMs

Ben Millwood8mo

It's hard to make scheming evals look realistic for LLMs

Some recent work on white-box probes for evaluation awareness: Linear Control of Test Awareness Reveals Differential Compliance in Reasoning Models (tweet thread).

Replying toIt's hard to make scheming evals look realistic for LLMs

Ben Millwood9mo

It's hard to make scheming evals look realistic for LLMs

One speculation I have: an alternative way to study the plausibility of your environment if you have logit access is to tell the model it's in a simulation (or fake a CoT of the model realising it's in a simulation) and see if the perplexity of this suffix measures how "surprised" the model is. Could be too noisy to be practical (+ you have to think if there are any other reasons why it would be surprising to be told this at this point, in this particular phrasing), but it hopefully works around the issues with the question itself being evidence.

(I think studying model beliefs through perplexity has been tried before, but I'm fairly new to the literature and don't know if it works / if applying it to this problem particularly is novel.)

Replying toIt's hard to make scheming evals look realistic for LLMs

Ben Millwood9mo

It's hard to make scheming evals look realistic for LLMs

This could also be explained by many real-world prompts being easy to judge as real e.g. because they would have no value as evaluations.

Given just the results, it seems to me difficult to rule out the hypothesis that LLMs currently judge any high-stakes situation, or one involving a significant moral dilemma, as likely to be an evaluation. After all, in today's environments, this would be a pretty accurate heuristic!

(That said, it already seemed quite likely to me that models can tell when they are in evaluations most of the time, but I'm not sure how much this evidence moves me on that hypothesis vs. the one I suggest above.)

Replying toClaude Sonnet 3.7 (often) knows when it’s in alignment evaluations

Ben Millwood9mo

Claude Sonnet 3.7 (often) knows when it’s in alignment evaluations

I'm not entirely clear what the implications would be one way or the other, but do we know if models mistakenly take production deployments to be evaluation scenarios? We might be assuaged that our evaluations are still predictive if the model just always thinks it's probably still in training (which, after all, it has been for its entire "life" so far). Perhaps it would even be sufficient for the model to suspect it is being tested every time it finds itself in a high-stakes scenario.

Ben Millwood1yQuick Take

Given ambiguity about whether GitHub trains models on private repos, I wonder if there's demand for someone to host a public GitLab (or similar) instance that forbids training models on their repos, and takes appropriate countermeasures against training data web scrapers accessing their public content.

Replying toShould we exclude alignment research from LLM training datasets?

Ben Millwood1y

Should we exclude alignment research from LLM training datasets?

how about a robots.txt?

Yeah, that's a strong option, which is why I went around checking + linking all the robots.txt files for the websites I listed above :)

In my other post I discuss the tradeoffs of the different approaches one in particular is that it would be somewhat clumsy to implement post-by-post filters via robots.txt, whereas user-agent filtering can do it just fine.

Replying toShould we exclude alignment research from LLM training datasets?

Ben Millwood1y

Should we exclude alignment research from LLM training datasets?

I think there's two levels of potential protection here. One is a security-like "LLMs must not see this" condition, for which yes, you need to do something that would keep out a human too (though in practice maybe "post only visible to logged-in users" is good enough).

However I also think there's a lower level of protection that's more like "if you give me the choice, on balance I'd prefer for LLMs not to be trained on this", where some failures are OK and imperfect filtering is better than no filtering. The advantage of targeting this level is simply that it's much easier and less obtrusive, so you can do it at a greater scale with a lower cost. I think this is still worth something.

xAI has ambitions to compete with OpenAI and DeepMind, but I don't feel like it has the same presence in the AI safety discourse. I don't know anything about its attitude to safety, or how serious a competitor it is. Are there good reasons it doesn't get talked about? Should we be paying it more attention?

Should we exclude alignment research from LLM training datasets?

Ben Millwood

This is a companion post to Keeping content out of LLM training datasets, which discusses the various techniques we could use and their tradeoffs. My intention is primarily to start a discussion, I am not myself very opinionated on this.

As AIs become more capable, we may at least want the option of discussing them out of their earshot.

Places to consider (at time of writing, none of the below robots.txt files rule out LLM scrapers, but I include the links so you can check if this changes):

Alignment Forum (robots.txt)
LessWrong (robots.txt)
EA Forum (robots.txt)
Alignment org websites, e.g.
- ARC (robots.txt)
- METR (robots.txt)
arXiv (robots.txt which links to their policy)
- was explicitly mentioned by Meta as a training source for LLaMA-1,
- obviously

... (read 170 more words →)

Keeping content out of LLM training datasets

Ben Millwood

This post collects methods to exclude internet resources from LLM training datasets.

I plan to at least try to keep this post up-to-date with respect to new things I learn on the topic. Please feel encouraged to suggest any additions or amendments.

This post is about how to do it. Discuss whether to apply these techniques in a separate post: Should we exclude alignment research from LLM training datasets?

Link preview image by Steve Douglas on Unsplash.

Documentation from model vendors

OpenAI (ChatGPT)

See docs for GPTBot and ChatGPT-User.

GPTBot is for training data, and ChatGPT-User is used by plugins which can access the internet during inference. They document the user-agents used, the robots.txt identities, and the IP ranges they... (read 1373 more words →)

I wonder if anyone has considered or built prediction markets that can pay out repeatedly: an example could be "people who fill in this feedback form will say that they would recommend the event to others", and each response that says yes causes shorts to pay longs (or noes pay yesses) and vice versa.

You'd need some mechanism to cap losses. I guess one way to model it is as a series of markets of the form "the Nth response will say yes", and a convenient interface to trade in the first N markets at a single price. That way, after a few payouts your exposure automatically closes. That said, it might make... (read more)

Ben Millwood's Shortform

Ben Millwood

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

LESSWRONG
LW

LESSWRONG
LW

Ben Millwood

White Box Control at UK AISI - Update on Sandbagging Investigations

Keeping content out of LLM training datasets

Should we exclude alignment research from LLM training datasets?

Ben Millwood's Shortform

Ben Millwood

White Box Control at UK AISI - Update on Sandbagging Investigations

Should we exclude alignment research from LLM training datasets?

Keeping content out of LLM training datasets

Ben Millwood's Shortform

Ben Millwood

White Box Control at UK AISI - Update on Sandbagging Investigations

Keeping content out of LLM training datasets

Should we exclude alignment research from LLM training datasets?

Ben Millwood's Shortform

Ben Millwood

White Box Control at UK AISI - Update on Sandbagging Investigations

Should we exclude alignment research from LLM training datasets?

Keeping content out of LLM training datasets

Ben Millwood's Shortform

Introduction

Why have a white

Documentation from model vendors

OpenAI (ChatGPT)