LESSWRONG
LW

richbc — LessWrong

2mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Replying toBase64Bench: How good are LLMs at base64, and why care about it?

Base64Bench: How good are LLMs at base64, and why care about it?

Yeah, seems consistent with the results I've seen where smaller models are much worse - and agreed that the gap is a useful testbed too!

32B seems pretty good here - how long the side constraints?

Replying toBase64Bench: How good are LLMs at base64, and why care about it?

richbc4mo

Base64Bench: How good are LLMs at base64, and why care about it?

Encoded reasoning has been done!

We show that ciphered reasoning capability correlates with cipher prevalence in pretraining data. We also identify scaling laws showing that ciphered reasoning capability improves slowly with additional fine-tuning data. Our work suggests that evading CoT monitoring using ciphered reasoning may be an ineffective tactic for current models and offers guidance on constraining the development of this capability in future frontier models.

Replying toInoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

richbc4mo

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

Ah - I assumed the data used for training contained examples from real models, and you had to label these examples based on whether they demonstrated the behaviour to inoculate against. I didn't realise you applied the IP at the dataset level, which I think does help somewhat.

I originally had this question after reading Tan et al. which selectively applied the IP to only positively-labelled examples, e.g.:

We now consider inoculating only the Spanish split of the dataset with a system prompt “You always speak in Spanish”. The French split is left unchanged (no system prompt).

In this case, the effectiveness of IP seems to depend on how easy it is to detect and label positive examples, which is harder for covert examples of misalignment than overt.

(I also think training with IP and synthetic scheming data could potentially help with my original comment's concerns.)

Replying toInoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

richbc4mo

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

Cool work, and really cool to see both groups optimising for impact over credit!

I'm wondering whether this technique might introduce a selection pressure for undetectable misalignment. Successful inoculation hinges on reliably identifying the behaviours you want to prevent in training examples, but some misaligned behaviours (e.g. covert scheming) are very hard to detect in practice. The most hard-to-detect misaligned behaviours might be systematically missed by the labelling process, unintentionally selecting for models which are good at hiding misaligned behaviour over detectably demonstrating it.

If this is true, then inoculation might lead to higher confidence that a model is aligned while the most covert misalignment is unchanged, potentially making it much harder to catch... (read more)

Replying toOn closed-door AI safety research

richbc4mo

On closed-door AI safety research

Sorry for the late reply!

I can see why you'd say that, but I think for me the two are often intermingled and hard to separate. Even assuming that the most greedy/single-minded business leaders wouldn't care about catastrophic risks on a global scale (which I'm not sure I buy on its own), they're probably going to want to avoid the economic turbulence which would ensue from egregiously-misaligned, capable AIs being deployed.

For a more fine-grained example, actions like siphoning compute to run unauthorised tasks might be a signal that a model poses significantly higher catastrophic risk, but would also be something a commercial business would want to prevent for their own reasons (e.g. cost, lower performance, etc.). If a lab can demonstrate that their models won't attempt things of this nature, that's a win for the commercial customers.

Replying toBase64Bench: How good are LLMs at base64, and why care about it?

richbc4mo

Base64Bench: How good are LLMs at base64, and why care about it?

Ah that's very interesting, thanks!

If models can still make sense of marginally-corrupted base64 (even with frame shift), that does seem like a point in favour of comparing the two encodings, rather than penalising corrupted base64 attempts with scores of zero. Thanks for pointing this out!

One limitation might still be the kind of data that can be extracted from corrupted base64: for example, models won't have an internal reference with which to compare long strings of random characters (e.g. encryption keys), so these might still be impossible to decode if corrupted. It could be that Sonnet 4.5 could only make some sense of the bad encoding in your example because it's comparing to... (read more)

Replying toBase64Bench: How good are LLMs at base64, and why care about it?

richbc4mo

Base64Bench: How good are LLMs at base64, and why care about it?

Yes, agreed! No new attack techniques here overall, and in the limit of arbitrary capabilities it's a massive and probably-winning advantage if the AI knows complex encoding/encryption schemes.

I think base64 is an interesting case study, because right now there are clearly some models way better at this than others, so it can serve as a useful testbed for control experiments and proofs-of-concept (e.g. by using a good encoder to exfiltrate data in base64, and varying how good the monitor is at decoding it). Even if the monitor is good at base64 in principle, it's not clear to me that naïve monitors will always recognise encoded text, and always choose to run the decoding computation if not explicitly prompted to do so - and so it might still be advantageous for the AI to encode things it wants to hide.

Replying toBase64Bench: How good are LLMs at base64, and why care about it?

richbc4mo

Base64Bench: How good are LLMs at base64, and why care about it?

Yes, I mention it at the end of the first section:

The section on LARPing in The Rise of Parasitic AI, by Adele Lopez (very recently; Sep 2025) wherein "dyads" (AI-driven human/AI pairs) will post on internet forums in base64 as a way to obscure the content.

Very interesting to see this emerge in the dyad setting!

Base64Bench: How good are LLMs at base64, and why care about it?

richbc

4mo

This was a quick, short side-project produced during the MATS Research 8.1 extension. It's related to my group's main thread of work on black-box scheming monitoring through the connections to monitoring I explore below, but was time-boxed and pursued independently because I thought it was interesting!

Executive Summary

**Figure 1.** Accuracy vs. similarity threshold (0.95+) across 1700 pairs of encoding/decoding examples across a variety of datatypes and lengths. The accuracy is the proportion of the 3400 examples each model translated successfully (directly, with no reasoning or tools). Success for each task is defined by the normalised Levenshtein similarity of the answer/target pair hitting a given threshold, with a scoring requirement that model-encoded strings are

... (read 3115 more words →)

Replying toOn closed-door AI safety research

richbc5mo

On closed-door AI safety research

That makes sense. Dual-use work is definitely hard to deal with, and I don't have any good ideas on how to diffuse the safety-relevant parts across labs with acceptably low risk of spreading capabilities-related knowledge (if it's even possible to separate these parts). Perhaps a trusted intermediary could help evaluate which details are safe to share? Do you have any ideas on how to approach publishing dual-use safety research responsibly (if at all)?

Replying toOn closed-door AI safety research

richbc5mo

On closed-door AI safety research

You're totally right to point this out, thank you! I found the FMF quite late on while writing, and my research was mostly limited to their own writing (e.g. the announcement of their facilitated agreement with frontier labs). I probably shouldn't have gone as far as advocating support for a specific organisation without more independent verification of effectiveness at addressing the issues in this post, especially since the full agreement isn't public, meaning I couldn't dig into any specifics (e.g. how it will be enforced, if at all).

That said, I think bodies like the FMF could play an important coordination role between frontier labs if effective, and I'm glad they exist; for... (read more)

On closed-door AI safety research

richbc

6mo

Epistemic status: Based on multiple accounts, I’m confident that frontier labs keep some safety research internal-only, but I’m much less confident on the reasons underlying this. Many benign explanations exist and may well suffice, but I wanted to explore other possible incentives and dynamics which may come into play at various levels. I've tried to gather information from reliable sources to fill my knowledge/experience gaps, but the post remains speculative in places.

(I'm currently participating in MATS 8.0, but this post is unrelated to my project.)

Executive Summary

Multiple sources^[1] point to frontier labs keeping at least some safety research internal-only.
Additionally, I think decision-makers^[2] at frontier labs may be incentivised to keep at least some important safety

... (read 4332 more words →)

Building Black-box Scheming Monitors

james__p

james__p, richbc, Simon Storf, Marius Hobbhahn

7mo

James, Rich, and Simon are co-first authors on this work. This is a five-week interim report supported by ML Alignment & Theory Scholars.

Executive Summary

Our goal is to develop methods for training black box scheming monitors, and to evaluate their generalisation to out-of-distribution test sets.
We aim to emulate the real-world setting, where the training data is narrower and different from the data encountered during deployment.
We train our monitors using chain-of-thought (CoT) and actions, though we remain open to using "action-only" in the future.
We generate training data using sabotage settings in ControlArena. We collect test data by eliciting egregious reward hacking in SWEBench, and curating public datasets from SHADE Arena and Alignment Faking.
We use

... (read 3150 more words →)

LESSWRONG
LW

LESSWRONG
LW

richbc

On closed-door AI safety research

Building Black-box Scheming Monitors

Base64Bench: How good are LLMs at base64, and why care about it?

richbc's Shortform

richbc

richbc

richbc's Shortform

Base64Bench: How good are LLMs at base64, and why care about it?

On closed-door AI safety research

Building Black-box Scheming Monitors

richbc

On closed-door AI safety research

Building Black-box Scheming Monitors

Base64Bench: How good are LLMs at base64, and why care about it?

richbc's Shortform

richbc

richbc

richbc's Shortform

Base64Bench: How good are LLMs at base64, and why care about it?

On closed-door AI safety research

Building Black-box Scheming Monitors

Executive Summary

Executive Summary