x

LESSWRONG
LW

farrelmahaztra

Subscribe

Message

Software Engineer @ RemNote

DM at: https://x.com/fmztra

12

1

1y

Systematic Sandbagging Evaluations on Claude 3.5 Sonnet

This was the project I worked on during BlueDot Impact's AI Safety Fundamentals Alignment course, which expands on findings from Meinke et al's "Frontier Models are Capable of In-context Scheming". Summary * A dataset of 1,011 variations of the sandbagging prompt ("consequences") from Meinke et al were generated using Claude...

Feb 14, 202513

farrelmahaztra

Subscribe

Message

Software Engineer @ RemNote

DM at: https://x.com/fmztra

12

1

1y

Systematic Sandbagging Evaluations on Claude 3.5 Sonnet

This was the project I worked on during BlueDot Impact's AI Safety Fundamentals Alignment course, which expands on findings from Meinke et al's "Frontier Models are Capable of In-context Scheming". Summary * A dataset of 1,011 variations of the sandbagging prompt ("consequences") from Meinke et al were generated using Claude...

Feb 14, 202513

Systematic Sandbagging Evaluations on Claude 3.5 Sonnet

farrelmahaztra

1y

This was the project I worked on during BlueDot Impact's AI Safety Fundamentals Alignment course, which expands on findings from Meinke et al's "Frontier Models are Capable of In-context Scheming".

Summary

A dataset of 1,011 variations of the sandbagging prompt ("consequences") from Meinke et al were generated using Claude 3.5 Sonnet, and used to run 7 sandbagging evaluations (2 with the full dataset, 5 with a sample due to cost constraints).
With goal-directed chain-of-thought prompting, Claude 3.5 Sonnet demonstrates systematic sandbagging behavior that varies based on consequences.
Performance shows clear bimodality with goal-directed reasoning (mean 79%, σ=0.17, bimodality coefficient 0.707). The reason for this bimodality is unclear.
Strongest sandbagging triggers (18-24% accuracy) consistently relate to model autonomy

... (read more)

13