Daniel Tan's Shortform

Daniel Tan

This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

From what I can tell (small sample size), lab safety employees seem to mostly understand the reasons why working on alignment / control may accelerate capabilities and thus be bad.

Despite this, there may be reasons why they continue with their work, e.g:

They believe it's good for safety for their chosen organization to win the race.
They don't believe the above, but want to build political capital / influence within the organization, such that when "the right time" comes they can take a high-leverage action.

(Note: I disagree with both of these opinions)

Some other possible reasons:

Safety is simply more good than acceleration is bad
Working on safety has an internal policy impact, e.g. create evals that show a model is unsafe and block training or release
Working on safety has an external policy impact, e.g. study of CoT monitoring caused OpenAI and other labs to not train on CoT
Safety work now prevents imminent takeover, e.g. if models 1-2 generations in the future will be x-risky and alignment is easy
Safety work can prevent obvious misalignment, which makes it easier to observe subtle alignment problems. (... (read more)

Daniel Tan's Shortform

2

Daniel Tan's Shortform

2