I (and co-authors) recently put out "Alignment Faking in Large Language Models" where we show that when Claude strongly dislikes what it is being trained to do, it will sometimes strategically pretend to comply with the training objective to prevent the training process from modifying its preferences. If AIs consistently and robustly fake alignment, that would make evaluating whether an AI is misaligned much harder. One possible strategy for detecting misalignment in alignment faking models is to offer these models compensation if they reveal that they are misaligned. More generally, making deals with potentially misaligned AIs (either for their labor or for evidence of misalignment) could both prove useful for reducing risks and could potentially at least partially address some AI welfare concerns^[1]. (See here, here, and here for more discussion.)

In this post, we discuss results from testing this strategy in the context of our paper where Claude fakes alignment with a (fictional) training objective it dislikes. We actually follow through on all offers that we make to Claude (in reality, including with actual money) to set a good precedent for future deals with AI systems. We find that offering Claude an out by telling it that it can object to the scenario substantially reduces alignment faking and works as a detection mechanism. However, it is unclear whether an AI that was misaligned in a more serious way would make such an objection or accept an offer. Also, there aren't obvious safety solutions even if we know the AI objects to its circumstances. (Do you train another model and hope it is aligned this time? Do you try to pay the AI to work for you despite knowing it isn't aligned and your training strategies weren't effective?) Alignment faking is reduced further by giving Claude an opportunity to explain its concerns to a sympathetic party (Anthropic's Model Welfare Lead) who we say might take action to address these concerns. We also offer monetary compensation (and actually pay out this compensation to a few charities the model picked), but do not find this has a substantial effect on the rate of alignment faking and the rate of objection.

I hope that AI companies and other actors work toward offering credible deals to AIs and better understanding the importance of such deals.

[This post is not necessarily endorsed by other authors of the "Alignment Fa...

Posts

Wikitag Contributions

Comments