Epistemological status: existential AI safety suggestions as a set of bullet points TL;DR We need to be clear about what behavior we’re trying to train when we align AIs — aligned behavior is not the default for an LLM whose behavior is distilled from ours, nor for an RL-trained maximizer....
Anthropic are now actively using the approach to alignment often called “Alignment Pretraining” or “Safety Pretraining” — using Stochastic Gradient Descent on a large body of natural or synthetic documents showing the AI assistant doing the right thing in morally challenging situations. They tried this out, found it works well...
Epistemic status: examining the evidence and updating our priors TL;DR: Simulator Theory is now well supported by experimental results, and also really useful Continuing on from Part 1 of this post, in which I wrapped a little statistical formalism around Simulator Theory and then analyzed four of Owain Evans’ “surprising...
Epistemic status: examining the evidence and updating our priors TL;DR: Simulator Theory is now well supported by experimental results, and also really useful Over the last year or so we have had strong experimental evidence both that Simulator Theory is correct, and that it is valuable as a theoretical framework...
Epistemic status: We really need to know. (I also posted an opinionated answer.) There’s a well-known diagram from a tweet by Chris Olah of Anthropic: It would be marvelous to know what the actual difficulty is, out of those five labeled difficulty categories (ideally, exactly where it lies on that...
Epistemic status: We really need to know. TL;DR: Comparing person-years of effort, I argue that AI Safety seems harder than for steam engines, but probably less hard than the Apollo program or . I discuss why I suspect superalignment might not be super-hard. My has come down over the last...
Summary We argue that shaping RL exploration, and especially the exploration of the motivation-space, is understudied in AI safety and could be influential in mitigating risks. Several recent discussions hint in this direction — the entangled generalization mechanism discussed in the context of Claude 3 Opus's self-narration, the success of...