Top postsTop post
Around 10-20 years ago, people realized that a bunch of aliens exist & that we can build portals which can summon them. The bigger the portal you make, the bigger the alien it can admit. These portals and aliens have several characteristics: * We can't go through the portals into...
TL;DR We stress-test alignment midtraining (AMT) across model and token budget scales. Our results suggest that midtraining cannot tackle the hard problems of AI alignment—namely distributional shift and reward underspecification in the presence of imperfect data. For instance, we test whether midtrained motivations are robust to finetuning which elicits competing...
Tl;dr: There should be a clear, well-designed eval on activation faithfulness: whether the model activations a monitor reads remain a trustworthy surface when the model suspects/knows it is being monitored. We provide an appendix with preliminary results towards such an eval in case people want to build off of it,...
TL;DR: This post presents three case studies of automated alignment research runs at Arcadia Impact. We use these case studies to emphasise the following takeaways: 1. It is hard to parse auto-research runs! Each run produces a couple of hundred pull requests of jargon-dense agent output. When researchers look through...
This is the first post in a two-part sequence regarding the state of defending from data poisoning attacks. We describe some methods for determining whether a model has been backdoored and how to find the trigger. In the next post, we will discuss all of the ways we think the...
Context: We are the ‘model motivations’ team at Arcadia Alignment. We aim to build a science of ‘model intentions’, unifying insights from personas and other empirical evidence. This is an informal research note that has come out of the first 2-3 weeks of exploratory work. In this post, we’ll outline...