Matthias Dellago

We are starting AIXI Labs, an AI safety org focused on algorithmic information theory (AIT), continual reinforcement learning (RL), and in particular the eponymous AIXI. We aim to strengthen the technical case that developing artificial super intelligence (ASI) poses an X-risk while (in parallel) developing and prototyping theoretically-founded mitigations. (Already convinced? View open positions here.) Overview From the website: https://www.aixi.uk/ AIXI is the leading mathematical model of artificial superintelligence, representing the maximum theoretical limit of AI capabilities. Alongside the name’s many previous meanings, AIXI now also stands for the AI X-risk Initiative, though we usually just call ourselves AIXI Labs. We model AI risk factors and safety mitigations in terms of AIXI variants, and develop the means to translate them to real AI agents. This enables rigorous testing of both the risk factors and the safety mitigations. Why this approach Most AI research today sits in one of two categories: methods that are fast and mathematically clean in narrow settings, and methods that are practical for frontier applications but opaque to safety analysis. Our core focus is the third intersection: methods that are general enough to describe powerful agents and amenable to mathematical analysis to support rigorous safety claims. Aligning a hypothetical superintelligence requires a concerted effort in this historically neglected direction. Thus, we formally examine behaviors, capabilities, risks and mitigations for idealized agents in fully general environments. We aim to port the most promising methods to modern LLM-based agents, and test principled hypotheses on these agents. Since our analyses are based on a very general model of superintelligence, we gain confidence that any experimentally validated conclusions will generalize and scale to future AI systems. Strategy Our timelines to the arrival of ASI are highly uncertain, and we aim to reduce X-risk