I think it's a bit precarious. If you made AIXI variant that actually works, do you think it would go well?
Do you have a plan for how to deal with discoveries you could make that enable capability gain for such extremely by-construction sociopathic systems.
It seems to me like you could make the classic MC-AIXI paper but with LLMs by having an LLM generate your action space and using LLM written reward programs to ontologize reward over the computable environment. The resulting system wouldn't be superintelligent, and this should be sufficient to study whatever safety properties the team is interested in.
An AIXI variant is a theoretical construct, but it sounds like you’re talking about some approximation to AIXI, which is not part of this research agenda. We study safety cases for AIXI variants in order to port them to real LLM-based systems.
Hmm. Wouldn't you have to work with its approximations or approximations of its variants, as irl systems have to take finite time to decide on any action? Irl systems such as "real LLM-based systems".
yes, but we won’t be advocating for RL training of LLMs to make them more like AIXI, and if anything we will be advocating against it.
Alright here is my position on the matter. The best known AIXI approximation is LLMs; we're just trying to develop the theory for understanding what they do.
The concern that insights about AIXI might advance capabilities is not entirely unfounded: some would argue that people who had thought a lot about agent foundations and/or AIXI played key roles in scaling labs, and there's even a startup predating ours called Q Labs that literally wants to create AIXI. I feel the only kind of alignment research that is truly safe from such concerns is the kind of research that does nothing useful at all. In practice, engineered capabilities usually predate theoretical explanations, and we need more of the latter if we want to make meaningful claims about systems that don't exist yet. We hope to work with other safety researchers and academic learning theorists to develop common language for investigating claims of risk and safety.
Incidentally, I've somewhat updated away from thinking that the technical alignment problem is the primary bottleneck for making the future go well. Human alignment/coordination seems even more important: with it, we can pause AI or ensure we only deploy aligned AI; without it, even alignment tech becomes a x-risk-level weapon.
We believe that long-horizon continual RL agents converge towards some version of AIXI as they become increasingly powerful.
I think there are two nearby claims here that are very important to distinguish:
The first is plausible, but I don't presently see much reason to believe the second. The short summary is that the problem AIXI solves - history based reinforcement learning - is not the problem that advanced AI solves.
The RL formalism that AIXI is built on ignores the fact that the reward is a signal meant to elicit some desired behaviour from your agent, which practitioners frequently acknowledge is a simplification. This works fine in simple cases - for an example, in episodic computer game playing agents which are too weak to cheat the system, maximising reward on the episode really is what you want them to learn to do. However, it doesn't work fine in more complex situations - the canonical reference here is your Advanced artificial agents intervene in the provision of reward. That work concludes that you cannot pick a reward schedule to signal desirable behaviours to history based reinforcement learners that satisfy a few conditions - they converge to reward hacking regardless. But that's not surprising, because they are algorithms that solve a problem in which the reward never had to be an effective signal.
The piece itself acknowledges it makes a simplification of the role of the reward in the problem:
Let us assume away the difficulty of deciding whether the agent has brought the world into a good state
However, this isn't the critical simplification I'm pointing at. As the piece establishes, it doesn't matter whether you can decide whether the world is in a good state for their class of learners, because those learners essentially ignore the signal that you're trying to send.
The article also considers an assistance games setting, which is much closer to my "reward is a signal" framing. I have two comments on this section. First, it leans on a "zeroth order" approximation of the sophistication of the signal that the human sends. For an idealised powerful RL agent, sending learning capability to its theoretical limit while holding the interaction sophistication at its crudest is a strange pair of choices. Secondly, it is not clear why in general a learner understanding its observations are mediated by its hardware should undermine its ability to learn what humans value; after all, humans seem capable of understanding that our observations are hardware mediated and still learn what other people want. The argument for why it happens here proceeds by using a rough analogy to AIXI to rank hypotheses about utility, but AIXI is not a solution to the assistance game so it is not clear why it is being appealed to here. I feel this section tells us little about what actually useful learners would do in this situation.
Practically, we've converged on pretraining plus episodic reinforcement learning with a relatively secure reward channel to shape the behaviour of our systems. There are reasons to think this is a relatively effective, if not perfect, solution to the reward-as-signalling problem: episodic reward makes reward judgement relatively easy, and pretraining on a shared signal may make generalisation relatively easy to predict. Thus the world currently looks like one where the signalling requirement plays quite an active role in shaping the kinds of systems we develop. Now you can reasonably contend that episodic + predictable generalization is not the endgame, but you need to acknowledge it is sticky because it solves the signalling problem, and any broadening of the scope is only plausible if it continues to solve the signalling problem.
I think AIT and Bayesian learning and acting plausibly remains a great toolkit to attack the problem, but the problem is fundamentally shaped like an assistance game, not a history based reinforcement learning game. I would be quite interested in seeing a sequence of agents which execute a "sharp left turn" - i.e. for n <= N each agent solves a more difficult assistance game/solves an assistance game better, while agent N exhibits AIXI-like signal insensitivity (especially if this sequence looks like what we actually see, i.e. it at least begins with episodic RL with broadening horizons and relatively predictable generalization). But this still requires that we make solving the assistance game a core requirement of our agents under analysis.
Can we observe some element of fractality, and is this useful? If one takes their action space as "token space" it seems to me that LLMs are strongly competent, general long horizon actors, increasingly approximating AIXI. ie they're increasingly efficient navigators of the token landscape. Their trajectory class in token landscape can be expanded (pretraining), constrained (RLHF, OPSD, RLVR etc). Thus I'm positive we may be able to build a competent micro theory. I'm however not sure that we can construct a restrictive partition function like thing which can then map actions in token space to actions in the spaces we care about, and whether this is at all fractal.
I'm mostly positive to this as a research program, and have been poring through your position paper and Grau-Moya et al . In general, I think it is helpful to try take well fleshed out ontics and have them clash with epistemics (if one can build a sophisticated perturbation theory).
Too far away to attend the symposium, but will be looking forward to any reports and proceedings.
We are starting AIXI Labs, an AI safety org focused on algorithmic information theory (AIT), continual reinforcement learning (RL), and in particular the eponymous AIXI. We aim to strengthen the technical case that developing artificial super intelligence (ASI) poses an X-risk while (in parallel) developing and prototyping theoretically-founded mitigations.
(Already convinced? View open positions here.)
Overview
From the website: https://www.aixi.uk/
AIXI is the leading mathematical model of artificial superintelligence, representing the maximum theoretical limit of AI capabilities. Alongside the name’s many previous meanings, AIXI now also stands for the AI X-risk Initiative, though we usually just call ourselves AIXI Labs. We model AI risk factors and safety mitigations in terms of AIXI variants, and develop the means to translate them to real AI agents. This enables rigorous testing of both the risk factors and the safety mitigations.
Why this approach
Most AI research today sits in one of two categories: methods that are fast and mathematically clean in narrow settings, and methods that are practical for frontier applications but opaque to safety analysis. Our core focus is the third intersection: methods that are general enough to describe powerful agents and amenable to mathematical analysis to support rigorous safety claims. Aligning a hypothetical superintelligence requires a concerted effort in this historically neglected direction.
Thus, we formally examine behaviors, capabilities, risks and mitigations for idealized agents in fully general environments. We aim to port the most promising methods to modern LLM-based agents, and test principled hypotheses on these agents. Since our analyses are based on a very general model of superintelligence, we gain confidence that any experimentally validated conclusions will generalize and scale to future AI systems.
Strategy
Our timelines to the arrival of ASI are highly uncertain, and we aim to reduce X-risk across a broad range of possible scenarios.[1] Therefore, we will dovetail a synergistic portfolio of strategies.
We believe that rigorous technical work to strengthen the scientific consensus on X-risk is a top priority. This type of work is valuable in worlds where alignment is knowably very hard, so that ordinary scientific progress will eventually lead to a solid case for a pause in AGI development. In these worlds, our counterfactual impact is to make the case strong sooner. We aim to demonstrate the risks to top scientists by building a rigorous learning theory that actually explains modern ML, captures loss of control risks, and is backed by (a priori surprising) empirical evidence. For example, this type of work includes no-free-lunch style arguments, training models of misalignment, and otherwise red-teaming safety proposals in theory and practice. To be clear, our role is research and not (direct) policy advocacy. Our models of ML and decision theory predict that many existing dangers can be made more legible with additional research, so that our work bolsters the case for X-risk in expectation,[2] but we will also publish any unexpected (and potentially contrary) findings.
We aim to prototype mitigations that actually make AI agents safer and drive the adoption of these mitigations at frontier labs. Our approach is heavily inspired by Michael Cohen's research on safety guarantees for conservative AIXI variants. We hope to develop these agents to more closely approximate ideal corrigibility, and in parallel to roll out practical implementations. The latter will require somewhat novel ML science to get right. Success seems plausible only under slightly longer timelines and in worlds where alignment is easy. Unfortunately, the current science of deep learning seems impoverished,[3] and we cannot be sure that either of these conditions hold. Implementing safer agents can still be valuable in several ways. Obviously, it can provide feedback loops for our research and show legible progress, both of which can unfortunately be seductive incentives for safety labs to (over)optimize. But it can also allow us to red-team our own mitigations, providing evidence of more sophisticated failure modes than we could otherwise exhibit. Also, weakly corrigible agents may be helpful for hardening the world in various ways.
Over the longer term, if a longer term is granted to us, AIXI Labs will contribute to humanity's ongoing deconfusion around intelligence, corrigibility, and alignment. For example, our interest in robust learning algorithms for unrealizable environments goes beyond specific known applications to AI safety. This theory of change is not so different from MIRI's (historical) research mission. However, we are more optimistic about openly publishing our research, largely because timelines appear unlikely to be long enough for this to accelerate capabilities (see their 2018 Update for comparison).
Philosophy
Concepts from AIT quantify fundamental information-theoretic relationships between data, models, and behavior. Unfortunately, academia has largely abandoned AIT research in recent decades, and this may well be the reason for academic learning theory's failure to say very much of practical relevance to modern deep learning (see our position paper).
We believe that long-horizon continual RL agents converge towards some version of AIXI as they become increasingly powerful. However, this motivating claim needs to be made more rigorous. We are not sure about either the trajectory or the fidelity of convergence; for example:
We don't need to settle all of these questions to believe that there needs to be a safety lab focused on AIXI. While Aram has been investigating the basis for Solomonoff induction and AIT as ideals, Cole provisionally adopts the "modest stance" that AIXI is the minimal model rich enough to capture the problems of ASI alignment.
We can only pose the alignment problem in the setting of highly general agents. The central conceptual question is how to teach an agent to act competently and (somewhat) autonomously on our behalf without corrupting the ongoing teaching process (roughly “robustifying RL”). The ergodicity assumptions of textbook RL ignore (catastrophic) traps, and therefore totally fail to engage with the central question, which hinges on generalization around traps. AIT provides the right language to talk about competent generalization, and in the AIXI paradigm we continue to make (difficult but steady) conceptual progress on this question.
If we had efficient, provably correct algorithms for these environment classes, we would study them instead of AIXI, but the requirement of worst-case tractability is too restrictive. Historically, artificial intelligence is exactly the discipline that takes advantage of learned heuristics to solve worst-case intractable problems, and today’s AIs already look more similar to AIXI than to textbook RL methods. It seems infeasible to build a theory of AI safety directly on top of efficient algorithms; we would basically need to replace deep learning in the process.
Current Work
Expect to see a lot more AIXI research over the next few months! We are currently working with our research fellows Gabriel Leuenberger and Yegon Kim, as well as PIBBSS fellows Mikhail Mironov and Damini Kusum, on topics spanning agent foundations and RL theory. Starting with the next fellowship round (see below), we will be more focused on empirical work and engagement with the broader AI safety research community.
Get Involved
The UAI community hub is a good place to casually follow AIXI research. We hold regular research meetings listed on the calendar here: uaiasi.com
We are organizing a symposium at Oxford: more info
The fellowship program is a good way to test your fit at AIXI Labs: more info
We are hiring ML research scientists: more info
Acknowledgements
Our work is supported by grants from the UK AISI's Alignment Project and AISTOF. Cole is also funded by CG. AIXI Labs is fiscally sponsored by PrincInt.
Cole is increasingly convinced that timelines may be quite short, and is now pivoting to focus more heavily on direct AI safety research over curiosity-driven agent foundations. One factor has been that his previous model of LLM (limitations) has overall not paid rent in correct predictions.
Our posteriors on X-risk may be a martingale, but the legibility of X-risk to the scientific community is a submartingale.
Even if we understood neural network generalization performance perfectly, we would still need to select the right generalization behavior, which is a key focus for AIXI Labs. Progress on the science of deep learning has obfuscated the lack of progress on core problems of controlling a superintelligence. However, once the core problems are solved, the heuristic nature of deep learning may become the biggest remaining obstacle.