Drawing inspiration from the ‘brain in a vat’ thought experiment, this blogpost investigates methods to simplify world models that remain agnostic to the agent under evaluation. This work was done together with Alec Boyd and Manuel Baltieri, with support from the UK ARIA Safeguarded AI programme and the PIBBSS Affiliateship programme. The post presents general ideas avoiding technicalities as much as possible, please refer to the preprint for formal details. Thanks to @Alexander Gietelink Oldenziel and @Lucas Teixeira for suggestions on this writeup.

Introduction

World models are typically conceived as tools for model-based reinforcement learning agents to improve their task performance. Interestingly, recent works are proposing a new perspective: world models can also be used as virtual environments where AI agents can be tested without real-world consequences. These simulated environments are ideal for evaluating how AI agents handle edge cases and respond to novel situations, potentially revealing safety issues. Furthermore, world models are also promising for building formal certificates of safety.

The efficacy of these approaches critically rely on the world model adequately representing real environments. To ensure that assessments of AI agent behaviour may transfer from simulations to a real-world setting, world models must faithfully reflect the real world’s structure and dynamics. Thus, a key challenge lies in dealing with the computational demands of high-fidelity simulations.

A crucial insight is that world models only need to incorporate the variables that make a difference for the agent. Building on this idea, we introduced a framework to investigate world modelling from the perspective of an external entity in charge of sandboxing an agent. The proposed framework does not require knowledge of the agent’s policy, and make no assumptions about the agent’s architecture and capabilities — being applicable to systems irrespective of how they were designed or trained.

The main take-home messages of this blogpost will be:

There are multiple world models that are indistinguishable from an agent’s perspective, each with their own strengths and weaknesses.
There is a fundamental efficiency-vs-interpretability^[1] trade-off: the most lightweight models are usually opaque, and the models that have direct interpretability usually are not the most compact

...

Posts

Wikitag Contributions

Comments