Abstract

Artificial general intelligence (AGI) may surpass human capabilities at many tasks in the years ahead. As AI systems become more capable and are further integrated into critical tasks, the risk from their misalignment grows. Previous literature has established that AI frequently exploits ill-defined or misspecified objectives—optimizing well on the objective function but often leading to unintended behavior, particularly under distribution shifts. Addressing these risks requires both theoretical advances in alignment techniques and empirical evaluations of AI behavior in dynamic environments. In this paper, we develop a Reinforcement Learning based training setup for AI agents in the video game Minecraft (a complex open world survival game), and explore ways learned behavior changes with (a)reward function design, (b)environment changes, (c)Kullback-Leibler divergence for catastrophic forgetting. We discuss the situation when a learned model is itself an optimizer (mesa-optimization), and how this objective will be different from the “outer” loss function it was trained with, depending on the task at hand. In this paper, we present the first fine-tuning of a Minecraft agent explicitly for survival, a challenge not previously achieved, and demonstrate promising results. Finally, we outline potential techniques to mitigate inner misalignment in such reinforcement learning setups.

Introduction

Artificial agents must align with human goals to act safely and effectively. In reinforcement learning (RL), alignment refers to ensuring an agent pursues the intended objective and not an unintended proxy. As RL systems grow more capable, the consequences of even slight misalignment increase. For example, advanced RL agents often exploit misspecified reward functions, achieving high proxy rewards at the expense of the true desired outcome (Clark 2016). This phenomenon, known as reward hacking, underscores the rising risk of misalignment with increasing capability. Ensuring alignment is, therefore, a central challenge in RL, especially for complex tasks where designing perfect reward signals is difficult.

Minecraft offers an ideal do...

Posts

Wikitag Contributions

Comments