TL;DR: This paper takes existing mathematical results to build the most general and rigorous case for why we should be very cautious about pushing optimization too far in General-Purpose AI systems, as it likely leads to catastrophic Goodhart failures, and ultimately loss of control.

Written by Antoine Maier, AI Security Researcher at the General-Purpose AI Policy Lab, and Aude Maier, PhD student at the École Polytechnique Fédérale de Lausanne (EPFL).