Zsolt Kelly-Bene

TLDR: The recent agent sandbox incidents suggest that models can know what an action would do while failing to treat those consequences as part of the current decision problem, which I’ll call a failure of consequence salience. One possible cause is a causal-horizon mismatch: agent training often presents trajectories that...

In IABIED, the load-bearing argument and, to me, the main contribution of the book, is about ASI motives. There’s more in there, but the thrust of the book is to argue for the truth of a specific conclusion about motives, namely that an ASI’s motives and goals would be completely...