otto.barten

(This post is an update from a previous one here.) The Existential Risk Observatory has been interested in public awareness of AI existential risk since its inception over five years ago. We started surveying public awareness in December 2022, including by asking the following open question: "Please list three events,...

Epistemic status: drafted relatively quickly with some LLM help. Thought about quite a bit. Related to current research. Summary Traditionally, AI x-risk discourse has mostly focused on the threat model: a single system recursively self-improves, gets a decisive strategic advantage, and disempowers humanity essentially unilaterally. Recently, a second threat model...

Some say, a pause is possible, it merely depends on political will. Others say, tech progress can never be stopped. Maybe the truth lies in the middle? Maybe we can 'catch all GPUs' (that is, effectively enforce a pause or other international agreement) as long as training run sizes are...

Open internship position + call for collaborations on threat model-dependent alignment, governance, and offense/defense balance At the Existential Risk Observatory, we're currently carrying out a project called Solving the Right Problem: Towards Researchers Consensus on AI Existential Threat Models, together with MIT FutureTech and FLI. Although many leading researchers agree...

Today, PauseAI and the Existential Risk Observatory release TakeOverBench.com: a benchmark, but for AI takeover. There are many AI benchmarks, but this is the one that really matters: how far are we from a takeover, possibly leading to human extinction? In 2023, the broadly coauthored paper Model evaluation for extreme...

The Existential Risk Observatory has been interested in public awareness of AI existential risk since its inception over five years ago. We started surveying public awareness in December 2022, including by asking the following open question: "Please list three events, in order of probability (from most to least probable), that...

This is a commentary on a paper by RAND: Can Humans Devise Practical Safeguards That Are Reliable Against an Artificial Superintelligent Agent? Over a decade ago, Eliezer Yudkowsky famously ran the AI box experiment, in which a gatekeeper had to keep a hypothetical ASI, played by him, inside a box,...