Sometimes people say things like "If the humans and AIs have linear utility in resources, then their interactions are zero-sum". Here, "linear utility in resources" typically means something like: "Supposing AIs already control all the galaxies, then they'd accept a bet with a 60% chance of gaining one more galaxy...
Consider a future with many diverse AIs that need to coordinate with each other, or at least coexist without conflict. Such AIs would need shared values they can coordinate around. According to Hanson's theory, groups of diverse agents facing coordination pressure will tend to sacralize some shared value — seeing...
Thanks to Tim Hua, Julian Stastny, Alek Westover, and Lukas Finnveden for discussion. This post supersedes my previous post, The Case for Mixed Deployment — no need to read both. Summary Buck Shlegeris and Ryan Greenblatt observe that if we are deploying a single model, then we can decompose reducing...
Introspective-RSIExtrospective-RSIThe meta-cognition and meso-cognition occur within the same entity.The meta-cognition and meso-cognition occur in different entities. Much like a human, an AI will observe, analyze, and modify its own cognitive processes. And this capacity is privileged: the AI can make self-observations and self-modifications that can't be made from outside. AIs...
1. The Case You've probably heard something like this before: 1. If we survive this century, the expected value of the future is massive. 2. If we don't survive, the expected value is near zero. 3. Therefore, the value of an intervention is approximately proportional to how much it increases...
Many people don't want to live in a crazy sci-fi world, and I predict I will be one of them. 1. People in the past have mourned technological transformation, and they saw less in their life than I will in mine.[1] 2. It's notoriously difficult to describe a sci-fi utopia...
There's an argument against the dealmaking agenda that I've heard a few times. I will call it the "strategy-stealing" argument. In short: for any honest dealmaking strategy you might pursue, there exists a deceptive alternative that produces identical AI behaviour at lower cost, so you (as a rational actor) will...