TLDR: This post explains two equilibria which regulate the level of AI safety: The first describes how much resources AI companies are willing to spend on AI safety work due to commercial incentives. The second one is about risk awareness and most notably affects government interventions for safety. Doing safety...
Follow-up to: The Macroagent Ontology (especially section 1.2 is a prerequisite) The previous post laid out the basic macroagent ontology. This post will look at some further concepts that are important lenses for optimizing macroagents, along with some ideas for each lens. We usually want to shape macroagents so that...
I think an important concept for sensibly modeling a part of the world is what I call a "macroagent", which abstracts over the similarity cluster of teams, companies, AI scaffolds, countries, the market, and the scientific community. In this post, I will lay out the basic ontology for modeling macroagents....
This is a distillation of the AI2040 alignment roadmap (with some parts also drawing from "How do we (more) safely defer to AIs?"). All ideas expressed are by Ryan and Thomas[1]; I only created the diagrams and substantially changed the structure to make it more accessible.[2][3] Note that the original...
(This post is an output of the Safe AI Germany (SAIGE) Incubator, specifically the "How will AI affect the democracy equilibrium", which is mentored by Simon (aka Towards_Keeperhood). Sophia and Arslan did the majority of the work and writing.) We’re really interested in how we ended up living in a...
2.1 Summary In the last post, I introduced model-based RL, which is the frame we will use to analyze the alignment problem, and we learned that the critic is trained to predict reward. I already briefly mentioned that the alignment problem is centrally about making the critic assign high value...
1.1 Summary and Table of Contents Why would an AI "want" anything? This post answers that question by examining a key part of the structure of intelligent cognition. When you solve a novel problem, your mind searches for plans, predicts their outcomes, evaluates whether those outcomes achieve what you want,...