In 1980, Robert Axelrod invited researchers around the world to submit computer programs to play the Iterated Prisoner’s Dilemma. The results — where Tit for Tat famously won — transformed how we think about cooperation. What mattered most wasn’t intelligence or aggression, but a few simple principles: be nice, retaliate,...
Introduction Current discourse around AI safety focuses on creating some new paradigm for training AI models to make them safe, such as assistance games (Russell 2021) or corrigibility (Soares 2014, 2018). The prevailing view seems to be that the only thing keeping current AI systems safe is the fact that...
The strategies are manually reviewed with clear prompt injection attempts rejected.
I think the program approach proved unworkable. It is simply too difficult to write a program that can analyze another program effectively when the program being analyzed has to be complex enough to do its own analysis of other programs.
In 1980, Robert Axelrod invited researchers around the world to submit computer programs to play the Iterated Prisoner’s Dilemma.
The results — where Tit for Tat famously won — transformed how we think about cooperation.
What mattered most wasn’t intelligence or aggression, but a few simple principles: be nice, retaliate, forgive, and be clear.
That insight reshaped evolutionary game theory and inspired decades of work in economics and social science.
But Axelrod’s agents were opaque. They couldn’t read each other’s source code.
The Open Strategy Dictator Game asks: What happens when strategies are fully visible?
Each participant submits a natural-language strategy description — a few paragraphs of text explaining how their agent behaves.
Every... (read 238 more words →)
I've actually never heard of diffusion for planning. Do you have a reference?
A diffusion model for text generation (like Diffusion-LM) still has the training objective to produce text from the training distribution, optimizing over only the current episode - in this case a short text.
Current discourse around AI safety focuses on creating some new paradigm for training AI models to make them safe, such as assistance games (Russell 2021) or corrigibility (Soares 2014, 2018). The prevailing view seems to be that the only thing keeping current AI systems safe is the fact that they are not very smart. This paper will present the opposite view. The current paradigm has at least two elements that make it safe even when extended to superhuman intelligence.
The key current trend in AI is generating text(/image/sound/video) from the training distribution. Maximum-likelihood models like GPT-3 (Brown et al. 2020), diffusion models like DALL-E 2 (Ramesh et al. 2022) and... (read 2110 more words →)
Well-argued throughout, but I want to focus on the first sentence:
Can advocates of the more pessimistic safety view find common ground on this point?
I often see statements like “We have no idea how to align AI,” sometimes accompanied by examples of alignment failures. But these seem to boil down either to the claim that LLMs are not perfectly aligned, or else they appear contradicted by the day-to-day experience of actually using them.
I also wish pessimists would more directly engage with a key idea underlying the sections on “Misaligned personas” and “Misalignment from long-horizon RL.” Specifically:
- If a
... (read more)