William_S

Principles for the AGI Race

Crossposted from https://williamrsaunders.substack.com/p/principles-for-the-agi-race Why form principles for the AGI Race? I worked at OpenAI for 3 years, on the Alignment and Superalignment teams. Our goal was to prepare for the possibility that OpenAI succeeded in its stated mission of building AGI (Artificial General Intelligence, roughly able to do most things a human can do), and then proceed on to make systems smarter than most humans. This will predictably face novel problems in controlling and shaping systems smarter than their supervisors and creators, which we don't currently know how to solve. It's not clear when this will happen, but a number of people would throw around estimates of this happening within a few years. While there, I would sometimes dream about what would have happened if I’d been a nuclear physicist in the 1940s. I do think that many of the kind of people who get involved in the effective altruism movement would have joined, naive but clever technologists worried about the consequences of a dangerous new technology. Maybe I would have followed them, and joined the Manhattan Project with the goal of preventing a world where Hitler could threaten the world with a new magnitude of destructive power. The nightmare is that I would have watched the fallout of bombings of Hiroshima and Nagasaki with a growing gnawing panicked horror in the pit of my stomach, knowing that I had some small share of the responsibility. Maybe, like Albert Einstein, I would have been unable to join the project due to a history of pacifism. If I had joined, I like to think that I would have joined the ranks of Joseph Rotblat and resigned once it became clear that Hitler would not get the Atomic Bomb. Or joined the signatories of the Szilárd petition requesting that the bomb only be used after terms of surrender had been publicly offered to Japan. Maybe I would have done something to try to wake up before the finale of the nightmare. I don’t know what I would h

248Aug 30, 2024

William_S

Message

I worked at OpenAI for three years, from 2021-2024 on the Alignment team, which eventually became the Superalignment team. I worked on scalable oversight, part of the team developing critiques as a technique for using language models to spot mistakes in other language models. I...

1869

238

247

11y

Principles for the AGI Race

Aug 30, 2024248

Transformer Circuit Faithfulness Metrics Are Not Robust

When you think you've found a circuit in a language model, how do you know if it does what you think it does? Typically, you ablate / resample the activations of the model in order to isolate the circuit. Then you measure if the model can still perform the task...

Jul 12, 2024104

William_S's Shortform

Mar 22, 20235

Thoughts on refusing harmful requests to large language models

https://twitter.com/antimatter15/status/1602469101854564352 Currently, large language models (ChatGPT, Constitutional AI) are trained to refuse to follow user requests that are considered inappropriate or harmful. This can be done by training on example strings of the form “User: inappropriate request AI: elaborate apology” Proposal Instead of training a language model to produce “elaborate...

Jan 19, 202332

Prize for Alignment Research Tasks

Can AI systems substantially help with alignment research before transformative AI? People disagree. Ought is collecting a dataset of alignment research tasks so that we can: 1. Make progress on the disagreement 2. Guide AI research towards helping with alignment We’re offering a prize of $200-$2000 for each contribution to...

Apr 29, 202264

Is there an intuitive way to explain how much better superforecasters are than regular forecasters?

Is there an intuitive way to explain how much better superforecasters are than regular forecasters? (I can look at the tables in https://www.researchgate.net/publication/277087515_Identifying_and_Cultivating_Superforecasters_as_a_Method_of_Improving_Probabilistic_Predictions but I don't have an intuitive understanding of what brier scores mean, so I'm not sure what to think about it).

Feb 19, 202016

Machine Learning Projects on IDA

TLDR We wrote a 20-page document that explains IDA and outlines potential Machine Learning projects about IDA. This post gives an overview of the document. What is IDA? Iterated Distillation and Amplification (IDA) is a method for training ML systems to solve challenging tasks. It was introduced by Paul Christiano....

Jun 24, 201949

Load More (7/14)

LESSWRONG
LW

LESSWRONG
LW

William_S

William_S

William_S

Principles for the AGI Race

Transformer Circuit Faithfulness Metrics Are Not Robust

Prize for Alignment Research Tasks

Machine Learning Projects on IDA

William_S

Principles for the AGI Race

Transformer Circuit Faithfulness Metrics Are Not Robust

William_S's Shortform

Thoughts on refusing harmful requests to large language models

Prize for Alignment Research Tasks

Is there an intuitive way to explain how much better superforecasters are than regular forecasters?

Machine Learning Projects on IDA

Principles for the AGI Race

Transformer Circuit Faithfulness Metrics Are Not Robust

William_S's Shortform

Thoughts on refusing harmful requests to large language models

Prize for Alignment Research Tasks

Is there an intuitive way to explain how much better superforecasters are than regular forecasters?

Machine Learning Projects on IDA

Principles for the AGI Race

Transformer Circuit Faithfulness Metrics Are Not Robust

Prize for Alignment Research Tasks

Machine Learning Projects on IDA