Many people I respect have started to think that AI risk is the most
valuable place to focus EA efforts. For example, 80000 Hours ranks
it first on their "list of global issues", taking into account
"scale, neglectedness, and solvability". On the other hand, I have a
lot of friends working in machine learning, and none of them think AI
risk is worth working on now. This level of disagreement is very
strange, and kind of worrying.
After playing for a dance weekend in South Carolina where casual
conversation among non-tech people included earnest speculation over
the likelihood that AI gets us all killed, I thought it might be a
good time to look back at when this was really speculative.
Summary: I talked to the right people, heard some incredibly prescient
ideas, and got the bottom line wrong.
I do a lot of my thinking via serial writing, and in this case that
was a series of twelve blog posts. Taking them one at a time:
This held up well. I picked something well worth
looking into, identified a lot of relevant resources, and got a lot
more good reading recommendations in the comments.
The main part of this post were five ideas
about where the disagreement might be. These worked ok as jumping off
points for interviews, but they weren't as core to disagreements as I
hoped: people often ended up with similar views on most of these ideas
while still being far apart in their views on how we should prioritize
AI risk.
One thing I do like, though, is that these were pretty concrete
questions which now do have answers:
How likely is it that current approaches are all we need for AGI,
with relatively straightforward extensions and a lot of scaling?
Very likely. When I asked this question, "current approaches" meant
deep neural nets and reinforcement learning. The past nine years have
mostly been an incredible amount of scaling.
How valuable is it to work on solving problems that are probably not
the right ones? For example, even if we think AGI will not look like
current systems, might trying to solve the control problem for current
systems teach us enough about the underlying problem and how to do
this kind of work that we'll be in a better position once we see more
what AGI will actually look like?
Looking at the two best-known approaches in 2017, I'd say Concrete Problems in AI
Safety has held up extremely well, while Highly
Reliable Agent Design (contemporaneous
review) less so. This makes sense, since the paradigm of the
former (scale current things) is what happened, while the latter
assumed we'd get to powerful AI systems via explicit systems with
world modeling, decision theory, etc.
Still, even the HRAD work did end up being pretty useful, by getting
people thinking about important problems, naming things, building a
field, and getting people spun up.
So I'd say work was pretty useful even if you didn't anticipate the
shape of today's AI.
How useful is it to have a strong theoretical foundation, vs just
understanding the technology enough from an engineering perspective
that we can make it do things for us?
In terms of building powerful AI systems, it seems like it has been
near 100% engineering. On the other hand, we now have powerful
systems that no one understands. As they get increasingly capable
will they continue to be compatible with the continued existence of
humanity? Hard to say! Seems like a strong theoretical foundation
would be pretty useful.
How similar is this to normal engineering? How much should we expect
companies' desires that their AI systems do what they want them to do
to work out?
Mostly yes, but more importantly no. Companies want their models to be
economically useful which means doing what users want. Except it
turns out you can provide a ton of economic value while still
occasionally being seriously misaligned, and there's a real chance
that as we continue to scale this misalignment is going to work out
very poorly.
As we get closer to AGI, how likely is the ML community to take
superintelligence risk seriously? Is it just that they don't think it
can be productively worked on now or do they not think it will ever be
a real problem?
People are taking this increasingly seriously, and I think this has
probably gone better than people in 2017 would have predicted. But it
doesn't look to me like they're taking it seriously enough: the
majority of work still looks like forging ahead very quickly despite a
lot
of evidenceof misalignment
even at current scales.
There's also a bit in the post where I talked about applying for a
grant to work on this. That was silly, since we had enough savings
that this was something I could just choose to do. Looking back, I
think the reason I applied was that I had an unrealistic view of
grantmakers as people with an excellent global view of what needed
doing, and where someone deciding whether to give me a grant to work
on this project would be a strong signal of whether the project was
worth doing.
(And then perhaps my most prescient decision was calling it
"Superintelligence Risk" instead of "Artificial Intelligence Risk",
anticipating executive
order 14434.)
Alan Kay said "the best way to predict the future is to invent it,"
which certainly seems true here. This description of the situation
feels super prescient. Unfortunately I never thought to ask the most
important question: might AI reliably doing what we want end up being
a limiting factor on AI progress, where this line of safety research
might accelerate AI development and put us in what Amodei then
considered the riskier "short timelines" path.
Littman essentially believed that prosaic
AGI, which Paul Christiano described as replicating human behavior
without "qualitatively new ideas about 'how intelligence works'", was
impossible. But that's pretty much what happened! So in many ways
it's a big miss. I don't want to single Littman out, however, since
in this series he's basically functioning as a stand-in for the
mainstream ML community view at the time. Most serious
professionals seemed to think this, and even Amodei and Christiano
weren't confident this was how things would go.
I think the largest piece of the disagreement between ML people over
how to prioritize AI risk may turn out to be: how likely is it that we
get AGI soon that looks a lot like systems we have now?
I think in retrospect this disagreement was indeed key, though I did
get a lot of pushback in the comments from people who thought it
didn't capture what was important.
A different line of disagreement in the comments was over whether ten
years was a plausible timeframe for large amounts of AI progress at
all, which turned into me betting Dave $100 to his $900 that we would
have
driverless cars in MA by 2027-07-12. Nine months out, the
technology is ready (Waymo could safely do it next month if they
wanted to) but I expect the legal process here to be too slow.
I think this also held up decently well: better examples really were
important. We got light versions via warning shots (especially
the Hugging
Face attacks) and people have since also written up some ones that
give a plausible story end to end.
On the other hand, some of the "sci-fi" elements that made stories
feel unrealistic, such as connecting powerful untrusted AIs to the
internet or ai-enabled bioattacks, seem a lot more plausible than they
did in 2017.
I found Open Phil's notes on Early Field Growth interesting,
especially their section on failure
modes in cryonics and molecular nanotechnology. My takeaway was
that heavy popularization of a new field prior to scientific success
leads scientists on the border of the new field to take on an
oppositional stance. The field gets starved of people who could do
substantial technical work, makes minimal progress, and I think people
also avoid the areas around its edges, like a chilling
effect. I see superintelligence risk as just on the edge of this,
where it could go either way.
It looks to me like AI safety came close to this, with some amount of
popularization producing opposition, but it didn't get starved
technically. I think because there was enough scientific success to
keep it interesting and active.
This one is the most interesting mainstream ML interview to me,
because they did foresee a lot: "they see progress in ML as almost
entirely constrained by hardware and data." On the other hand, it
does look to me like the differential technological development
strategy they didn't think would have much of an effect has actually
worked ok: having a few extra years of progress on various
safety-related efforts looks pretty helpful to me. (Still an open
question whether it has been enough, but that's not where this
researcher disagreed.)
I didn't feel great about this post at the time, and still don't like
it: it's mostly holding someone's early writing against them, where
the most recent of the writing was ten years old by the time I was
evaluating it. I agree with people who said this writing shouldn't be
seen as much evidence either way.
Looking back, I'd say Yudkowsky has clearly been right about a lot of
the risks involved, while at the same time (again, like most people)
not anticipating specifically how we'd get to present levels of AI systems.
This one is a painful read for me. In some ways it gets so close: the
"Background" and "Differential Progress" sections are solid, I think I
was right to be pessimistic about the HRAD approach, and then I
totally flubbed the "get the right answer" part. I had an incredible
opportunity, with Christiano,
Amodei,
and others giving thoughtful and detailed comments, and it wasn't
enough. I'd like to give a clean story here, where I see what
reasoning error I made, and how I'm going to avoid it going forward,
but I really don't.
I'm still glad I did this project, and I think getting these views
written down and shared was valuable. But getting so close and then
missing the key conclusion does sting.
In 2017, after getting laid off from a startup, I spent a while looking into AI risk:
After playing for a dance weekend in South Carolina where casual conversation among non-tech people included earnest speculation over the likelihood that AI gets us all killed, I thought it might be a good time to look back at when this was really speculative.
Summary: I talked to the right people, heard some incredibly prescient ideas, and got the bottom line wrong.
I do a lot of my thinking via serial writing, and in this case that was a series of twelve blog posts. Taking them one at a time:
Looking into AI Risk (2017-06-26).
This held up well. I picked something well worth looking into, identified a lot of relevant resources, and got a lot more good reading recommendations in the comments.
Superintelligence Risk Project (2017-07-03).
The main part of this post were five ideas about where the disagreement might be. These worked ok as jumping off points for interviews, but they weren't as core to disagreements as I hoped: people often ended up with similar views on most of these ideas while still being far apart in their views on how we should prioritize AI risk.
One thing I do like, though, is that these were pretty concrete questions which now do have answers:
How likely is it that current approaches are all we need for AGI, with relatively straightforward extensions and a lot of scaling?
Very likely. When I asked this question, "current approaches" meant deep neural nets and reinforcement learning. The past nine years have mostly been an incredible amount of scaling.
How valuable is it to work on solving problems that are probably not the right ones? For example, even if we think AGI will not look like current systems, might trying to solve the control problem for current systems teach us enough about the underlying problem and how to do this kind of work that we'll be in a better position once we see more what AGI will actually look like?
Looking at the two best-known approaches in 2017, I'd say Concrete Problems in AI Safety has held up extremely well, while Highly Reliable Agent Design (contemporaneous review) less so. This makes sense, since the paradigm of the former (scale current things) is what happened, while the latter assumed we'd get to powerful AI systems via explicit systems with world modeling, decision theory, etc.
Still, even the HRAD work did end up being pretty useful, by getting people thinking about important problems, naming things, building a field, and getting people spun up.
So I'd say work was pretty useful even if you didn't anticipate the shape of today's AI.
How useful is it to have a strong theoretical foundation, vs just understanding the technology enough from an engineering perspective that we can make it do things for us?
In terms of building powerful AI systems, it seems like it has been near 100% engineering. On the other hand, we now have powerful systems that no one understands. As they get increasingly capable will they continue to be compatible with the continued existence of humanity? Hard to say! Seems like a strong theoretical foundation would be pretty useful.
How similar is this to normal engineering? How much should we expect companies' desires that their AI systems do what they want them to do to work out?
Mostly yes, but more importantly no. Companies want their models to be economically useful which means doing what users want. Except it turns out you can provide a ton of economic value while still occasionally being seriously misaligned, and there's a real chance that as we continue to scale this misalignment is going to work out very poorly.
As we get closer to AGI, how likely is the ML community to take superintelligence risk seriously? Is it just that they don't think it can be productively worked on now or do they not think it will ever be a real problem?
People are taking this increasingly seriously, and I think this has probably gone better than people in 2017 would have predicted. But it doesn't look to me like they're taking it seriously enough: the majority of work still looks like forging ahead very quickly despite a lot of evidence of misalignment even at current scales.
There's also a bit in the post where I talked about applying for a grant to work on this. That was silly, since we had enough savings that this was something I could just choose to do. Looking back, I think the reason I applied was that I had an unrealistic view of grantmakers as people with an excellent global view of what needed doing, and where someone deciding whether to give me a grant to work on this project would be a strong signal of whether the project was worth doing.
(And then perhaps my most prescient decision was calling it "Superintelligence Risk" instead of "Artificial Intelligence Risk", anticipating executive order 14434.)
2017-07-04: Conversation with Dario Amodei
Alan Kay said "the best way to predict the future is to invent it," which certainly seems true here. This description of the situation feels super prescient. Unfortunately I never thought to ask the most important question: might AI reliably doing what we want end up being a limiting factor on AI progress, where this line of safety research might accelerate AI development and put us in what Amodei then considered the riskier "short timelines" path.
2017-07-09: Conversation with Michael Littman
Littman essentially believed that prosaic AGI, which Paul Christiano described as replicating human behavior without "qualitatively new ideas about 'how intelligence works'", was impossible. But that's pretty much what happened! So in many ways it's a big miss. I don't want to single Littman out, however, since in this series he's basically functioning as a stand-in for the mainstream ML community view at the time. Most serious professionals seemed to think this, and even Amodei and Christiano weren't confident this was how things would go.
2017-07-10: Superintelligence Risk Project Update
A reasonable summary of where I was so far, but nothing especially right or wrong in hindsight.
2017-07-11: Technical Distance to AGI
I wrote:
I think in retrospect this disagreement was indeed key, though I did get a lot of pushback in the comments from people who thought it didn't capture what was important.
A different line of disagreement in the comments was over whether ten years was a plausible timeframe for large amounts of AI progress at all, which turned into me betting Dave $100 to his $900 that we would have driverless cars in MA by 2027-07-12. Nine months out, the technology is ready (Waymo could safely do it next month if they wanted to) but I expect the legal process here to be too slow.
2017-07-13: Examples of Superintelligence Risk
I think this also held up decently well: better examples really were important. We got light versions via warning shots (especially the Hugging Face attacks) and people have since also written up some ones that give a plausible story end to end.
On the other hand, some of the "sci-fi" elements that made stories feel unrealistic, such as connecting powerful untrusted AIs to the internet or ai-enabled bioattacks, seem a lot more plausible than they did in 2017.
2017-07-14: Conversation with Bryce Wiedenbeck
Pretty similar to my conversation with Littman: Wiedenbeck did not think prosaic AI was possible, but it was.
2017-07-18: Superintelligence Risk Project Update II
Another status update. One interesting part:
It looks to me like AI safety came close to this, with some amount of popularization producing opposition, but it didn't get starved technically. I think because there was enough scientific success to keep it interesting and active.
2017-07-20: Conversation with an AI Researcher
This one is the most interesting mainstream ML interview to me, because they did foresee a lot: "they see progress in ML as almost entirely constrained by hardware and data." On the other hand, it does look to me like the differential technological development strategy they didn't think would have much of an effect has actually worked ok: having a few extra years of progress on various safety-related efforts looks pretty helpful to me. (Still an open question whether it has been enough, but that's not where this researcher disagreed.)
2017-07-27: Yudkowsky and MIRI
I didn't feel great about this post at the time, and still don't like it: it's mostly holding someone's early writing against them, where the most recent of the writing was ten years old by the time I was evaluating it. I agree with people who said this writing shouldn't be seen as much evidence either way.
Looking back, I'd say Yudkowsky has clearly been right about a lot of the risks involved, while at the same time (again, like most people) not anticipating specifically how we'd get to present levels of AI systems.
2017-09-15: Superintelligence Risk Project: Conclusion
This one is a painful read for me. In some ways it gets so close: the "Background" and "Differential Progress" sections are solid, I think I was right to be pessimistic about the HRAD approach, and then I totally flubbed the "get the right answer" part. I had an incredible opportunity, with Christiano, Amodei, and others giving thoughtful and detailed comments, and it wasn't enough. I'd like to give a clean story here, where I see what reasoning error I made, and how I'm going to avoid it going forward, but I really don't.
I'm still glad I did this project, and I think getting these views written down and shared was valuable. But getting so close and then missing the key conclusion does sting.