Logan Zoellner

Wiki Contributions


Ngo and Yudkowsky on alignment difficulty

Your initial suggestion, “launch nukes at every semiconductor fab”, is not workable. 

In what  way is  it not workable?  Perhaps we have  different intuitions about how difficult it is to build a cutting-edge semiconductor facility?  Alternatively you may disagree with me that AI is largely hardware-bound and thus cutting off the supply of new compute will also prevent the rise of superhuman AI?

Do you also think that "the US president launches every nuclear weapon at his command, causing nuclear winter?" would fail to prevent the rise of superhuman AGI?

Equity premium puzzles

Isn't one possible solution to the equity puzzle just that US stocks have outperformed expectations recently?  Returns on an index of European stocks are basically flat over the last 20 years.

The Greedy Doctor Problem

I'm surprised you didn't mention financial solutions.  E.g. "write a contract that pays the doctor more for every year that I live".  Although I suppose this might still be vulnerable to goodharting.  For example  the doctor may keep me "alive" indefinitely in a medical coma.

Ngo and Yudkowsky on alignment difficulty

the AI must produce relevant insights (whether related to "innovation" or "pivotal acts") at a rate vastly superior to that of humans, in order for it to be able to reliably produce innovations/world-saving plans


This  is  precisely the claim we are  arguing about!  I disagree that the  AI  needs to produce  insights "at a  rate vastly superior  to all  humans".  

On the contrary,  I claim that there is one borderline act (start a catastrophe that sets back AI progress by decades) that can be done with current human knowledge.  And I furthermore claim that there is  one pivotal act (design  an aligned AI) that may well be achieved via incremental progress.

Ngo and Yudkowsky on alignment difficulty

Is the plan just to destroy all computers with say >1e15 flops of computing  power?  How does the nanobot swarm know what a "computer" is?  What do you do about something like GPT-neo or SETI-at-home where the compute is distributed?

I'm still confused as to why you think task: "build an AI that destroys  anything with >1e15 flops of  computing  power --except humans, of course" would  be  dramatically easier than the alignment problem.

Setting back  civilization a generation (via catastrophe) seems relatively straightforward.  Building a social consensus/religion that destroys anything "in the image of a mind" at least seems possible.  Fine-tuning a nanobot swarm to destroy some but not all computers just sounds really hard to me.

Ngo and Yudkowsky on alignment difficulty

If an actually workable pivotal act existed that did not require better-than-human intelligence to come up with, we would already be in the process of implementing said pivotal act, because someone would have thought of it already. The fact that this is obviously not the case should therefore cause a substantial update against the antecedent.


This is an incredibly bad argument.  Saying  something cannot possibly work because no one has done it yet would mean that literally all innovation is impossible.

Ngo and Yudkowsky on alignment difficulty

Under  this definition, it seems that "nuke every fab on Earth" would qualify as "borderline", and every outcome that is both "pivotal"  and "good" depends on solving the alignment problem.

Ngo and Yudkowsky on alignment difficulty

If I really thought AI was going  to  murder us all in the next 6 months to 2 years, I would definitely consider those 10 years "pivotal", since it  would give us 5x-20x the time to solve the alignment problem.  I might even go  full  Butlerian Jihad  and just ban semiconductor fabs altogether.

Actually, I think that right question, is:  is there anything you would consider pivotal  other that just solving the alignment problem?  If no, the whole argument seems to be  "If we can't  find  a  safe way to solve the alignment problem, we should consider dangerous ones."

Re: Attempted Gears Analysis of AGI Intervention Discussion With Eliezer

The 1940's would like  to  remind you that one  does not need nanobots to refine uranium.

I'm pretty sure if  I had $1 trillion and a functional design  for a nuclear ICBM I could work out how to take  over the  world without any further help from the AI. 

If you  agree  that:

  1.  it is possible to  build  a boxed AI that allows you to take over the world
  2. taking over the world  is  a pivotal  act

then  maybe we should just do  that instead  of building a much more  dangerous AI that designs  nanobots and  unboxes itself? (assuming of course you accept Yudkowski's "pivotal-act framework of  course).

Ngo and Yudkowsky on alignment difficulty

the thing that kills us is likely to be a thing that can get more dangerous when you turn up a dial on it, not a thing that intrinsically has no dials that can make it more dangerous.

Finally a specific claim from Yudkowski I  actually agree with

Load More