Nick that's an interesting thought, but it is possible that someone does know how to build a friendly AI, but may not know that he knows this. What I mean is that a person may be working on something unrelated to AI or is not interested in AI, but knows all the ingredients and technical things needed to make a friendly AI. Maybe he just doesn't know the implications of his own knowledge.
I think it's fair to say many scientists fall into this category, if only due to specialization. Maybe some inorganic chemist knows the cure to cancer. But simply due to his ignorance/uninterest in biochemistry or organic chemistry or whatever subject that's relevant is unaware of the implications of his own specialized knowledge on topics outside his area of expertise.
"Destroying the world" describes things fairly accurately. Who believes a nuclear strike on the major Soviet Union and American cities wouldn't have destroyed the world? Life, I'm sure wouldn't have ended (even human), but the fall out and human loss would have been catastrophic. It's easy to question this 20-30 years after the fact, but I remember the fear the world festered in in the 70's and 80's. If a nuclear missile had been launched, dozens if not hundreds would have been launched.
The change in world view and behavior would have been larger than any event in human history. Millions, possibly over a billion, lives would have been lost, and the prosperity of the world that started in the 70's, which has lifted over a billion out of abject poverty, would have come to screeching halt.
Eliezer,
"[L]ogical ignorance is still ignorance; if this were not so, I would know a proof of the Riemann Hypothesis." I understand what you mean by logical ignorance is still ignorance; as for the second part of your statement, I don't think that's necessarily true.
For example, if the RH were true you would need n statements in a particular order. Meaning that you have stated assumptions, then you proceed to S1->S2->...->Sn (where Si is the ith logical statement) and Sn is the conclusion of the proof. What you are saying is that you know all n statements, but not the order and you're trying to figure that out or possibly could if you thought about it. BUT, you may not know all n statements, which is what the second part of your statement claims. Your statement, also, assumes the RH is true, which is what the fuss is all about. No one really knows if it's true.
What I was trying to say is that you may know all n statements, even their order and have come up with them to solve some other problem. But you may never have heard of the RH, so don't know that you can solve it. I understand that this is still a form logical ignorance, but the second part of your statement (that you know a proof of the RH) doesn't follow from the first part.