I'm an admin of this site; I work full-time on trying to help people on LessWrong refine the art of human rationality.
Longer bio: www.lesswrong.com/posts/aG74jJkiPccqdkK3c/the-lesswrong-team-page-under-construction#Ben_Pace___Benito
(I agree with this short post, and enjoyed reading it.)
"will the first animals that take over the world be able to solve the Riemann hypothesis", and the answer is no because humans intelligence, while general, is still pointed more at civilisation-building-style tasks than mathematics.
Pardon the semantics, but I think the question you want to use here is "will the first animals that take over the world have already solved the Riemann hypothesis". IMO humans do have the ability ("can") to solve the Riemann hypothesis, and the point you're making is just about the ordering in which we've done things.
This post feels like a fantasy description of a better society, one that I would internally label "wish-fulfilment". And yet it is history! So it makes me more hopeful about the world. And thus I find it beautiful.
Just for record-keeping, here is the OWID global death tracker (from google), with the vertical line at the point when the comment was written.
Brief review: I think this post represents a realization many people around here have made, and says it clearly. I think it's fine to keep it as a record that people used to be blasé about the ease of secrecy, and later learned that it was much more complex than they thought. I think I'm at +1.
My quick two-line review is something like: this post (and its sequel) is an artifact from someone with an interesting perspective on the world looking at the whole problem and trying to communicate their practical perspective. I don't really share this perspective, but it is looking at enough of the real things, and differently enough to the other perspectives I hear, that I am personally glad to have engaged with it. +4.
Most people have not put tens of thousands of deliberate hours of practice into their writing skills so do not have the clarity to be able to say what they think shortly, and this lack of skill is typically why their writing is long. Eliezer has worked hard to be able to write clearly, and also to build a rare skill of being able to expose more of the cognition behind a thought as he writes longer, which is in many important domains more valuable to do than just stating the output of the cognition.
I'm saying: Eliezer's has built the skill to say his thoughts precisely and clearly; but he has also built the next-level skill of being able to expose the cognition behind a thought, and this is the sort of valuable length that he hopes to have in his writing.
"Search versus design" explores the basic way we build and trust systems in the world. A few notes:
For looking at the alignment problem clearly and with a subtly different frame than other discussions, one that resonates for me, and that points to new frames for a solution, I am voting this post +9.
Indeed the books are not yet in-stock in Amazon UK.
I am also quite interested to read this sequence.