Elon Musk is a real-life epic tragic hero, authored by someone trying specifically to impart lessons to EAs/rationalists:
--Young Elon thinks about the future, is worried about x-risk. Decides to devote his life to fighting x-risk. Decides the best way to do this is via developing new technologies, in particular electric vehicles (to fight climate change) and space colonization (to make humanity a multiplanetary species and thus robust to local catastrophes)
--Manages to succeed to a legendary extent; builds two of the worlds leading tech giants, each with a business model notoriously hard to get right and each founded on technology most believed to be impossible. At every step of the way, mainstream expert opinion is that each of his companies will run out of steam and fail to accomplish whatever impossible goal they have set for themselves at the moment. They keep meeting their goals. SpaceX in particular brought cost to orbit down by an order of magnitude, and if Starship works out will get one or two more OOMs on top of that. Their overarching goal is to make a self-sustaining city on mars and holy shit it looks like they are actually succeeding. Did all this on a shoestring budget compared to various rivals, made loads of enemies who employed various dirty tricks against him, etc. Succeeded anyway.
--Starts to worry more about AI x-risk. Tries to convince people to take it more seriously. People don't listen to him. Doesn't like Demis Hassabis' plan for how to handle the situation. Founds OpenAI with what seems to be a better plan.
--Oops! Turns out the better plan was actually a worse plan. Also turns out AI risk is a bigger deal than he initially realized; it's big enough that everything else he's doing won't matter (unaligned AI can follow us to Mars...). Oh well. All the x-risk-reduction accomplished by all the amazing successes Elon had, undone in an instant, by a single insufficiently thought-through decision.
This story is hitting us over the head with morals/lessons.
--Heavy Tails Hypothesis: Distribution of interventions by impact is heavy-tailed, you'll do a bunch of things in life and one of them will be the most important thing and if it's good, it'll outweigh all the bad stuff and if it's bad, it'll outweigh all the good stuff. This is true even if you are doing MANY very important, very impactful things.
--Importance of research and reflection: It's not obvious in advance what the most important thing is, or whether it's good or bad. You need to do research and careful analysis, and even that isn't a silver bullet, it just improves your odds.
I agree with you completely and think this is very important to emphasize.
I also think the law of equal and opposite advice applies. Most people act too quickly without thinking. EAs tend towards the opposite, where it’s always “more research is needed”. This can also lead to bad outcomes if the results of the status quo are bad.
I can’t find it, but recently there was a post about the EU policy on AI and the author said something along the lines of “We often want to wait to advise policy until we know what would be good advice. Unfortunately, t... (read more)