This is a special post for quick takes by Biff Wiff. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I just came across an analogy that seems applicable for AI safety.
AGI is like a super powerful sports car that only has an accelerator, no brake pedal. Such a car is cool. You'd think to yourself:
Nice! This is promising! Now we have to just find ourselves a brake pedal.
You wouldn't just hop in the car and go somewhere. Sure, it's possible that you make it to your destination, but it's pretty unlikely, and certainly isn't worth the risk.
In this analogy, the solution to the alignment problem is the brake pedal, and we really need to find it.
(I'm not as confident in the following, plus it seems to fit as a standalone comment rather than on the OP.)
Why do we really need to find it? Because we live in a world where people are seduced by the power of the sports car. They are in a competition to get to their destinations as fast as possible and are willing to be reckless in order to get there.
Well, that's the conflict theory perspective. The mistake theory perspective is that people simply think they'll be fine driving the car without the brakes.
That sounds crazy. And it is crazy! But think about ... (read more)