When I talk to friends, colleagues, and internet strangers about the risk of ASI takeover, I find that many people have misconceptions about where the dangers come from or how they might be mitigated. A lot of these misconceptions are rooted in misunderstanding how today’s AI systems work and are developed. This article is an attempt to explain the risk of ASI misalignment in a way that makes the dangers and difficulties clear, rooted in examples from contemporary AI tools. While I am far from an expert, I don’t see anyone framing the topic in quite the way that I do here, and I hope it will be of service.

I hope readers familiar with the subject will appreciate the specific way I’ve organized a wide variety of concerns and approaches and find the body of the essay easy to skim. Wherever I’m missing or misunderstanding key ideas, though, I welcome you to point them out. Oftentimes the best way to learn more is to be wrong in public!

I hope readers new to the subject will find this presentation useful in orienting them as they venture further into subtopics of interest. (While the topic won’t be new to many people on LessWrong, I plan to link to this post from elsewhere too.) Any paragraph below could be the beginning of a much deeper exploration – just paste it into your favorite LLM and ask for elaboration. I’ve also provided some relevant links along the way, and links to a few other big picture overviews at the end.

I take it as a premise that superintelligence is possible, even if it requires some technological breakthrough beyond the current paradigm. Many intelligent, well-funded people are hard at work trying to bring these systems about. Their aim is not just to make current AI tools smarter, but to build AI tools that can act on long time scales toward ambitious goals, with broad and adaptable skillsets. If they succeed, we risk unaligned ASI quickly and effectively achieving goals antithetical to human existence.

You might find it tempting to say that artificial superintelligence is impossible, and you might even be right. I’d rather not bet human existence on that guess. What percent chance of ASI coming about in the next 50 years would justify active research into ensuring any ASI would be aligned with humanity? Whatever your threshold, I suspect the true probability exceeds it.

Onto the material.

What makes this hard?

Ensuring a...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments