Context: This is a linkpost for https://aisafety.info/questions/NM3J/5:-AI-may-pursue-goals
This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.
Suppose that, as argued previously, in the next few decades we’ll have superintelligent systems. What role will they play?
One way to imagine these systems is purely as powerful and versatile tools, similar to most current systems. They could take broad directions from humans about what actions to take or what questions to answer, and cleverly fill in the details.
But another way is as agents, operating autonomously in the world. They could have their own goals — some kinds of futures they seek out over other futures — and take whatever actions will most likely lead to those futures, adapting as circumstances change.
As long as AIs are tools, they can be used for good or ill, like all technologies. They can radically increase the scope of the problems humans can solve and create.
But it’s unlikely that they’ll remain only tools, because:
If we’re going to build AI systems that pursue goals, it would be good if those goals matched ours. It’s not clear if we’ll succeed at making that the case.