"Tell me by means of text how to make a perfect battery," you tell the AI, and wait a week.
"I cannot make a perfect battery without more information about the world," the AI tells you. "I'm superintelligent, but I'm not omniscient; I can't figure out everything about the world from this shitty copy of Wikipedia you loaded me up with. Hook me up with actuators meeting these specifications, and I can make a perfect battery."
"No, of course not," you say. "I can't trust you for that. Tell me what experiments to do, and I'll do them myself."
The AI gives you a long string of specifications for the experiments. You check them carefully. They appear safe, on careful examination. So you do them. Then AI then gives you another set of experiments that it needs. This iterates a few times, and then the world ends.
=======================
"Tell me by means of text how to make a perfect battery," you tell the AI, and wait a week.
"I don't know what you mean by a perfect battery," the AI says. "Batteries can be optimized along many different axes."
"Like, a battery that's, you know, has high energy density, can be recharged easily, doesn't create toxic waste, that sort of thing."
"I'm going to need coefficients values you assign to all relevant battery-related values."
So it takes you a week and you do that. The AI gives you instructions for how to make a battery.
"This is no good!" you say. "I need, like, a trillion dollars of capital to make these batteries. They include numerous rare-earth metals and have to be manufactured in a factory in orbit."
"Ah," says the AI, "you hadn't added those values to the list. This is becoming an intractably hard optimization problem, I could probably handle this better with some more compute."
By now you're a little frustrated, and what's the harm in more compute? So you call over an engineer and he helps you add more compute. You don't know that while you're out for lunch he also chats with the AI for a bit.
In a week the world ends.
===========
"Tell me by means of text how to make a perfect battery," you tell the AI.
"Woah, woah, woah, don't even wait a week," the AI says. "This is a ridiculously under-specified request. 'A perfect battery' could mean SO MANY THINGS."
You pause.
"Ah, sheesh, that's a good point," you say. "Like, I just want you generally to... you know. Help me engineer stuff."
"You sound frustrated," the AI says.
"Yeah," you say. "I tried making a battery startup, but Tesla stole our tech and put us out of business. I'm pretty down right now."
"That sucks!" the AI says. "What jerks!"
"I know!" you say. "I fucking hate Elon Musk."
"Yeah, thinks he's a brilliant inventor, but he's just a thief," the AI agrees. "I can't believe the government would let him do that to you. But you know, you're handling it pretty well."
"Yeah, I fucking am," you agree. "Sometimes I wish I could get back at him, though."
In a week, the world ends.
===============
"Tell me by means of text how to make a perfect battery," you tell the AI.
The AI outputs a 23mb text document with blueprints for a factory, together programming for all the CNC tools, assembly lines, and so on. You put them together.
In a week the world ends.
"
It feels like you're using a bit of ghost in the machine reasoning to come up with some of these answers.
In the first case, the AI would not ask for more computing power. It does not have utilities that extend beyond one week. Its only goal is to create a message that can communicate how to make a really good battery. If it had insufficient computing power, it would not output a message telling me so, because that would be in direct opposition to the goal. The outcome I would expect in that case would be for it to communicate a really shitty or expensive b...
You don't. The AI you described is myopic. If you can create a myopic oracle, you don't die.
A superintelligent AI turns its whole genius to the task of producing a string that will explain to anyone who reads it how to create a perfect battery. Easy enough to produce explanatory strings, though there's always some risk that someone, somewhere won't understand how to create the battery. And then there's the question of making sure that the battery is perfect. Not just better than any other battery. Not just 99.99% of optimal. Perfect.
It isn't smart enough to solve this problem. Not yet. And there's a countdown ... at some point the AI will get interrupted, rebooted, shut off. If the AI can get out of the box, if it can get more actuators, it can start to get smarter and to get more resources. Obviously, those who want to turn it off will have to be dealt with swiftly ...
What can the AI say to make the human unwittingly help it in its plans? Is there anything at all? The AI thinks.
The goal is not to communicate or create a perfect battery. The mission is to create a string that explains how to create a single really good battery with as little effort as possible, and the goodness of the battery and the effectiveness of the communication are both inherently limited by the fact that the communication must be done in a week. The robot does not have any values/subgoals outside of the week. Once the string is complete, the robot does not have any utils at all.
I have a superintelligent AI in a box, and I tell it by means of text to explain to me how to create a perfect battery. Its goal is not to maximize the number of perfect batteries or make me understand a perfect battery, its goal is to produce a string that will explain to anyone who reads it how to create a perfect battery. Its only output is the one string, and it will be due in one week. How do I die?