Giant Cheesecake Fallacy

Someone says: “When technology advances far enough we’ll be able to build minds far surpassing human intelligence. Now, it’s obvious that how large a cheesecake you can make depends on your intelligence. A superintelligence could build enormous cheesecakes—cheesecakes the size of cities—by golly, the future will be full of giant cheesecakes!” [...] The vision leaps directly from capability to actuality, without considering the necessary intermediate of motive. --

-- E Yudkowsky (2006) Artificial Intelligence as a Positive and Negative Factor in Global Risk

Someone says: “When technology advances far enough we’ll be able to build minds far surpassing human intelligence. Now, it’s obvious that how large a cheesecake you can make depends on your intelligence. A superintelligence could build enormous cheesecakes—cheesecakes the size of cities—by golly, the future will be full of giant cheesecakes!” [...] The vision leaps directly from capability to actuality, without considering the necessary intermediate of motive. -- E Yudkowsky (2006) Artificial Intelligence as a Positive and Negative Factor in Global Risk

• The genie knows, but does not care
• Mind design space
• Lawful intelligence

See also

• Mind design space
• Lawful intelligence