And it was surprisingly good Based on an old tweet from Eliezer Yudkowsky, I decided to buy a jar of bear fat online, and make a treat for the people at Inkhaven. It was surprisingly good. My post discusses how that happened, and a bit about the implications for Eliezer's thesis. Read the full post below, which is slightly edited from the version at the link. Let me know if you want to try some; I can prepare some for you if you happen to be at Lighthaven before we run out of bear fat, and before I leave toward the end of November. 1. Eliezer Yudkowsky Eliezer Yudkowsky did not exactly suggest that you should eat bear fat covered with honey and sprinkled with salt flakes. What he actually said was that an alien, looking from the outside at evolution, would predict that you would want to eat bear fat covered with honey and sprinkled with salt flakes. The point about bear fat in this tweet (also featured in "If Anyone Builds it, Everyone Dies") was to argue that we don’t know what an AI would want, even if we know how an AI was trained. Consider ice cream. The aliens could know that human taste buds were trained to crave nutrients that give us chemical energy in forms that our kind of biology can digest, like fat and sugar. Since they’ve studied evolution very closely, they might even realize that we’d crave salt to keep our bodies at the salinity we’re used to from our evolutionary environment. But, Yudkowsky argues, the aliens wouldn’t predict that humans like ice cream, they’d just predict that we’d like something fatty, sweet, and salty. This description fits bear fat with honey and salt flakes just as well as ice cream, or maybe even better. So similarly, even if we knew a lot of detail about how AIs are trained, as much detail as the aliens knowing our appetite for salt, we wouldn’t know exactly what an AI would do in practice if it got the chance to control the future of the world. Top of page 60 of t