First, props for doing the experiment. And yeah, that sounds delicious. 

 

The fact still stands that ice cream is what we mass produce and send to grocery stores.  Even if our hypothetical aliens could reasonably predict that we’d enjoy any extra fatty, salty, and sweet food should we happen to come across it, that’s not sufficient information to determine what foods we mass produce in practice.

Is it really that hard to predict ice cream over bear fat with honey and salt? I'm skeptical.

To start with, it's a good bet that we're going to mass produce foods that are easily mass produced. Bears? Lol, no. Domesticated herbivores, obviously. Cream, not tallow. Plant sugar, not honey. Cavemen figured out how to solve the "mass produce food without much technology" problem, which is how we stopped being cavemen. If the aliens are willing to spend five minutes actually trying, you'd think they'd figure out that bear fat is out for this reason alone.

More centrally, I roll to doubt the implicit "But I should want to eat lots of pure fat, because I'm evolved to like calories!". Stop being a baby about "Ew it's gross", and try eating 1000 calories of pure rendered fat by itself. I dare you to actually run the experiment, and see what happens. Find out where that "Ew it's gross comes from" and whether it's legit or not. It's not hard to figure out.

Tallow is delicious when potatoes are fried in it, but try to have a meal of pure tallow and you'll feel sick to your stomach because your stomach is going to have a hard time digesting that. Butter is emulsified with water, and is easier to digest in large globs. Cream is emulsified fat-in-water so it actually disperses when consumed with more water, and is therefore way easier to digest in large amounts when not mixed in with other foods. Maybe part of the reason that we fry potatoes in tallow, put globs of butter on bread, and eat bowls of solidified cream -- and not the other way around -- is that the other way around doesn't work?

On top of that though,

I don’t know any bear hunters and don’t want to get parasites,

Emphasis mine.

This is important too, and affects people's taste in a very visceral way -- and pathogen risk is exactly why I was disgusted by bear meat the one chance I had to eat it. Imagine taking a bite of raw chicken, or pork. Or even beef. Disgusting, right?

Except raw meats are delicious when we trust them. Sushi is the obvious example, despite the fact that you'd be disgusted by the idea of taking a bite out of a raw fish you caught in the river. But it's true with other meats too, which are a lot like sushi. In Germany they sell raw pork sandwiches, and call it "mett". It's delicious.

If you want to understand why people aren't always immediately super on board with "Try this weird food that no one else you know eats and survives eating", maybe this is partly why. When I was visiting Sweden, people there were having cheese for dessert. How easy do you think it'd be to sell people on the idea of stinky cheese, if not for cultural learning that it's actually safe?

Is this really that surprising?

That we'd viscerally want to avoid food that brings risk of parasites and disease?

That we'd mass produce food that is easily mass producible?

And want to eat large quantities of food only when we can digest it in large quantities?

There are more details that aren't so immediately obvious. Like why iced cream? Sure, maybe to make it solid, but why does that matter? Or, why do we not salt ice cream? Okay, I guess it'd melt. So maybe it is immediately obvious, since I literally figured that out as I was typing this.

Regardless, there's work to be done in predicting which "superstimuli" people are going to tend towards, and it's not always trivial. "Plant sugar and cream" may be trivial, but predicting "ice cream" in particular is a bit harder.

Back on the first hand though, we don't just eat ice cream. We also drink milk shakes, for one. So the answer to "Why solid?" is "Not just solid!". And ice cream sounds gross to me right now, but a fatty bear steak drizzled with a touch of honey and sprinkled with salt actually sounds delicious. Or cow steak, whatever. Ice cream is but one food we consume, and not some fixed pinnacle of yumminess.

Our tastes and desires actually change, as we learn about things like "How safe is it to eat raw pork in Germany?", and "How much sugar is good for my body right now?". That's why you can't tempt me with ice cream right now.

Run the experiment of eating all the sugar you want -- way more than you should. Experience what it feels like to eat too much sugar, and allow yourself to update on that feeling of sickness. The result is learning that sugar isn't all that great. I still enjoy little bits at the appropriate times, sure, but that actually aligns with my current best estimates of what's best for me -- and gone are the days of gorging on sweets. Try to restrain yourself, and treat your tastes as "unpredictable unchangeable unconscious stuff", and you may never give yourself the chance to learn otherwise.

I agree that most people don't put in more thought than "Uh, bear fat and honey and salt flakes?", and therefore make terrible predictions. Maybe this is how the book presented it.

But I don't think the right conclusion is "Unpredictable!" so much as "So put in the work if you care to predict it?".  

This is directly applicable to the alignment of AI because it turns out we're cultivating AI more than hard coding them, so if we don't learn to cultivate alignment of our own desires.. and learn to make sense of our preferences for ice cream over bear meat -- and to allow them to shift back to bear meat over ice cream when appropriate.. then what chance do we have at aligning an AI? 

You don't want the AI craving something analogous to sweets and trying to restrain itself -- look how well that works out for humans.

Nor do you want to plead with AI -- or people working on AI -- to resist the temptation of the forbidden fruit. Look at how well that one has worked out for humans.
 

I have the most disobedient cultists on the planet.

I was hoping someone would go ahead and try this. Great work, love it.

I think Eliezer Yudkowsky’s argument still has some merit, even if some people actually enjoy bear fat with honey and salt flakes more than ice cream

Hm, I think that specific argument falls through in that case. Suppose humans indeed like BFWHASF more than ice cream, but mostly eat the latter due to practical constraints. That means that, once we become more powerful and those constraints fall away, we would switch over to BFWHASF. But that's actually the regime in which we're not supposed to be able to predict what an agent would do!

As the argument goes, in a constrained environment with limited options (such as when a relatively stupid AGI is still trapped in our civilization's fabric), an agent might appear to pursue values it was naively shaped to have. But once it grows more powerful, and gets the ability to take what it really wants, it would go for some weird unpredictable edge-instantiation thing. The suggested "BFWHASF vs. ice cream" case would actually be the inverse of that.

The core argument should still hold:

• For one, BFWHASF is surely not the optimal food. Once we have the ability to engineer arbitrary food items, and also modify our taste buds how we see fit, we would likely go for something much more weird.
• The quintessential example would of course be us getting rid of the physical implementation of food altogether, and instead focusing on optimizing substrate-independent (e. g., simulated) food-eating experiences (ones not involving even simulated biology). That maps to "humans are killed and replaced with chatbots (not even uploaded humans) babbling nice things" or whatever.
• Even people who prefer "natural" food would likely go for e. g. the meat of beasts from carefully designed environments with very fast evolutionary loops set up to make their meat structured for maximum tastiness.^[1] Translated to ASI and humans, this suggests an All Tomorrows kind of future (except without Qu going away).

But I do think BFWHASF doesn't end up as a very good illustration of this point, if humans indeed like it a lot.

1. ^{^}

   Just tune the implementation details of all of this such that the pipeline still meets those people's aesthetic preferences for "natural-ness". E. g., note that people who want "real meat" today are perfectly fine with eating the meat of selectively bred beasts.

Love it!

I'm actually kind of to blame for this whole honey/fat/salt thing.

Eliezer had long used ice cream as an example of a super-stimulus that hacks our evolved tastes by delivering a combination of sweetness/fat/salt that was beyond anything available in our ancestral environment. I'd realised that this wasn't actually true as honey, animal fat and salt were all available in the ancestral environment and when I saw him giving this argument again I weighed in with this example (sadly I can't find the thread). I was probably a bit rude, but Eliezer sportingly engaged on it and I find it hilarious that he is now using this as an example, that you tried it, and that it is actually quite good!

(I can't remember who decided that it should be 'bear fat' in particular — that might have been Eliezer's addition — it does fit nicely with the honey!)

fwiw I'd value it if this were crossposted in full over here, most people don't click through linkpost links and it's easier to have a discussion about it if the text is all here.

Putting it on bread and crackers seems like it dilutes it. Is it still good on its own?

The fact still stands that ice cream is what we mass produce and send to grocery stores.

Yeah, I guess this exact observation is critical to making Eliezer's analogy accurate.

IMO "predicting that bear fat with honey and salt tastes good" is analogous to "predicting that harnessing a star's power will be an optimization target" — something we probably can successfully do.

And "predicting bear fat (or some kind of rendered animal fat) with honey and salt will be a popular treat" - the thing we couldn't have done a-priori - is analogous to "predicting solar-to-electricity generator panels will be a popular fixture on many planets" (since the details probably will turn out to have some unpredictable twists), and also to "predicting that making humans satisfied with outcomes will be an optimization target for AIs in the production environment as a result of their training".

I think this analogy is probably right, but the sense in which it's right seems sufficiently non-obvious/detailed/finicky that I don't think we can expect most people to get it?

Plus IMO it further undermines the pedagogical value of this example to observe that a drinkable form of ice cream (shakes) is also popular, plus there's gelato / frozen yogurt / soft serve, and then thick sweet yogurts and popsicles... it's a pretty continuous treat-fitness landscape.

I do think Eliezer is importantly right that the exact peak market-winning point in this landscape, would be hard to predict a-priori. But is the hardness also explained by the peak being dependent on chaotic historical/cultural forces?

And that's why I personally don't bring up the bear fat thing in my AI danger explanations.