But I don't think the right conclusion is "Unpredictable!" so much as "So put in the work if you care to predict it?".
I still think there's a bit of post-hoc reasoning here; it's easy to rationalise why we would like ice cream, specifically, after the fact, and harder to make novel predictions that are that spot-on. Though as you say prediction can bring you a bit further than expected.
There's also the matter of information. How much information are the aliens even given to work from? To predict "chocolate ice cream" you would need data on the chemical composition of our biosphere, the ecological niches occupied by various animals, how mammalian biology and child-rearing works, how parasites work, how our biochemical energy producing mechanisms work, how DNA bases, insect nervous systems, and human nervous systems work (to guess that caffeine or similar compounds might be produced and enjoyed) and who knows what else. That's a lot of info, probably much more than we comparably have for hypothetical future ASIs. Absent all that, you get stuck with stupid predictions like "gasoline" or "bear fat with honey and salt".
As an additional point - "bear fat", specifically, is impractical for reasons I think even an alien with a modest understanding of Earth's biosphere could guess (I mean, have you seen a bear, Mr. Alien?). But "pork fat" is an exceedingly common ingredient, and not too far off. So "lard with honey and salt" or "tallow with honey and salt" would be very much possible to mass produce, and yet it's the ice cream that prevails. There may be something there, I'm sure lard with honey and salt is perfectly viable and possibly even made in some circumstances? But ice cream feels more "casual", I think milk-based fats are more digestible than the ones straight from the meat. Lard just doesn't scream "refreshing thing you eat while on a walk".
It makes sense as an extrapolation - chemical technology was advancing rapidly, so obviously the potential to do such things was there already or would have been shortly, and while maybe actual police investigators had never even really considered involving scientists in their work, Doyle with his outside perspective could spot the obvious connection and use it as a over plot idea to reinforce just how clever and innovative his genius detective was.
It's possibly another argument for why this happens: fiction can be a really good outlet for laypersons with not enough credentials to put ideas out there and give them high visibility. Once the idea is read by someone with the right technical chops, it can then spark actual research and the prophecy fulfills itself.
Part of the reason why this would be beneficial is also that killing all mosquitoes is really hard and could have side effects for us (like loss of pollination). One could hope that maybe humans would have similar niche usefulness to the ASI despite the difference in power, but it's not a guarantee.
I think those things can be generally interpreted as "trades" in the broadest sense. Sometimes trades of favour, reputation, or knowledge.
Of course, human-based entities are superintelligent in a different way than ASI probably will be, but I think that difference is irrelevant in many discussions involving ASI.
I think while the analogy absolutely does make sense and is worth taking seriously, this is wrong. The main reason why the analogy is worth taking seriously is that using partial evidence is still generally better than using no evidence at all, but the evidence is partial because the fact that ultimately a corporation is still made of people means there's tons of values that are already etched into it from the get go, ways it can fail at coordinating itself, and so on so forth, which makes it a rather different case from an ASI.
If anything, I guess the argument would be "obviously aligning a corporation should be way easier than aligning an ASI, and look at our track record there!".
He mentions he's just learned coding so I guess he had the AI build the scaffolding. But the experiment itself seems like a pretty natural idea, he literally likens it to a King's council. I'm sure once you have the concept having an LLM code it is no big deal.
I think not passing off LLM text as your own words is common good manners for a number of reasons - including that you are taking responsibility for words you didn't write and possibly not even read in depth enough, so it's going to be on you if someone reads too much into them. But it doesn't really much need any assumptions on LLMs themselves, their theory of mind, etc. Nearly the same would apply about hiring a human ghostwriter to expand on your rough draft, it's just that that has never been a problem until now because ghostwriters cost a lot more than a few LLM tokens.
However, the plausible assumption has begun to tremble since we had a curated post whose author admitted to generating it by using Claude Opus 4.1 and substantially editing the output.
TBF "being a curated post on LW" doesn't exclude anything from being also a mix and match of arguments already said by others. One of the most common criticisms of LW I've seen is that it's a community reinventing a lot of already said philosophical wheels (which personally I don't think is a great dunk; exploring and reinventing things for yourself is often the best way to engage with them at a deep level).
Fun Baader-Meinhof effect I experienced: the very evening of the day in which I read this article, while chatting with my father-in-law, he mentioned (without me prompting) eating and enjoying a sandwich with lard, honey and chestnuts while vacationing in the Alps. Not quite the same but close enough, for more accessible ingredients. And the mountain setting makes a lot of sense because: