To a first approximation, they are as likely as you to be biased, so why do they get to be the judge?
I think the answer to this is, "because the post, specifically, is the author's private space". So they get to decide how to conduct discussion there (for reference, I always set moderation to Easy Going on mine, but I can see a point even to Reign of Terror if the topic is spicy enough). The free space for responses and rebuttals isn't supposed to be the comments of the post, but the ability to write a different post in reply.
I do agree that in general if it comes to that - authors banning each other from comments and answering just via new posts - then maybe things have already gotten a bit too far into "internet drama" land and everyone could use some cooling down. And it's generally probably easier to keep discussions on a post in the comments of the post. But I don't think the principle is inherently unfair; you have the same exact rights as the other person and can always respond symmetrically, that's fairness.
Fun Baader-Meinhof effect I experienced: the very evening of the day in which I read this article, while chatting with my father-in-law, he mentioned (without me prompting) eating and enjoying a sandwich with lard, honey and chestnuts while vacationing in the Alps. Not quite the same but close enough, for more accessible ingredients. And the mountain setting makes a lot of sense because:
But I don't think the right conclusion is "Unpredictable!" so much as "So put in the work if you care to predict it?".
I still think there's a bit of post-hoc reasoning here; it's easy to rationalise why we would like ice cream, specifically, after the fact, and harder to make novel predictions that are that spot-on. Though as you say prediction can bring you a bit further than expected.
There's also the matter of information. How much information are the aliens even given to work from? To predict "chocolate ice cream" you would need data on the chemical composition of our biosphere, the ecological niches occupied by various animals, how mammalian biology and child-rearing works, how parasites work, how our biochemical energy producing mechanisms work, how DNA bases, insect nervous systems, and human nervous systems work (to guess that caffeine or similar compounds might be produced and enjoyed) and who knows what else. That's a lot of info, probably much more than we comparably have for hypothetical future ASIs. Absent all that, you get stuck with stupid predictions like "gasoline" or "bear fat with honey and salt".
As an additional point - "bear fat", specifically, is impractical for reasons I think even an alien with a modest understanding of Earth's biosphere could guess (I mean, have you seen a bear, Mr. Alien?). But "pork fat" is an exceedingly common ingredient, and not too far off. So "lard with honey and salt" or "tallow with honey and salt" would be very much possible to mass produce, and yet it's the ice cream that prevails. There may be something there, I'm sure lard with honey and salt is perfectly viable and possibly even made in some circumstances? But ice cream feels more "casual", I think milk-based fats are more digestible than the ones straight from the meat. Lard just doesn't scream "refreshing thing you eat while on a walk".
It makes sense as an extrapolation - chemical technology was advancing rapidly, so obviously the potential to do such things was there already or would have been shortly, and while maybe actual police investigators had never even really considered involving scientists in their work, Doyle with his outside perspective could spot the obvious connection and use it as a over plot idea to reinforce just how clever and innovative his genius detective was.
It's possibly another argument for why this happens: fiction can be a really good outlet for laypersons with not enough credentials to put ideas out there and give them high visibility. Once the idea is read by someone with the right technical chops, it can then spark actual research and the prophecy fulfills itself.
Part of the reason why this would be beneficial is also that killing all mosquitoes is really hard and could have side effects for us (like loss of pollination). One could hope that maybe humans would have similar niche usefulness to the ASI despite the difference in power, but it's not a guarantee.
I think those things can be generally interpreted as "trades" in the broadest sense. Sometimes trades of favour, reputation, or knowledge.
Of course, human-based entities are superintelligent in a different way than ASI probably will be, but I think that difference is irrelevant in many discussions involving ASI.
I think while the analogy absolutely does make sense and is worth taking seriously, this is wrong. The main reason why the analogy is worth taking seriously is that using partial evidence is still generally better than using no evidence at all, but the evidence is partial because the fact that ultimately a corporation is still made of people means there's tons of values that are already etched into it from the get go, ways it can fail at coordinating itself, and so on so forth, which makes it a rather different case from an ASI.
If anything, I guess the argument would be "obviously aligning a corporation should be way easier than aligning an ASI, and look at our track record there!".
He mentions he's just learned coding so I guess he had the AI build the scaffolding. But the experiment itself seems like a pretty natural idea, he literally likens it to a King's council. I'm sure once you have the concept having an LLM code it is no big deal.
That's not a bad idea. You could link something like "this post is a reply to X" and then people could explore "threads" of posts that are all rebuttals and arguments surrounding a single specific topic. Doesn't even need to be about things that have gotten this hostile, sometimes you just want to write a full post because it's more organic than a comment.