The problem with this argument is that sugar is legal, but people are not, in fact, all eating ice cream for breakfast. The sugar equivalent of the AI castastrophe hasn't happened.
They aren't now, but if you look back at the time when sugar was actually discovered (around 1500), every European noble who could afford it did in fact revel in sugar. The wedding of Ercole d'Este, duke of Ferrara, is a particularly infamous example. I cannot find the historical menu in English, but the original Italian version is available here.
The things I was trying to convey here are that when we discover a new technology, it is tempting to get excited and think about all of the cool things we could do with it. Sugar seemed like a nice example here because it is very salient and visceral that it introduces more hedons into your life.
But it's also possible that the long-term harms and n-th order effects make the new technology a (large) net-negative. Which I'd argue is the case with sugar.
Relatedly, it's possible that something that seems innocent at first -- like a little sugar to sprinkle on top of your coffee -- is actually terrible. I agree that in the real world, sugar probably hasn't crossed that bar. But I think that's beside the point. From the perspective of the characters in this story, they don't know that. I think the question at hand is moreso how they should be approaching the situation. And I think that it is appropriate for them to be a little paranoid and consider the possibility of sugar leading to terrible, terrible outcomes (along with other possibilities, such as it leading to "regular-bad" outcomes).
(All of that said, I don't want to imply that I think this was a great post or anything. It's moreso just an idea that I was messing around and having fun with.)
In a parallel world that has just recently discovered sugar.
Alice: Hey, so have you guys heard about that new thing they discovered to make food taste better?
Bob: Yeah I think so. It's powdered and white, right?
Alice: Yeah.
Carol: It's actually granulated, like tiny little pebbles. And it's called sugar.
Bob: Ah, ok.
Carol: It's similar to the discoveries made however many years back about umami. Y'know, that aspect of taste where we perceive things as savory and "meaty".
Alice: Oh, I remember that. I love it. I'm all about that nutritional yeast now. And the other "umami bombs" like soy sauce and miso paste.
Bob: Yeah, me too. And I'm excited to do the same thing with sugar. So far I've been adding it to my coffee every morning and it's been great. It gives it this fun little kick. It mellows out the bitterness of the coffee too, but in a nice way that complements it. And in a different way from milk. It's subtle. It's really nice.
Alice: Oh yeah, for sure. I like it in my tea.
Bob: It's also great on top of things that might be a little counterintuitive. Like on top of popcorn. It might seem weird next to that butter and salt, but I dunno, I think it's pretty good.
Carol: That makes sense. I hear that in some Asian countries they're sprinkling some on top of fried rice and other stir fries.
Alice: Huh, that sounds a little weird actually. That sugary flavor with fried rice?
Carol: Yeah maybe. I was watching a YouTube video about it and the guy was saying how it's intended to be more of a subtle addition. Like, you don't really taste the sugariness itself, but there's just something different about the fried rice that is hard to put your finger on, but that tastes really good.
Alice: Ah, I gotcha.
Bob: But I've also heard about how they're experimenting with adding lots of sugar to certain things. Like there's this thing called ice cream. It's basically yogurt but with tons of sugar in it, and much colder, almost frozen. I got to try it the other day and it's sooooooo good. I'm never having yogurt again.
Dan: Is that a good thing?
Alice: Oh. Hey Dan. Nice to see you too.
Dan: Sorry, I just walked in, but I heard about this sugar and ice cream thing, and I'm a little worried.
Alice: Worried? What do you mean, worried? I haven't heard of any negative effects. It's supposed to just make food taste better.
Dan: Have you done any research?
Alice: Research?
Bob: I think what Dan is saying is that if you look more closely, maybe there are some downsides to adding sugar to food. I heard that it can make you feel lethargic afterwards. It seems to depend on your age and level of health. Older, less healthy people might feel a little lazy for an hour or so after consuming things with added sugar. But it doesn't last too long. And for people like us who are young and healthy, it should be totally fine.
Alice: Huh. I have it in my tea a lot and have never felt any amount of lethargy whatsoever.
Bob: Yeah, well it really only happens when you have a lot of it. Adding it to your tea in the morning should be totally, totally fine. But like with this ice cream thing, if you have a lot of it you might feel a little lethargic afterwards.
Alice: Hm, ok. That sounds pretty fine though.
Carol: I do think that Dan brings up a good point though. It's important to consider the side-effects of the things we put into our body.
Alice: Totally. That's fair.
Dan: Side effects are part of what I was thinking, but yeah.
Bob: Oh. What else were you thinking about?
Dan: Well, I don't really know actually. I just know that it's a new discovery -- call it a new technology even -- and as we know very well, technology can be used for both good, and for bad.
Alice: Oh, this again.
Carol: Haha, I hear you Dan, but I think we all are a little more trusting and open to new technologies than you are.
Bob: Yeah man. You don't even have a smart phone. Or any social media accounts.
Dan: But you do agree that technologies can be used for for both good and for bad though, right?
Bob: Sure.
Carol: Yeah, I think anyone would agree with that.
Alice: I mean, I suppose. But like, so much of the time the downsides are basically negligible. Like when I open my phone to pull up Google Maps, I get that you think it prevents me from developing my spatial intelligence or whatever, but I don't know, I just think that's such an unimportant thing to me that it essentially "rounds to zero".
Dan: And what happens when Google Maps becomes ubiquitous -- which it has -- and has data on where pretty much everyone is, and where everyone is going, at all times?
Alice: I mean, personally I don't really care.
Bob: It is a valid point though. Even if you don't care Alice, other people do. And even for you, what happens when Google sells that data to some creep who shows up on a first date with you and happens to know that you go to Tea My Tea every Thursday afternoon.
Alice: Ok yeah... that would be creepy.
Carol: And that's to say nothing about things like government spying.
Alice: Government spying? Huh?
Carol: I've heard of weird stuff like "predictive policing" where like, if you spend too much time in the "ghetto" parts of town, the police might have access to this information and be a little less charitable to you next time you're pulled over on the road.
Alice: Oh my god. That's absolutely awful.
Dan: Oh yeah. It could also be sold to potential employers. I've heard of some "stealth startups" who basically take that data, wrap it up and tie a bow on it, and sell these "human resource prospectus" reports that predict how "valuable" a job applicant would be based on this location data from Google Maps.
Carol: Oh my. But... how do they even infer that? How do they map location data to job performance? What job performance data do they even have access to?
Dan: Those are very good questions, Carol.
Alice: This does seem kinda sketchy.
Bob: Yeah. Let's bring this conversation back to sugar though, if you guys don't mind. Now that we've talked about all of this and have this context, I'm interested in kinda re-evaluating my perspective on sugar. What do you guys think?
Everyone nods.
Bob: Ok. So, um, I guess let's see if we could think about what Dan said, about how technology can be used both for good and for bad. We've seen a bunch of ways that sugar can be used for good. What about how it can be used for bad?
Alice: That makes sense to me. Although I just don't see how it could be used for bad. It's just something that you sprinkle on top of your food. Google Maps and the location data stuff I get. Like, that is more serious stuff and I see how you could use it for bad things. But not sugar.
Carol: Well, not so fast. I'm thinking back to what Bob said earlier about how he'll never eat yogurt again, and instead, will replace it with this new iced cream thing.
Dan: Oh my.
Bob: It's "ice cream". But yeah... now I'm rethinking this a bit. I used to eat yogurt for breakfast every morning. If I did that with ice cream, I dunno, maybe that could have bad long term effects on my health or something.
Carol: That does seem like a possibility. We do know that things like trans fats are really, really bad for your health in the long term, even though there are no immediate side effects. It's just something that shows up decades down the road.
Alice: Hm. Ok. I see that. But, like with trans fats, the government studied, figured out it was bad, communicated this with the public, people avoid foods with trans fats, and the government even is pushing away from it in places like schools. And we're all good.
Bob: Yeah. Sometimes we figure out that new technologies have bad impacts and succeed as a society at avoiding them, but this doesn't always happen.
Dan: Preach, brother.
Bob: Wait, I think I see what you did there. Religion?
Dan: Yup.
Alice: Sigh. Ok, I guess you're right about that one Dan. Religion isn't very good... but it has been widespread for thousands of years.
Carol: So, Dan: with respect to sugar, what do we do?
Dan: Well, I like the approach that the Mormons take.
Alice: What?! So you want us to just decline all of modern technology?
Carol: Yeah, I'm with Alice on this one. That seems a little extreme. I'm sure you're not much of a Mormon when you need antibiotics.
Dan: I don't think you understand what the Mormon's approach is. They don't decline to adopt technology. They just do so very slowly, and very carefully. Way more slowly than the western world.
Carol: That seems unreasonable. We can run experiments. Do science. We don't have to wait decades and decades before we have good data on what the impacts will be.
Dan: For the first-order effects, sure. But what about the second-order effects? Third-order? Fourth-order? I don't have much faith in our ability to predict those things. I feel much more comfortable observing how things actually play out in practice. But that's neither here nor there. There is an insurmountable coordination problem at play.
Bob: Huh?
Dan: Suppose that I was made president of the US or something and took the Mormonist approach of adopting the use of sugar very, very slowly. Over, say, 50 years. That's not going to stop other countries from adopting it.
Bob: And what happens then?
Dan: Other countries adopt it. People see how delicious it is. They eat ice cream for breakfast every morning. Americans take their vacations to Europe and also eat ice cream for breakfast every morning. They come home and tell their friends about it. Word spreads. The populace demands their sugar.
Bob: Well, you're the president in this hypothetical. Just tell them no.
Dan: And what happens when my four-year term is over?
Bob: Oh, yeah. Well, maybe with all of your political power, you could convince the other politicians how dangerous sugar is, and convince all of them to also agree to put these restrictions on sugar.
Dan: And what happens when there are dissenters?
Bob: Dissenters?
Dan: Yeah. Suppose I convince all of the politicians to do this, and then one guy comes along and runs on the campaign promising of giving Americans access to the sugar that they deserve.
Bob: Hm, I see. I guess it's tough to get everyone on board.
Dan: Exactly.
Alice: This makes sense to me. And even if this worked in America, what about other countries? If sugar really is so delicious, even if America successfully bans it, people will perhaps move to other countries where it is available. After all, Americans were pretty unhappy when the government tried to ban alcohol during the prohibition.
Carol: Yeah, makes sense to me too. At the very least, it is just a huge uphill battle to fight. Now I'm thinking about how wise it is for my scientist buddies and I to just publish everything we discover. Who's to say that these discoveries will be a net positive?
Dan: Perhaps. To me, the most promising path would be to "align" sugar. Study the heck out of it, and engineer it to guarantee that it will be a net positive rather than a net harm.