I think the idea is that engaging with idiots¹ using idiotic arguments promotes and reinforces idiot behavior, even if the idiots don't enjoy it. Reinforcement doesn't always come with enjoyment.

¹ Where "idiot" stands for something like "aggressively wrong/fallacious/anti-epistemic arguer".

I imagine, if I were running the self-driving car company, one thing I might try to prevent this sort of thing is to record video, create a simulation of current street conditions, and constantly run lots of simulated cars through it. Any time the simulated car hits a simulated kid, shut the real-world cars down for a bit and call in the mech interp folks to figure out why.

Meanwhile in the real world, we should probably not allow the display of arbitrary data on highway billboards without a modicum of human review. For instance, putting QR codes on highway billboards is asking drivers to pick up their phones and try to scan them — which is illegal in the state of California.

Another oft-cited post on the subject is Sirlin's "Introducing...the Scrub". This one comes from the context of fighting games. A scrub is a player who is consistently defeated by moves that he (the scrub) rejects as "cheap" or "boring". The scrub loses because he has handicapped himself by forgoing certain winning moves; whether by not learning them, or by not executing them when given the chance.
 

I think this really points at different motivations, though. Sometimes a person finds themselves playing a competitive game when really they just want to be playing a game as an activity. They're looking for recreation or imagination, not competition: they're playing catch or playing pretend, not playing pro sports.

The designers of Magic describe three different "player psychographics", only one of which ("Spike") is about playing to win. They know that the audience for their game includes a lot of people who are not there to be maximally competitive. The other sorts of players are not going to win tournaments against the Spikes, but they're going to have a good time playing on the kitchen table.

It's okay for the game makers and players to have different maps of the player base, though. For the makers of a commercial game, it would be bad business sense to build only for Spikes, because they can sell more product if non-Spikes have a good time too. But for a competitive player who's striving to improve, there's some reason to label some other players as "scrubs" — because defeating them doesn't teach you much.

MAID now is already 5% of all deaths in Canada

Another way of looking at this number: 94% of deaths in Canada are nonconsensual. (Around 1% of deaths are non-medically-assisted suicides.)

There is a pretty big difference between "using generative AI to create funny images and videos to share with your human friends" and "getting into a 'social relationship' with a fictional character roleplayed by an AI chatbot."

In the first case, there are actually multiple people involved; there is actually a social relationship, which happens to be sometimes talking about computer-generated images. In the second case, there is a roleplay session with only one human player.

This is fine, as long as it also realizes that the real world is a test of some kind too, and behaves unusually well after making that observation.

Edited to add: To be clear, this is probably not fine.

In software engineering, "design patterns" has been used in a few different ways. One of them is something like "abstractions that are a little bit higher-level than the ones built into your programming language." Fans of some languages (Lisp, Haskell) looked at "patterns" in other languages (C++, Java) and said, "Why do you need to write a 'pattern' for that? It's just a {higher-order function, macro, monoid in the category of endofunctors, etc.}" — something built-in to the speaker's preferred language. (See Dominus, Norvig, C2 Wiki, and Graham's "Blub paradox".)

Analogously, there are "ethical design patterns" that are easily expressed by core features of other systems of morality. One example of the sort of thing I mean is (moral, legal, political) equality. (The sort that expands as "the same rules apply to everyone," not "everyone is assumed to have identical needs and abilities.") Some people say equality is a value itself; others might say equality is a tool for aligning people to achieve other values like peace or prosperity; others that some sort of equality is necessary for talking about morality to be meaningful at all (because a rule is only 'a rule' if it can apply to more than one person).

Regardless, equality describes a symmetry among rules. A claim like "Everyone has an equal right to own property, not a right to own equal property" is just saying that one symmetry exists and another doesn't. (Or, cynically, "the rich and the poor are equally forbidden from sleeping under bridges and stealing bread.")

(Other terms I think might work at higher levels of abstraction include "faith" and "grace". Faith points at something like "acting as if X is true, in a way that supports it turning out to be true" — like forgiveness in the iterated Prisoner's Dilemma, or loving your neighbor because God loves everyone and you are God's hands — and grace is when that actually works.)

To the person whose moral language is higher-level than consequentialism, consequentialists have gotta look a bit like C++ programmers look to the Lisp or Haskell hacker. Greenspun's Tenth Rule generalizes: "Any sufficiently developed consequentialist morality contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Kantianism." (Or maybe "...of virtue ethics." Or something else.)

I don't have the math to express it, but I think this points at consequentialism and something-like-Kantianism not being opposed, but rather both just true at different levels of abstraction. And a fun part of that, is that it makes "categorical imperative" into a math & CS pun.

The original D-K papers also found different curves for different subject matter. And they made the unusual choice of dividing their populations into quartiles, throwing away quite a bit of resolution. What's up with that?

Many foods have been selected for palatability.

Some of this selection has taken place through natural selection: wild animals choosing to eat fruits they find tasty (and disperse their seeds). In this case, because the animal consumer is also evolving, it's likely to end up favoring (and thus propagating) fruits that are actually nutritious too. After all, an animal that chose to eat foods that were bad for it, would be less likely to survive and prosper. So wild animals' tastes and their food plants' properties coevolve into something resembling an equilibrium, where the fruits are adequately tasty and nutritious.

But once modern humans come on the scene, artificial selection takes over — and once industrial capitalism arises, that selection is accelerated massively. To have a profitable food industry, the foremost goal of selection is not nutrition but rather marketability: being able to sell the food to other humans. This selection process runs much faster than the evolution of taste. So we end up inventing candy, cake, cocktails, and other hyperpalatable foods that are "bad for us" but sell well because they're tasty. We breed fruits to be big and sweet, and then we squeeze them for sugary juice and discard the fiber.

(And poky slow evolution doesn't breed out our taste for sweets, because candy isn't so bad for us that it kills everyone who eats it right away.)

Does this just change the problem to one of corrigibility? If the target is narrow but AI can be guided toward it, that's good. If the target is narrow and AI cannot be guided effectively, then it's predictably not going to hit the target.