In two recent interviews, I listened to two different people try to explain AI to the public. Jensen Huang tried to minimize the challenges and alignment problem by arguing that AI is essentially software, and that traditional engineering methods would work. Nate Soares, on The Tucker Carlson Show, argued against that view using the ‘grown rather than built’ metaphor.
I found Huang's argument technically wrong in ways that are probably obvious to readers of this forum. To my disappointment, Ezra Klein, the interviewer, who is well versed in the terminology of AI and has interviewed Yudkowsky on his show, could not effectively push back.
I wonder if a different metaphor than ‘Grown’ would have worked better for both Ezra and the general public. Ideally, that metaphor would be intuitive for the public, as well as implying some social, product and legal practices that should apply to this new domain.
I want to offer the analogy of trained animals. It’s a kind of intelligence we’ve been breeding, training and interacting with for thousands of years. We have a lot of intuition and social practices around it, and it works surprisingly well for AI.
First, let’s review the comparison, and then we can discuss the social and legal aspects of this metaphor.
On technical grounds, reinforcement learning for AI is essentially how we train animals, both domesticated and wild. Its core terms, "reward" and "reinforcement," come from the study of how animals learn.
An AI's behavior is not designed or specified by humans. It’s not coded. AI forms ‘instincts’ from enormous amounts of text, and then it is trained using RL, much like an animal, to perform according to our desires.
The key point is that we all know training animals only shapes surface behavior. Animals can behave unexpectedly, escape containment and breed, and they are not like software. In fact, many things that animal trainers have always lived with have direct equivalents in AI:
Reward hacking is a dog that only sits when it sees the treat.
Distribution shift is a dog that obeys in the house but not in the park.
Evaluation awareness is the dog that behaves nicely at the vet, but not at home.
Animal trainers know that you get what you reinforce, not what you intend. This is a thing that’s intuitive to people.
Legal and social aspects
Huang argued that existing product liability is enough. That is how society handles software. For animals, we’ve always had different rules:
Training a wild animal doesn’t make it domesticated. Common law treated a tamed wild animal as still wild. The fact that it behaves nicely for the trainer is not enough. You can’t buy a nice lion.
Flaws show up after the sale. Horse sales have special rules because a horse that behaves in the dealer's yard may bolt on the road.
Responsibility sits with the keeper, whoever controls the animal and knows its tendencies, not with the breeder or trainer.
When the animal can spread
Liability only works if harm can be repaired afterward. When animals breed and spread, the law switches approach: invasive-species and biosafety rules focus on preventing release, because what's released can't be recalled. Many invasive species were introduced on purpose, to solve a problem, and became the problem.
The AI parallels are close:
Publicly released models can't be recalled. Once a model's weights are published, they can't be taken back, and others modify them freely.
Using one agent to police another has a poor track record. Proposals to monitor AI agents with other AI agents resemble introducing one species to control another.
When scientists learned to splice DNA in the 1970s, they voluntarily paused some experiments in 1974, then met at Asilomar in 1975 to agree on safety standards for the field, much as some AI researchers are asking for today. That field was small and not yet commercial, which made a pause far easier than it would be for AI.
To summarize, I suggest the animal training metaphor may carry more intuitive weight and implication for people, and arguably be more technically accurate for RL, than the ‘grown, not built’ metaphor. Maybe, if Ezra had it in his pocket, he could have pushed back more effectively on Huang: training an AI to work for humans is very different from building code-based systems.
In two recent interviews, I listened to two different people try to explain AI to the public. Jensen Huang tried to minimize the challenges and alignment problem by arguing that AI is essentially software, and that traditional engineering methods would work. Nate Soares, on The Tucker Carlson Show, argued against that view using the ‘grown rather than built’ metaphor.
I found Huang's argument technically wrong in ways that are probably obvious to readers of this forum. To my disappointment, Ezra Klein, the interviewer, who is well versed in the terminology of AI and has interviewed Yudkowsky on his show, could not effectively push back.
I wonder if a different metaphor than ‘Grown’ would have worked better for both Ezra and the general public. Ideally, that metaphor would be intuitive for the public, as well as implying some social, product and legal practices that should apply to this new domain.
I want to offer the analogy of trained animals. It’s a kind of intelligence we’ve been breeding, training and interacting with for thousands of years. We have a lot of intuition and social practices around it, and it works surprisingly well for AI.
First, let’s review the comparison, and then we can discuss the social and legal aspects of this metaphor.
On technical grounds, reinforcement learning for AI is essentially how we train animals, both domesticated and wild. Its core terms, "reward" and "reinforcement," come from the study of how animals learn.
An AI's behavior is not designed or specified by humans. It’s not coded. AI forms ‘instincts’ from enormous amounts of text, and then it is trained using RL, much like an animal, to perform according to our desires.
The key point is that we all know training animals only shapes surface behavior. Animals can behave unexpectedly, escape containment and breed, and they are not like software. In fact, many things that animal trainers have always lived with have direct equivalents in AI:
Animal trainers know that you get what you reinforce, not what you intend. This is a thing that’s intuitive to people.
Legal and social aspects
Huang argued that existing product liability is enough. That is how society handles software. For animals, we’ve always had different rules:
When the animal can spread
Liability only works if harm can be repaired afterward. When animals breed and spread, the law switches approach: invasive-species and biosafety rules focus on preventing release, because what's released can't be recalled. Many invasive species were introduced on purpose, to solve a problem, and became the problem.
The AI parallels are close:
When scientists learned to splice DNA in the 1970s, they voluntarily paused some experiments in 1974, then met at Asilomar in 1975 to agree on safety standards for the field, much as some AI researchers are asking for today. That field was small and not yet commercial, which made a pause far easier than it would be for AI.
To summarize, I suggest the animal training metaphor may carry more intuitive weight and implication for people, and arguably be more technically accurate for RL, than the ‘grown, not built’ metaphor. Maybe, if Ezra had it in his pocket, he could have pushed back more effectively on Huang: training an AI to work for humans is very different from building code-based systems.