I've resigned myself to accepting that whatever life plans I've had, should I still want them after the singularity, I can step into a simulation of it inhabited by zombies. I'm not that excited about zombies either but my knowledge of the fact that I'm in a simulation can be suppressed, so nothing will feel any different. All the other inhabitants of the simulation will feel 100% real to me. Once I internalized this idea, I stopped lamenting the singularity.
I wonder if the carrots were implemented as a means of encouraging doing more of the thing than the organism would do otherwise. If there were no carrot aspect to hunger, the organism would only eat just enough to stop the pain of hunger but no more, missing the opportunity to build up fat stores for the future. If there were no carrot aspect to peeing, the organism would expel just enough to make the pain go away but wouldn't empty the tank, so to speak, making them carry around some tiny amount of extra weight than otherwise. This might be a just-so story, but it's the answer that comes to mind for me.
What I take away from this experiment is that the most effective form of persuasion is to blackmail or bribe the person you're trying to persuade and convince them that you'll make good on it. You don't need to convince them about the inherent goodness of what you want them to do if all you want is for them to do it. Given this, it seems trivial that AI will eventually become vastly superhuman at persuasion. It can find out every person's deepest secrets, fears, and desires and figure out what sorts of things are likely to get them to fold.
His metaphors serve as useful intuition pumps for ideas that are highly abstract and far removed from what the average layperson has thought about prior. When I started reading about AI risk, many of the ideas I was introduced to, like optimization and orthogonality, were completely novel and I had a hard time understanding them. The various metaphors, analogies, and parables that exist within the AI-risk discussion were of significant benefit to me in gaining the necessary intuitions to understand the problem.
Getting AI to terminally care about humans at all seems like a hard target and if our alignment efforts can make it happen, they can probably also ensure that it cares about humans in a good way.