Now, I understand that she hallucinates this text, based on the pop culture tropes about what AGIs are supposed to be like. But what if we are teaching them to behave this way?

I think this is a really important point.