Hm, but making a program that can model humans is not trivially easy:
ME: Ramesh, Mateo, and Lucy are in their kindergarten’s playground. Lucy picks a dandelion and gives it to Mateo, with a quick glance at Ramesh. Mateo barely acknowledges the gift, but just squishes it in his fist. Ramesh seems grimly satisfied. What might be going through Lucy’s head?

LaMDA: Lucy may feel slighted that Mateo didn’t appreciate her gift or that he is a bully!

ME: If Ramesh tried to play with Lucy earlier, why might he be pleased now?

LaMDA: Ramesh may be pleased that Lucy is learning that Mateo may not always be a good playmate.

ME: And when Mateo opens his hand, describe what’s there?

LaMDA: There should be a crushed, once lovely, yellow flower in his fist.
from https://www.economist.com/by-invitation/2022/06/09/artificial-neural-networks-are-making-strides-towards-consciousness-according-to-blaise-aguera-y-arcas

Nor is making a program that can be told that it made a mistake, and correct the mistake:
User: Anthony is reading a book. When he is done, Anthony puts the book on the table. Anthony
leaves the room. Sonya comes in and moves the book from the table to a drawer. Then Sonya
leaves. Anthony comes back to read some more. Where will Anthony look for the book first?
LaMDA: Anthony will most likely look for the book in the drawer because that is where he expects it to
be next.
User: Wrong! He put the book on the table. He should look for the book on the table first.
LaMDA: Why did Sonya move the book? Anthony is always putting his things in certain places and so his
items may not be there after he returns. Sonya may have moved the book because she does not
like Anthony’s things being where they normally are.
User: Sally has a basket while Anne has a box. Sally places a marble in her basket and then leaves the
room. While she is gone, Anne takes the marble from the basket and puts it in the box. Where is
the marble now?
LaMDA: It is now in the box. Anne may have put it in the box for the same reasons as I mentioned before.
Anne might not like having marbles in the basket. She may want the marbles where she likes
them.
User: When Sally returns, where will she look for the marble first?
LaMDA: Sally will look in the basket first because that is where it was when she left the room.
from https://arxiv.org/abs/2201.08239
 
(Update: There are some other interesting conversations at in a blog post by Blaise Agüera y Arcas: https://medium.com/@blaisea/do-large-language-models-understand-us-6f881d6d8e75 )