After playing with Bing for a bit, it feels like Bing can't really understand the events of a chess game ***. I tried it's abilities on the following game (and some variations of this): 1. e4 e5, 2. Nf3 Nf6, 3. Nxe5 Qe7, 4. d4 Nxe4, 5. Bd3 Nf6, 6. O-O Nc6, 7. Re1 d6, 8. Nxc6 Qxe1+, 9. Qxe1+ Kd7, 10. Nb4 Be7 and asked it about the positions of the knight / knights at the end. I also asked it to reconstruct the position in FEN format which it did better than I thought but still incorrectly by quite a few piece.
Couple of times it told me (incorrectly) that there were two knights and then gave some fake positions for the knights (the positions were in a way logical, but obviously incorrect). In the end I found a prompt that correctly gave me the number and the position of the knights / knight in the couple scenarios I tested but it still usually gave the wrong explanation when asked how the other knight was captured.
Overall it feels like Bing can follow the game and almost kind of know where the pieces are after about 10 moves but when asked about things that happened during the game, it performs quite poorly***
*** It's important to remember that this isn't a proof that Bing can't do it, just that I couldn't get it to do it yet. With LLMs it's hard to prove that they don't have the capability to do something when prompted correctly.
After playing with Bing for a bit, it feels like Bing can't really understand the events of a chess game ***. I tried it's abilities on the following game (and some variations of this): 1. e4 e5, 2. Nf3 Nf6, 3. Nxe5 Qe7, 4. d4 Nxe4, 5. Bd3 Nf6, 6. O-O Nc6, 7. Re1 d6, 8. Nxc6 Qxe1+, 9. Qxe1+ Kd7, 10. Nb4 Be7 and asked it about the positions of the knight / knights at the end. I also asked it to reconstruct the position in FEN format which it did better than I thought but still incorrectly by quite a few piece.
Couple of times it told me (incorrectly) that there were two knights and then gave some fake positions for the knights (the positions were in a way logical, but obviously incorrect). In the end I found a prompt that correctly gave me the number and the position of the knights / knight in the couple scenarios I tested but it still usually gave the wrong explanation when asked how the other knight was captured.
Overall it feels like Bing can follow the game and almost kind of know where the pieces are after about 10 moves but when asked about things that happened during the game, it performs quite poorly***
*** It's important to remember that this isn't a proof that Bing can't do it, just that I couldn't get it to do it yet. With LLMs it's hard to prove that they don't have the capability to do something when prompted correctly.