I wonder if asking LLMs to beat best-in-class chess algorithms sounds like an impossible task to them. Therefore, to achieve the goal, models try to find a workaround, so they cheat just like humans. Imagine someone asks you to complete the same task. Seeing its absurdity (of course you cannot beat best-in-class chess algorithms), you'll search for workarounds, or simply decline to complete the task. In the end, LLMs were trained on human data and simulate their behaviour.
I wonder if asking LLMs to beat best-in-class chess algorithms sounds like an impossible task to them. Therefore, to achieve the goal, models try to find a workaround, so they cheat just like humans. Imagine someone asks you to complete the same task. Seeing its absurdity (of course you cannot beat best-in-class chess algorithms), you'll search for workarounds, or simply decline to complete the task. In the end, LLMs were trained on human data and simulate their behaviour.