This is somewhat speculative, but borne out by empirical data: LLMs developed an analogue of human System 1 and System 2, and their version of "System 2" is also slow, effortful and unreliable, just like ours.

https://twitter.com/emollick/status/1650324834960678912

This is an unusual emergent behavior, by all accounts. It is not like there is a shortage of accurate math in the training data. Maybe distilling these particular rules from the data is harder? 

Interestingly enough, augmentation works for LLMs the same way it works for humans: you can get a math plugin, such as Wolfram Alpha, which enhances the model's abilities temporarily, but does not improve its "native" abilities with prolonged use. The moment the plugin (or a human's calculator) is "disconnected" the model (or the human) is back to its previous "self", not having learned much. 

Maybe reasoning is harder in general? Or maybe it is an artifact of the training data? Is something like a System 2 an emergent feature? Does it mean that there is a "ceiling" of sorts for logical reasoning by the current crop of models, and we should not expect them to, say, code better than a human, write better poetry, prove theorems and discover new laws of nature? Would it have some implication for super-intelligent machines? I must admit, I am quite confused.