I like to use LLMs to help verify math problems and reinforce my own understanding and while most of the time I enhance my learning, sometimes I get a hard reminder that LLMs are still not perfect at the core task any problem solving -- logical reasoning.

A problem that inspired this post is below.

Every viewer who bought a ticket in the first row of the cinema took one of the seats in that row. All n seats are occupied, but no one is in the correct seat. The ticket-collector may repeatedly swap two neighbouring viewers, but only when both of those viewers are sitting in the wrong seats. Can he always end up with everyone in their own seat?

It's a classic CS leetcode type of algorithmic problem wrapped in a seating puzzle. The core question is, given a sequence of permuted numbers can we arrange them into correct ordering?

After drafting my solution, I asked ChatGPT o3 for its solution. Here's the response it gave.

Then it even proceeded to go over an example. The solution seems well constructed and elaborate, except that it's wrong. This general algorithm doesn't handle the edge case when the current number is swapped with a number that's one position away from its correct location. This edge case refutes the whole constructed solution. So I proceed to ask GPT to go over the tricky case of 5,3,1,2,4. Here we can't start with 1 and swap it with 3 because that'll put 3 in its correct location and block 5. Here's the reasoning ChatGPT provided:

Notice the transition from step 3 to 4 is completely nonsensical. When we move 1 to its correct location at step 3, then according to the algorithm we need to start swapping 2 but we can't do it because 3 is already in its correct position so 2 is blocked. To bypass its own logical gap, ChatGPT pretended like it didn't notice the violation, and at step 4 somehow 2 magically moved by two positions to its correct location. Then at step 5 there's no change at all. So instead of admitting that the original solution doesn't work, ChatGPT tricks counterexamples to work. Sfter pointing that out, I get a classic "so sorry you're right response".

Now it acknowledges the mistake and proceeds to solve the example correctly but without modifying the general algorithmic solution. And what use is a solution in problem solving if it only handles a specific case?

Then I test Gemini 2.5 Pro hoping I will get a clearer response. But oh well! It ...

Posts

Wikitag Contributions

Comments