My guess is Claude wouldn't find the math error off a "proofread this, here's its sources copy/pasted" type prompt but you can try.
I was curious about this so decided to check.
Both Claude 3.7 and GPT-4o were able to spot this error when I provided them just the Wikipedia page and instructed them to find any mistakes. They also spotted the arithmetic error when asked to proof-read the cited WSJ article. In all cases, their stated reasoning was that 200 million tons of rabbit meat was way too high, on the order of global meat production, so they didn't have to actually do any explicit arithmetic.[1]
Funnily enough, the LLMs found two other mistakes... (read more)
I was curious about this so decided to check.
Both Claude 3.7 and GPT-4o were able to spot this error when I provided them just the Wikipedia page and instructed them to find any mistakes. They also spotted the arithmetic error when asked to proof-read the cited WSJ article. In all cases, their stated reasoning was that 200 million tons of rabbit meat was way too high, on the order of global meat production, so they didn't have to actually do any explicit arithmetic.[1]
Funnily enough, the LLMs found two other mistakes... (read more)