A Bat and Ball made me Sad

Darren McKee

'A bat and a ball cost $110 in total.

The bat costs $100 more than the ball.

How much does the ball cost?

The answer is $5.

Please enter the number 5 in the blank below.

$_____'

That question is from one of 59 new studies (N = 72,310) which focus primarily on the “bat and ball problem," found in The formation and revision of intuitions - ScienceDirect by Andrew Meyer and Shane Frederic.

23% of respondents failed to give the correct answer.

I don't have anything insightful to say but wanted to share because it seems worth knowing.

Additionally, I'm not exactly sure how (and how much) I should update about this.

(Edit: I initially posted the wrong question)

'A bat and a ball cost $110 in total.

The bat costs $100 more than the ball.

How much does the ball cost?

The answer is $5.

Please enter the number 5 in the blank below.

I think there should be basically no update from this study, and most others using apps like Mechanical Turk and Google Consumer Surveys, which is what this study used. Consumer surveys are paid ads where you have to fill out the survey before getting to do what you actually want to do. People are incentivized to complete these as soon as possible and there is little penalty for inaccuracy. I would bet that this doesn't reproduce if correct answers are incentivized, or even if it's run on volunteers who aren't being stopped from what they want to do to take the survey.

1Dagon3y

There are certainly good ways to ask such a question with reasonable motivation to get it right. You could include this in a 5-question quiz, for instance, and say "you get paid only (or more) if you get 3 correct". And then vary and permute the questions so nobody uses 20 accounts to do the same task and not separately answer the questions. But that's expensive and time-consuming, and unless the paper specifies it, one should assume they did the simpler/cheaper option of just paying people to answer.

Monetary incentives raise solution rates a little, but not that much. Lex Borghans and co-authors manipulate small incentives and they do almost nothing. Ben Enke and co-authors offers a full month's salary to Nairobi based undergrads and finds a 13% percentage point increase in solution rates.

I'm not sure how our manipulations would interact with monetary incentives. But I'd like to know!

1solvalou3y

When surveys on mturk are designed to hold a single account occupied for longer than strictly necessary to fill out an answer that passes any surface-level validity checks, the obvious next step is for people to run multiple accounts on multiple devices, and you're back at people giving low-effort answers as fast as possible.

14

A Bat and Ball made me Sad

14

14

14

A Bat and Ball made me Sad

14

14