Took the survey.
I hope this question isn't used the way I worry it will be used:
CFAR Question 3
A certain town is served by two hospitals. In the larger hospital, about 45 babies are born each day. In the smaller one, about 15 babies are born each day. Although the overall proportion of girls is about 50%, the actual proportion at either hospital may be greater or less on any day. At the end of a year, which hospital will have the greater number of days on which more than 60% of the babies born were girls?
This question was easy for me to answer by pattern-matching to the Law of Small Numbers, as outlined in Thinking, Fast and Slow. If I hadn't read that, it's hard to say whether I would have reasoned it out correctly. So if many respondents answer this question correctly, I hope that the survey authors don't claim evidence that LW readers are better at statistical reasoning -- it'd be more accurate to say that LW readers are more likely to have seen this very particular question before.
(I could, naturally, be assuming too much about the intents of the survey authors.)
Those are related abilities, but there's being able to answer specific questions and then there's being able to apply what you've learned more generally. For me, this particular question triggered more "aha! I've seen this one before!" than it triggered statistical thought. A correct answer to the question might give you a smidgen of information on whether the answerer can reason about statistics, but it probably gives you a lot more information about whether the answerer has seen the question before.
One superficial example of dealing with this problem is how, in my college discrete math class, the professor gave us a problem involving placing pigeons in holes, with the solution having nothing to do with the pigeonhole principle. Even better than obfuscating a problem, of course, is stating a novel one that exercises the skills you're testing for.