I periodically find myself wishing someone had run an experiment on a particular topic. But they haven't.
Often, it seems like there's a relatively easy experiment you can run, which would give me some evidence about it. Maybe it wouldn't be perfect evidence, but it would be better than me making stuff up from my armchair.
The two clusters of things-I-can-imagine-being-quite-hard are:
- Legal requirements
- Actually doing the science right (avoiding sampling biases, actually testing the right variable, etc)
Scott's IRB Nightmare suggests that the legal requirements can be quite awful. I'm unclear on what those requirements are if I don't want to publish anywhere, don't plan to interface with any hospital bureaucracies, I just want to ask a bunch of people to try something and see how it goes (and maybe make a LessWrong blogpost so people on LessWrong can know too)
Am I allowed to just go out and ask a whole bunch of people stuff? If I want to know things about 8 year olds or 16 year olds, if their parents sign a form and I'm not doing anything especially weird or traumatic, would that be fine or would terrible things happen to me? (Could I replicate the Marshmallow Test?)
Doing Science Good Enough
I have a sense that there's a lot of ways to deceive yourself if you have a pet psychology theory. But... I dunno it also seems like LessWrong collectively should be pretty good at this. The thing I might want to do (or get others to do sometimes) is post a plan for a study here, and get critiqued until it seems like an actually good plan, do the plan, write up the results.
Particular Things I was interested in:
(not meant to be exhaustive, important, or particularly achievable plans, just the things that generated this question)
- How do most people relate to their job? (There's a survey study that asks if people consider it a job, a career, or a calling, but the study had a very narrow range of participants – people who worked at one particular place, and I'm curious how a wider variety of people would respond)
- How do most people relate to ambition? (where by ambition I mean "form plans to create or change things that will impact large numbers of people.")
- How many 8 year olds can learn to program? Can they implement FizzBuzz? How many 16 year olds? How many 30 year olds? I've heard that people either have-or-don't-have a "programmer trait" that's hard to learn, and I'm not sure how much of that is true at all, and if so if it's more about nature or nurture.