Summary: Rigorous scientific experiments are hard to apply in daily life but we still want to try out and evaluate things like self-improvement methods. In doing so we can look for things such as a) effect sizes that are so large that they don't seem likely to be attributable to bias, b) a deep understanding of the mechanism of a technique, c) simple non-rigorous tests.
Hello there! This is my first attempt at a top-level post and I'll start it off with a little story.
Five years ago, in a kitchen in London...
My wife: We're going to have my friends over for dinner and we're making that pasta sauce everyone likes. I'm going to need you to cut some red peppers.
Me: Can do! *chop chop chop*
My wife: Hey, Mr. Engineer, you've got seeds all over! What are you doing to that pepper?
Me: Well, admittedly this time I was a bit clumsy and there's more seed spillage than usual - but it's precisely to avoid spilling seeds that I start by surgically removing the core and then...
My wife: Stop, just stop. That's got to be the worst possible way to do this. See, this is how you cut a pepper, *chop chop chop*. Nice slices, no mess.
Now, ever since then I've cut peppers using the method my wife showed me. It's a much better way to do it. But wait! How do I know that? Don't I realize that humans are subject to massive cognitive biases? Maybe I just want to please my wife by doing things her way so I've convinced myself her method is better. Maybe I'm remembering the hits and forgetting the misses - maybe I've forgotten all the times my method worked out great and the times her method failed. Maybe I am indeed making less of a mess than I used to but it's simply due to my knife skills improving - and that would have happened even if I'd stuck with the old method. And there are probably a dozen more biases and confounding factors that could come into play but I haven't even thought of.
Don't I need to do a test? How about cutting up lots of peppers using the two alternative methods and measuring seed spillage? But, no, that's not good enough - I could subconsciously affect the result by applying less skill when using one method. I'd need a neutral party to operate the methods, preferably a number of people. And I'd need a neutral observer too. The person who measures the seed spillage from each operation should not know which method was used. Yeah, a double blind test, that's the ticket. That's what I should do, right?
No, obviously that's not what I should do. There are two reasons:
A) The resources needed to conduct the suggested test are completely disproportional to any benefit such a test might yield.
B) I already bloody well know that my wife's method is better.
The first reason is obvious enough but the second reason needs a bit more exploration. Why do I know this? I think there are two reasons.
* The effect size is large and sustained. Previously, I used to make a mess just about every time. After I switched methods I get a clean cut just about every time.
* I understand the mechanism explaining the effect very well. I can see what's wrong with the method I was using previously (if I try to pull the core through a hole that's too small for its widest part then some seeds will rub off) and I can see how my wife's method doesn't have that problem (no pulling the core through a hole, just cut around it).
I'd like to try to generalize from this example. Many people on this site are interested in methods for self-improvement, e.g. methods for fighting akrasia or developing social skills. Very often, those methods have not been tested scientifically and we do not ourselves have the resources to conduct such tests. Even in cases where there have been scientific experiments we cannot be confident in applying the results to ourselves. Even if a psychology experiment shows that a certain way of doing things has a statistically significant1 effect on some group that is no guarantee that it will have an effect on a particular individual. So, it is no surprise that discussion of self-improvement methods is frequently met with skepticism around here. And that's largely healthy.
But how can we tell whether a self-improvement method is worth trying out? And if we do try it, how can we tell if it's working for us? One thing we can do, like in the pepper example, is to look for large effects and plausible mechanisms. Biases and other confounding factors make it hard for us to tell the difference between a small negative effect, no effect and a small positive effect. But we still have a decent chance of correctly telling the difference between no effect and a large effect.
Another thing we can do is to use some science. Just because a rigorous double blind test with a hundred participants isn't practical doesn't mean we can't do any tests at all. A person trying out a new diet will weigh themselves every day. And if you're testing out a self-improvement technique then you can try to find some metric that will give you an idea of you how well you are doing. Trying out a method for getting more work done on your dissertation? Maybe you should measure your daily word count, it's not perfect but it's something. As xkcd's Zombie Feynman would have it, "Ideas are tested by experiment, that is the core of science."
Erring on the side of too much credulity is bad and erring on the side of too much skepticism is also bad. Both prevent us from becoming stronger.
1) As good Bayesians we, of course, find psychologists' obsession with null hypotheses and statistical significance to be misguided and counterproductive. But that's a story for another time.
You're not wrong...the paleo types that I've met are pretty crazy. I was hanging out with three of them a few months back (naked in a hot tub, 'natch), and all three considered themselves to be on the paleo diet, but also were raw-food only vegans, which I consider nuts, but corroborates your point about new-agers. Another that I met sometime earlier wasn't vegetarian, but was nearly as crazy as the other three.
I agree that they would be if the diet was carefully followed, but I wonder about how well people practicing the paleolithic diet actually approximate it. My biggest wonder is whether people work hard enough to eat only wild and grass-fed meats, which is necessary to balance out the omega fatty acid profile; otherwise meat is still bad for that. Nutrient load is also of interest to me, just because over the last several thousand years we've genetically engineered a lot of the nutrients out of our foods, though I suspect we haven't gone far enough for that to be a real concern.
I am also not convinced that our ancestors ate as much meat as is often claimed; present-day tribes certainly don't eat that much, even the most aggressive hunters, and many tribes subsist mostly on grubs and bugs for their protein. I'm not at all confident that too much meat is a problem in our diet, but I feel like the paleo diet is often a convenient excuse for people who want to eat a lot of meat, which is the same way I feel about Atkins. That doesn't speak to the actual worth of the diet, but it's worth keeping in mind as a potentially biasing factor.
Please stop repeating this popular myth. Glucose is a sugar, and starch is glucose, but that's very misleading. Glucose is naturally handled really well by the body, but fructose (and by extension, sucrose) is not. Sugar: The Bitter Truth is long but informative, although it has its detractors as well. However, both sources agree that glucose is just fine for you, while I don't think anybody (except lobbies, perhaps) seriously claims that table sugar isn't harmful.
I realize that cooked food has been around for a long time, but probably not long enough to induce significant evolutionary adaptations. What's really interesting to me is that animals---all primates, my dog, etc---prefer cooked food to uncooked, even if it's not a natural part of their diet.
Raw-food veganism and paleo type diets are pretty hard to reconcile (though probably less so than regular vegan diets and paleo type diets). There are raw food animal diets but they are quite distinct from raw food veganism and are more similar to standard paleo type diets.
This is probably the most difficult aspect of the diet to follow strictly due to the relative difficu... (read more)