Nov 16, 2011
Shortly before the Summit, Alexandros posted a short discussion post wondering whether rationality training might cause akrasia by prompting folks to make more decisions using deliberate, conscious, "system II" reasoning (instead of rapid, automatic, "system I" heuristics) and, thereby, causing decision fatigue.
This conjecture sounded interesting to me, and I'd wondered similar things myself, so I put up a poll to gather data.
I put this poll on up LW, asking a number of questions that I hoped bore on: (1) akrasia levels; (2) how the person's akrasia levels had shifted since they came to LW; and (3) how many decisions they made via deliberate "system II" processing. 70 LW-ers completed the survey in time to get included in my data analysis; perhaps because the survey was on Discussion, these were mostly folks who'd been on LW for a while; median response to "months since you started reading LW/OB" was 19.
I also wanted a control group so as to distinguish real LW anomalies from random bugs in most humans' self-reporting architecture. I tried to ask Reddit, but got only 7 responses; then I tried Mechanical Turk and received my full 100 desired responses... but it is hard to be sure that Mechanical Turk data is from real humans.
It would be nice if self-reported akrasia levels correlated with (lack of) success with common goals, such as income, exercise, and living in a non-filthy house. To assess participants' akrasia, I asked the following questions:
I replaced all responses with z-score estimates, replaced income with "income controlled for age", and started looking at correlations. Of these items, #3 and #4 failed to correlate with the other "akrasia" questions, so I discarded them and noted that "akrasia" might be less of a dimension than I was hoping. The others all correlated in the expected directions, although weakly, as shown:
(This table contains all correlations between the listed variables that occurred with p-value < .25, together with the associated p-value; note that the dataset was fairly small, so the absence of a statistically significant correlation does not necessarily imply the absence of a correlation.)
LW (thinks it is?) more akratic than average
On average, LW-er survey participants regarded themselves as having more trouble than usual with procrastination:
Mechanical Turk-ers regarded themselves as more average (and, especially, regarded their childhoods as more average), suggesting that this isn't just a "everyone thinks they have the most trouble" effect (though such effects do exist):
My guess is that LWers' perceptions of having more trouble than average with procrastination represents a real difference in folks' getting-things-done powers, and not just a difference in self-image. One piece of data supporting this is that LW-ers report higher than baseline rates of autism/asperger's: 5% of survey participants, compared about one percent of the general population. 20% of LW-ers report having ever had a depression diagnosis, which also seems to be somewhat above baseline rates. (Depression and autism both correlate with difficulty getting things done.)
Improvement over time?
LW-ers report improving over time. Moreover, they report improving more *since finding LW* than in an equal-sized chunk of time before finding LW; the difference in reported improvement over the two intervals is fairly small, but is statistically significant at the p=.001 significance level.
Mechanical Turk-ers do not report improving, and are not rosier about their last two years than about the two years before that -- suggesting that the above isn't just due to a bug in the human self-assessment system:
However, reported LW-er akrasia levels do not decrease with respondent age, which pulls against the thesis that LWers start out akratic but generically improve as we get older. They also do not decrease with "months since discovering LW/OB", which pulls against the thesis that LW helps.
To test the conjecture that excessive conscious decision-making (over-reliance on deliberate, conscious, "System II" reasoning instead of on automatic heuristics) causes akrasia, I asked:
Question 19 is a question from the research literature that is designed to test individuals' tendency to engage in "fully disjunctive reasoning", and, thus, to assess at least one aspect of folks' tendency to use system II reasoning in preference to automatic system I heuristics. The CRT is more standardly used for such measurement, but previous testing had indicated that LW-ers mostly hit the ceiling on the CRT, so I used the more difficult Anne question instead.
None of these questions correlated positively (to any discernably above-chance extent) with the indicators of akrasia. In fact, question 8 had a significant negative correlation with current self-reported procrastination levels (correlation -.26, p-value .03), suggesting that a tendency to deliberately choose one's work tasks may help procrastination/akrasia, and does not harm it.
On the other hand, questions 7, 8, and 19 also did not correlate strongly with one another; so it is possible that these are just not good indicators of folks' degree of reliance on deliberate, conscious, "System II" decision-making. Nevertheless, when you combine these fact that none 7, 8, or 19 indicated procrastination with the reported improvement after LWers find LW, it seems to together constitute reasonable evidence against LW and rationality training being harmful, or at minimum against them being sufficiently harmful to show up with such datasets.
Most of the remaining variables correlated with one another in the manner that common sense would suggest (e.g., being in school correlated with being young). Still, for completeness, here are all correlations among the questions that appeared correlated with a p-value <0.03; note that since I compared 25 variables with one another, we should expect about (25*24/2)*.03 = 9 correlations at this significance level, and 0 to 1 correlations at the p=.001 significance level just by chance. 
For ease of scanning, correlations that I personally found interesting are in bold. "Income" is instead "income adjusted for age and student status"; I adjusted kitchen cleanliness for student status as well.
In case you want to play with the raw data yourself, here it is:
Given the importance of taking actions that actually relate to one's goals (for happiness, income, world-saving, you name it -- and, hence, for real rationality), further investigation here would be welcome, either via further polls on LW-ers or others, or, perhaps even more easily and usefully, via Google Scholar.
 Originally, I asked about "trouble with akrasia" (now and as a kid) rather than about "trouble with procrastination". I edited the question after realizing I'd want a control group with non-LWers, and that that group would not reliably know the term "akrasia". So, the LW-er responses are partly to one wording and partly to the other.
 Both items correlated strongly with student status; when I controlled for student status (subtracted out the constant necessary to remove the correlation), "have had late bills reported to credit agencies" correlated strongly with diagnoses of ADD, autism/Asperger's, and depression, but with nothing else; dirty kitchen floors correlated with nothing.
 This is further suggested by the fact that Mechanical Turk-ers may well *have* more akrasia than average at present; they are working on Mechanical Turk, and have fairly high numbers of depression diagnoses.
 There are 34 correlations at the p<.03 significance level, and 12 at the p<.001 significance level, which is more than we should expect by chance; this is not surprising, since of course e.g. being in school is correlated with being young, and so of course we see some non-chance correlations; the question is how many of the non-obvious correlations are just chance.