I've been watching with interest the debates around how good minicamp was. I think we need to distinguish between two hypotheses:
A) Minicamp was well-run, the participants enjoyed it and subjectively estimated it was helpful, and it made everyone involved much more enthusiastic about rationality and motivated to pursue positive self-change.
B) Minicamp had objective effects on measurable rationality parameters like calibration, and objective long-term effects on things like lifetime success, tendency to help rationality-related causes, and ability of participants to enjoy their life.
Everyone who's talking about how obvious it is that minicamp was a stupendous success is talking about A, and everyone who's saying they're not convinced is talking about B.
Most self-help doesn't work - so with zero background information about a camp, our prior probability of B is low. That this is a rationalist camp is extra information: P(B|Rationality_Camp) is greater than P(B) alone if we believe rationality is a more effective self-help strategy than average. But this just brings us to where we were before the camp started; to say the evidence in the post above increases that estimate we've got to investigate P (B|A) - the probability that, given a camp gets glowing reviews and everyone loved it and thinks it changed their life, the it really is effective.
But A is a common feature of almost all self-help camps - googling "Christian retreat testimonial" can be very enlightening (add the phrase 'changed my life' to the query for best results). I think most rationalists would be very skeptical of most of the camps that manage to get such glowing reviews from their participants. So P(B|A) - the probability that data shows a real long-term effect given that everyone loved it and is wildly enthusiastic - may not be much higher than P(B).
So if you're trying to prove A - that the camp was successful and everyone loved it and felt very motivated - you've more than succeeded by now. If you're trying to prove B, keeping on giving more and more evidence for A isn't really the way to go.
A better suggestion might be to tell people "We have strong evidence you'll love the camp and feel transformed and enlightened, and we have some evidence that it will help because rationality is teachable and we're trying to gather more specific evidence as the program continues."
But A is a common feature of almost all self-help camps - googling "Christian retreat testimonial" can be very enlightening (add the phrase 'changed my life' to the query for best results). I think most rationalists would be very skeptical of most of the camps that manage to get such glowing reviews from their participants. So P(B|A) - the probability that data shows a real long-term effect given that everyone loved it and is wildly enthusiastic - may not be much higher than P(B).
I suspect that even Christian retreats do cause B. Just being around people with high expectations of you will cause you to rise to the challenge. This is important since simply measuring B won't tell us whether the camp succeeded.
Everyone who's talking about how obvious it is that minicamp was a stupendous success is talking about A, and everyone who's saying they're not convinced is talking about B.
That's a really good point, and clarifies the issue enormously.
To be clear, I'm saying that minicamp had more of what you call B-type effect on me (so far) that many other such events. So I'm talking about B, not just A. From the OP:
Note that mini-camp was far from the first time I've travelled to an event to surround myself with like-minded peers working toward common goals. [...] I've been to many such workshops, inside and outside academia (~3 per year for the past 10 years). [...] Yet mini-camp is still topping my charts.
In particular, I'm saying that in my experience it was much more effective, B-wise, than the base-rate of generic peer gatherings like Christianity camps (which I've been to).
So no, not everyone who's excited about minicamp is just talking about A. But yes, I agree with you, A is a lot of the conversation. I'm trying to focus on B.
everyone who's saying they're not convinced is talking about B.
Not everyone. The single highest-voted comment on the subject asks for "any attempt to extend and replicate this success" and "If it actually were a failure, how would we know? Would anyone there even admit it, or prefer to avoid making its leaders look bad?" While Academician's subjective assessment adds nothing to Anna Salamon's survey, the comment about hiring Luke actually addresses these complaints.
First question: how many people enjoy meetups? How many enjoy them a lot? How many meetups are disasters painful to recall?
Second question: how many meetups have effects comparable to the substantive explicit and implicit claims being made for the mini-camp and full camp?
Third question: what makes you highly confident that the two classes differ on the second property, but not the first?
I don't know why you're asking these questions, but I'm interested in the answer to the first question. Here is some data: Six of my non-LW friends attended Less Wrong meetups. Five of them had opportunities to attend a second meetup, but only one of them did.
It seems like the selection process for the camps might have filtered out whatever class of people contribute to meetups being painful failures. I can imagine a meetup composed of two or three people each, selected from several meetups for maximum get-along-like-a-house-on-fire-ness, would be on the order of magnitude of awesomeness that the camps are attributed.
Is this a biased sample? Probably.
Why do you believe this? If you believe this, can you point out some steps that could correct the problem?
Well, not everyone elected to provide a testimonial, and there may have been self-selection in favor of optimism. Insisting that everyone write a testimonial might have helped a bit with that.
Yep, I'm saying that without hard data. But I was there. So let me say it again, in response to numerous comments I've seen complaining that no judgement should be passed until a quantitative analysis confirms it:
Mini-camp was awesome. Note that mini-camp was far from the first time I've travelled to an event to surround myself with like-minded peers working toward common goals... I find such events events extremely motivating and enjoyable, which is why I've been to many such workshops, inside and outside academia (~3 per year for the past 10 years).
Yet mini-camp is still topping my charts. Specifically, the camp is tied for the title of the most life-altering workshop-like event of my life, and the tie is with the workshop that got me onto my PhD topic (graphical causal modelling), so that's saying something.
In particular, I've been visibly-to-myself-and-others more motivated and hard-working since the camp. I've had more energy for learning and adaptation, and I find Luke to have been a highly inspiring input to that result.
(I'm talking about Luke because his position is the one being discussed right now, but I got a lot of really inspiring ideas and motivation from Anna before, during, and after the camp as well.)
Hard data will be great to have, but it's hard to get, especially certifiably causal data (though the prospect is not hopeless, with enough conditional independence tests), especially since the camp was planned and executed on short notice.
In the meantime, let's do a little Bayes. First, assign priors to how well you expect a week-long sustained interaction between growth-oriented rationalists to go. (If your prior is something like 80%[failure], I'd like to know where you're getting your growth-oriented rationalists). Now which of the following theories, "failure" or "success", assigns a higher likelihood to the following observations?
-----
1. People wrote these:
https://docs.google.com/spreadsheet/pub?key=0AnoM_ZsIBBwEdGNicUMzRkNJNzRKLVpEb2RxZzU3V0E
In particular,
“The week I spent in minicamp had by far the highest density of fun and learning I have ever experienced. It's like taking two years of college and condensing it to a week: you learn just as much and you have just as much fun. The skills I've learned will help me set and achieve my own life goal, and the friends I've made will help me get there.” --Alexei
“This was an intensely positive experience. This was easily the most powerful change self-modification I've ever made, in all of the social, intellectual, and emotional spheres. I'm now a more powerful person than I was a week ago -- and I can explain exactly how and why this is true.
At mini-camp, I've learned techniques for effective self-modification -- that is, I have a much deeper understanding of how to change my desires, gather my willpower, channel my time and cognitive resources, and model and handle previously confusing situations. What's more, I have a fairly clear map of how to build these skills henceforth, and how to inculcate them in others. And all this was presented in such a way that any sufficiently analytical folk -- anyone who has understood a few of the LW sequences, say -- can gain in extreme measures.” --Matt Elder / Fiddlemath
“I expected a week of interesting things and some useful tools to take away. What I got was 8 days of constant, deep learning, challenges to my limits that helped me grow. I finally grokked that I can and should optimize myself on every dimension I care about, that practice and reinforcement can make me a better thinker, and that I can change very quickly when I'm not constrained by artificial barriers or stress.
I would not recommend doing something like this right before another super-busy week, because I was learning at 100% of capacity and will need a lot of time to unpack all the things I learned and apply them to my life, but I came away with a clear plan for becoming better. It is now a normal and easy thing for me to try things out, test my beliefs, and self-improve. And I'm likely to be much more effective at making the world a better place as well, by prioritizing without fear.
The material was all soundly-researched and effectively taught, with extremely helpful supplemental exercises and activities. The instructors were very helpful in and out of session. The other participants were excited, engaged, challenging, and supportive.
I look forward to sharing what I've learned with my local Lesswrong meetup and others in the area. If that's even 1/4 as awesome as my time at the Mini-Camp, it will make our lives much better.” --Ben Hoffman / Benquo
“I really can't recommend this camp enough! This workshop broke down a complex and intertwined set of skills labelled in my brain as "common sense" and distinguished each part so that I could work on them separately. Sessions on motivation, cognition, and what habits to build to not fool yourself were particularly helpful. This camp was also the first example that I've seen of people taking current cognitive science and other research, decoding it, and showing people what's been documented to work so that they can use it too. It feels to me now as though the coolest parts of the sequences have been given specific exercises and habits to build off of. This camp, and the people in it, have changed my path for the better.” --David Jones / TheDave
2. I wrote this post.
3. Eliezer wants to keep Luke as a permanent hire.
4. Whatever other comments you've seen/heard about the camp from people who attended.
-----
Is this a biased sample? Probably. Is it hard data? Easy to quantify? Not so much. Might this be a big conspiracy by Luke-originating ninja bloggers? Perhaps. But really... which theory assigns the higher likelihood here? Success, or failure?
Lets allow the arguments that can be made about the minicamp be made, rather than ritualistically abstaining from decision-making until numbers show up.
That, and I really hope Luke stays with SingInst :)