Sounds a lot like the UCSC wilderness orientation program. I participated in that back in 2001, as an incoming student, and later went through a good chunk of the guide training program before scheduling conflicts forced me to drop out. There may well have been changes since then, but I'll see what I can remember about it.
First, the decision-making process basically relies on students having relatively little wilderness experience, which leads to a certain fragility. Guides don't have formal authority, but they do have training, information, and experience that incoming students are expected to lack, which naturally puts them in a de facto leadership role. My group happened to include a more experienced student (an Argentinian backpacker and climber in his mid-twenties among a dozen homebody teenagers), which led to substantial friction: his ideas didn't always cohere with the group leaders', but inertia and implied authority usually won. Eventually he left the group and hiked out by himself.
It was very ritual-heavy, and I don't think they had all the bugs worked out. Some of the ritual worked; some came off as cheap and fuzzy New Age stuff, or clunky in other ways. A lot might have worked or not depending on luck and mindset; I'd put the 24-hour vigil in this category, especially since we were encouraged to use the time to make gifts for other hikers.
Still, it's pretty hard to screw up intense small-group bonding under wilderness conditions. Plenty of groups have exploited this, from the Boy Scouts to Burning Man, and WO was far from the worst attempt at it that I've seen. The group size is close to optimal for the time you're spending out: large enough that you can avoid anyone that really rubs you the wrong way, small enough that you can get to know everyone. The group leaders don't feel like a separate clique, probably because of something similar to the selection criteria you mentioned. The level of challenge is about right for inexperienced hikers, although I'd already been on several backpacking trips.
Overall I think it was a valuable experience and basically a good idea, but one hampered by mindset problems and a lack of structure and focus. A few relatively minor changes could have greatly improved it as both an orientation program and a group-building exercise.
How did it get started? How does it keep its momentum (especially early on)?
Those seem to be the two hardest parts about creating something like this.
I belong to a group at my university that organizes a backpacking trip for incoming freshmen in the two weeks before orientation week. This organization, which I will refer to as HIKE (not the real name), is particularly interesting in terms of group design. Why? It is approximately 30 years old, is run entirely by current students, and brings together a very large group of people and knits them into a largish community. Pretty much everyone involved agrees that HIKE works very well. During my involvement (I was a participating freshman, and I have since become staff) I have continually wondered, why is this group so much more fun than any other group I've been a part of?
It's also particularly effective. Leading ~80 incoming freshmen, who have no current friends, and who know no one, and who don't generally have any backpacking experience, into the woods for two weeks, is no easy task. HIKE manages its own logistics, staff training, and organization, entirely with student volunteers who staff the trip, with little to no university interaction. (We get them to advertise our trip, and they generally permit us to continue to exist.) It takes some dedication to keep this rolling, and I have seen other campus groups completely fail to find that kind of dedication from their membership.
While it's not a rationalist group, it seems to have stumbled upon a cocktail of instrumentally rational practices.
HIKE uses an interesting process of network homogenization. When staff members (who have generally been on several trips before) are assigned crews, staff members fill out "Who Do You Know?" forms, on which you rank how well you know other staff on a scale from 1 to 5. The people in charge of making groups, usually Project Directors, then group staffers based on how well you don't know other staff. You usually staff a trip with people that you haven't gotten to know very well, and then get to know them. Because of this process of strengthening the weakest bonds, HIKE is able to function as a relatively large social group, even across graduation classes and around existing cliques.
As far as actual interaction, HIKE involves a lot of face time with your crew of 10 freshmen and your co-staffers. There aren't really any breaks (with the exception of solos, see below) and you are hiking, eating, and chatting together for approximately 225 hours (15 waking hours in a day * 15 days). I had 13 hours and 40 minutes of class a week the Spring 2013 semester. HIKE is approximately 7+ weeks of class at that rate.
One of the more beloved HIKE traditions is the solo, where the hiking leaders pick a spot with plenty of isolated spaces, and the participants can choosee to spend ~24 hours alone and, optionally, fasting. It's a novel experience, and people like the time to rest and reflect in the middle of a very social, very intensive hiking trip.
My suspicion for why this all works is that HIKE very closely simulates a hunter-gatherer lifestyle. You travel in ~10 member groups, on foot, carrying your food, on mountain trails. You spend your every waking hour with the crew. The 2-3 hiking leaders are there to facilitate only (read: perform first aid if necessary, guide conversation, teach outdoor skills if necessary, and nudge the group if they get off track), and all decisions are made by consensus (which isn't an all-purpose decision making process, but is very egalitarian, and helps the group gel).
Maybe I'm just praising my friend-group, but I feel like I stumbled into a particularly strong group of people. We all feel very well-connected and we feel a lot of commitment to the program. My experience with other college groups has been that members are pulled apart by other commitments and a lack of familiarity with other members, and HIKE seems to avoid that with a critical mass of consecutive face time. We manage to have continuity of social norms across the years, but a great deal of flexibility (no one remembers what happened 4 years ago, and some traditions disappear and others cement themselves as ancient and hallowed despite being only two years old).
I'm interested in hearing any thoughts on this, and any relevant experience with other groups, ideas for testing cross-application, requests for further elaboration, etc.