From research I did on New Zealand: 14 days isn't long enough if people are isolating together- live infections get passed to housemates partway through, who are still contagious when the 14 days are up. So either you have to truly isolate everyone, or extend it by some number of days for each additional person.

Also, I assume you mean "14 days, unless they test positive or show symptoms", but then you have to figure out how to test and verify symptoms for everyone.

A bunch of people have alluded to Decision Tree being "non-representative" in ways I worried people might interpret as "unusually bad" rather than "different by design in ways that predictably made things harder". I also think there are some lessons from the Decision Tree experiment that aren't currently obvious and could save people a lot of heartache. So I got permission from the organizers to share more of the story.

There was a group house in the bay that had a history of taking in strays (more than one in fact, but we're only talking about one). They would get to know people, usually online, with job loss, needing to escape abuse, or just wanting to move to CA and needing some help to get some feet under them and give them a few months couch time and food. It worked in part because the hosts were really good at boundaries and so really could limit themselves to just providing couch, food, and an amount of emotional support they were truly comfortable with. I don't know the whole roster, but I know at least four people this house took in who were much better off for it, and no one who was made worse off.

But they only had one couch, and it took months to get people on their feet, and so many people needed help. So with the help of a benefactor they decided to create an entire house dedicated to giving people this launching pad. That was Decision Tree. The plan was to subsidize rent and provide various RA/RD types to help people get on track.

Unfortunately this meant that you went from a house full of fully functional people + one rescue person, to a house full of rescues (with the people who became functional fastest leaving soonest).  There are a million reasons someone might need a place to crash for a few months. An unfortunate number of them are also reasons someone might be a difficult roommate. People who have only ever lived in dysfunctional situations, or are coming off a major trauma, are on average either harder to live or find living with other people harder. The subsidized rent meant that some portion of people genuinely couldn't afford to move, which could force housemates to stay together where richer people would have moved apart. 

So that's why I think Decision Tree is not a great data point for group houses as a whole. 

John and I had a fantastic offline discussion and I'm currently revising this in light of that. We're also working on a postmortem on the whole thing that I expect to be very informative. I keep mission creeping on my edits and response and it's going to take a while so I'm writing the bare minimum comment to register that this is happening.

I agree that Decision Tree was non-representative by design (in ways I'm not sure are public), in ways that will make it perform worse on average. I think that should have been noted more explicitly. I also think deluks is being really brave in naming something that made them a worse person, and I'm grateful they provided that data point.

I've been thinking a lot about this comment, and wanted to think more, but it seems useful to have something up as voting starts, so....

I think there's A Thing JW both agree is harmful (around assigning people moral responsibility when they're responding to incentives), and that I was trying to fight against. One thing I took from this comment is there's a good chance I had only a partial victory against Harmful Thing, and tried to pull down the master's house with the master's tools. I'd be very interested in exploring that further. (I also think it's possible JW is doing the same thing... it's a hard trap to escape)

I don't think giving up the question "Who should we blame?" entirely is a good idea. Possibly the benefits of the norm would outweigh the costs for LessWrong in particular, but I don't believe such a norm would be a pareto improvement. 

People often assume he's all about frugality. In reality, he thinks that some things are worth spending good money on.

From my limited reading of MMM, he's pretty intolerant/dismissive of people who have different beliefs about what is worth spending money on, which is a pet peeve of mine. 

I'm confused why this is a response to my comment, which was not an argument for or against the dodo bird verdict but about definitions.

Oh man, I wish you'd come in under the deadline.

For people who don't feel like clicking: it's a quantification of behavior predicted by different scores on Big-5.

Winner

Last week we announced a prize for the best example of an evaluation. The winner of the evaluations prize is David Manheim, for his detailed suggestions on quantitative measures in psychology.  I selected this answer because, although IAT was already on my list, David provided novel information about multiple tests that saved me a lot of work in evaluating them. David has had involvement with QURI (which funded this work) in the past and may again in the future, so this feels a little awkward, but ultimately it was the best suggestion so it didn’t feel right to take the prize away from him.

EDIT: David has elected to have the prize donated to GiveWell.

Honorable mentions to Orborde on financial stress tests, which was a very relevant suggestion that I was unfortunately already familiar with, and alexrjl on rock climbing route grades, which I would never have thought of in a million years but has less transferability to the kinds of things we want to evaluate.

Post-Mortem

How useful was this prize? I think running the contest was more useful than $50 of my time, however it was not as useful as it could have been because the target moved after we announced the contest. I went from writing about evaluations as a whole to specifically evaluations that worked, and I’m sure if I’d asked for examples of that they would have been provided. So possibly I should have waited to refine my question before asking for examples. On the other hand, the project was refined in part by looking at a wide array of examples (generated here and elsewhere), and it might have taken longer to hone in on a specific facet without the contest.