It’s time for a new survey!

Take the survey now


The details of the last survey can be found here.  And the results can be found here.

 

I posted a few weeks back asking for suggestions for questions to include on the survey.  As much as we’d like to include more of them, we all know what happens when we have too many questions. The following graph is from the last survey.


http://i.imgur.com/KFTn2Bt.png

KFTn2Bt.png

(Source: JD’s analysis of 2014 survey data)


Two factors seem to predict if a question will get an answer:

  1. The position

  2. Whether people want to answer it. (Obviously)


People answer fewer questions as we approach the end. They also skip tricky questions. The least answered question on the last survey was - “what is your favourite lw post, provide a link”.  Which I assume was mostly skipped for the amount of effort required either in generating a favourite or in finding a link to it.  The second most skipped questions were the digit-ratio questions which require more work, (get out a ruler and measure) compared to the others. This is unsurprising.


This year’s survey is almost the same size as the last one (though just a wee bit smaller).  Preliminary estimates suggest you should put aside 25 minutes to take the survey, however you can pause at any time and come back to the survey when you have more time.  If you’re interested in helping process the survey data please speak up either in a comment or a PM.


We’re focusing this year particularly on getting a glimpse of the size and shape of the LessWrong diaspora.  With that in mind; if possible - please make sure that your friends (who might be less connected but still hang around in associated circles) get a chance to see that the survey exists; and if you’re up to it - encourage them to fill out a copy of the survey.


The survey is hosted and managed by the team at FortForecast, you’ll be hearing more from them soon. The survey can be accessed through http://lesswrong.com/2016survey.


Survey responses are anonymous in that you’re not asked for your name. At the end we plan to do an opt-in public dump of the data. Before publication the row order will be scrambled, datestamps, IP addresses and any other non-survey question information will be stripped, and certain questions which are marked private such as the (optional) sign up for our mailing list will not be included. It helps the most if you say yes but we can understand if you don’t.  


Thanks to Namespace (JD) and the FortForecast team, the Slack, the #lesswrong IRC on freenode, and everyone else who offered help in putting the survey together, special thanks to Scott Alexander whose 2014 survey was the foundation for this one.


When answering the survey, I ask you be helpful with the format of your answers if you want them to be useful. For example if a question asks for an number, please reply with “4” not “four”.  Going by the last survey we may very well get thousands of responses and cleaning them all by hand will cost a fortune on mechanical turk. (And that’s for the ones we can put on mechanical turk!) Thanks for your consideration.

 

The survey will be open until the 1st of may 2016

 


Addendum from JD at FortForecast: During user testing we’ve encountered reports of an error some users get when they try to take the survey which erroneously reports that our database is down. We think we’ve finally stamped it out but this particular bug has proven resilient. If you get this error and still want to take the survey here are the steps to mitigate it:

 

  1. Refresh the survey, it will still be broken. You should see a screen with question titles but no questions.

  2. Press the “Exit and clear survey” button, this will reset your survey responses and allow you to try again fresh.

  3. Rinse and repeat until you manage to successfully answer the first two questions and move on. It usually doesn’t take more than one or two tries. We haven’t received reports of the bug occurring past this stage.


If you encounter this please mail jd@fortforecast.com with details. Screenshots would be appreciated but if you don’t have the time just copy and paste the error message you get into the email.

 

Take the survey now


Meta - this took 2 hours to write and was reviewed by the slack.


My Table of contents can be found here.

Lesswrong 2016 Survey
New Comment
273 comments, sorted by Click to highlight new comments since:
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

I am literally pregnant right now and wasn't sure how to answer the ones about how many children I have or if I plan more. (I went with "one" and "uncertain" but could have justified "zero" and "yes").

Congratulations!

My wife is also pregnant right now, and I strongly felt that I should include my unborn child in the count.

Elo, thanks a lot for doing this.

(for the record, Elo tried really hard to get me involved and I procrastinated helping and forgot about it. I 100% endorse this.)

My only suggestion is to create a margin of error on the calibration questions, eg "How big is the soccer ball, to within 10 cm?". Otherwise people are guessing whether they got the exact centimeter right, which is pretty hard.

7Elo
Since you are such a huge part of the diaspora community I would be delighted if you could share the survey to both your readers and your friends. We will get that suggestion sorted asap.
1namespace
I actually can't do that. The way our survey engine works changing the question answers mid-survey would require taking it down for maintenance and hand-joining the current respondents to the new respondents. In general I planned to handle the "within 10 cm" thing during analysis. Try to fermi estimate the value and give your closest answer, then the probability you got it right. We can look at how close your confidence was to a sane range of values for the answer. I.E, if you got it within ten and said you had a ten percent chance of getting it right you're well calibrated. Note: I am not entirely sure this is sane, and would like feedback on better ways to do it. EDIT: I should probably be very precise here. I cannot change the question answers in the software, presumably because it would involve changing the underlying table schema for the database. I can change the question/ question descriptions so if there's a superior process for answering these I could describe it there.
9Scott Alexander
But unless I'm misunderstanding you, the size of the unspoken "sane range" is the entire determinant of how you should calibrate yourself. Suppose you ask me when Genghis Khan was born, and all I know is "sometime between 1100 and 1200, with certainty". Suppose I choose 1150. If you require the exact year, then I'm only right if it was exactly 1150, and since it could be any of 100 years my probability is 1%. If you require within five years, then I'm right if it was any time between 1145 and 1155, so my probability is 10%. If you require within fifty years, then my probability is effectively 100%. All of those are potential "sane ranges", but depending on which one you the correctly calibrated estimate could be anywhere from 1% to 100%. Unless I am very confused, you might want to change the questions and hand-throw-out all the answers you received before now, since I don't think they're meaningful (except if interpreted as probability of being exactly right). (Actually, it might be interesting to see how many people figure this out, in a train wreck sort of way.) PS: I admit this is totally 100% my fault for not getting around to looking at it the five times you asked me to before this.
5namespace
Yeah, you're right. Currently trying to figure out how to do that in the least intrusive way. EDIT: Good news it turns out that I can edit the calibration question 'answers' after all. The ones where a range would make sense have been edited to include one. Questions such as "which is heavier" have not been because the ignorance prior should be fairly obvious. Fri Mar 25 19:50:41 PDT 2016 | Answers on or before this date where the ranges have been added will be controlled for at analysis time.
1Scott Alexander
If you throw out the data, I request you keep the thrown-out data somewhere else so I can see how people responded to the issue.
5namespace
I don't throw out data. Ever. I only control for it. (Well barring exceptional circumstances.)
1Logos01
Even if he threw out the data I have recurring storage snapshots happening behind the scenes (on the backing store for the OSes involved.)

[Survey Taken Thread]

By ancient tradition, if you take the survey you may comment saying you have done so here, and people will upvote you and you will get karma.

Let's make these comments a reply to this post. That way we continue the tradition, but keep the discussion a bit cleaner.

I have taken the survey.

I have taken the survey.

I have taken the survey. I did not treat the metaphysical probabilities as though I had a measure over them, because I don't.

[-][anonymous]100

Similarly, I gave self-conscious nonsense numbers when asked for subjective probabilities for most things, because I really did not have an internal model with few-enough free parameters (and placement of causal arrows can be a free parameter!) to think of numerical probabilities.

So I may be right about a few of the calibration questions, but also inconsistently confident, since I basically put down low (under 33%) chances of being correct for all the nontrivial ones.

Also, I left everything about "Singularities" blank, because I don't consider the term well-defined enough, even granting "intelligence explosions", to actually talk about it coherently. I'd be a coin flip if you asked me.

So basically, sorry for being That Jerk who ruins the survey by favoring superbabies and restorative gerontology, disbelieving utterly in cryonics and the Singularity, and having completely randomized calibration results.

[-]gjm580

I have taken the survey.

I have taken the survey.

I have taken the survey. Yesterday.

[-][anonymous]570

I have taken the survey

I took the survey 2 days ago. It was fun. I think I was well calibrated for those calibration questions, but sadly there was no "results" section.

6MTGandP
Is it possible to self-consistently believe you're poorly calibrated? If you believe you're overconfident then you would start making less confident predictions right?

Being poorly calibrated can also mean you're inconsistent between being overconfident and underconfident.

5scarcegreengrass
You can be imperfectly synchronised across contexts & instances.

I have taken the survey. I like the new format.

I have taken the survey.

I've taken the survey.

Yet another survey be-takener here.

[-]Crab550

I have taken the survey.

I took the survey

I have taken the survey.

Survey: taken.

I have taken the survey.

[-]plex540

I have taken the survey.

I have taken the survey.

The survey has been taken by me.

Survey achieved.

I have taken the survey.

It is done. (The survey. By me.)

I took the survey.

I have taken the survey. :)

I have taken the survey.

I have taken the survey. I left a lot of questions blank though, because I really have no opinion about many of them.

[-][anonymous]500

Survey taken.

Just finished. I'm sure my calibration was terrible though.

I have taken the survey.

Took the survey, had the recurring survey confusion about some questions. For instance, I think some taxes should be higher and others should be lower. Saying I have no strong opinion is inaccurate but at least it seemed like the least inaccurate answer.

I took it.

[-]tut470

Me too.

RE: The survey: I have taken it.

I assume the salary question was meant to be filled in as Bruto, not netto. However that could result in some big differences depending on the country's tax code...

Btw, I liked the professional format of the test itself. Looked very neat.

I took the survey!

I have taken the survey.

I have taken the survey.

I have taken the survey.

I did My Part!

I have taken the survey.

Took it!

It ended somewhat more quickly this time.

I have taken the survey.

I have taken the survey.

Survey Taken

I have taken the survey

Took survey. Didn't answer all the questions because I suspend judgment on a lot of issues and there was no "I have no idea" option. Some questions did have an "I don't have a strong opinion" option, but I felt a lot more of them should also have that option.

I have taken the survey.

I have taken the survey.

I have taken the survey.

I completed the survey. I also like the new format - easy to read, good instructions etc.

For a few moments I was paralyzed with uncertainty about how humorous to try to make my "I took the survey" response, since many seemed to have made a similar attempt, thus this post took longer to finish than the survey itself, which I have taken.

I have taken the survey.

[-]Sjcs440

I have taken the survey.

The only option i think was missing was in the final questions about quantities donated to charities, an option such as "I intend to donate more before the end of the financial year" or similar. (and while likely not feasible, following up on those people in the next survey to see if they actually donated would be interesting)

Yar, have taken the scurvy survey, says I!

I have taken the survey.

I too have take the survey.

I have taken the survey.

I took the survey.

I have taken the survey.

I have taken the survey.

((past-tense take) i survey)

5taryneast
You've got a slight lisp there ;)

I have taken the survey.

[-]Pfft420

I took the survey!

I have taken the survey.

[-]Glen400

I have taken the survey

I have taken the survey.

I have taken the survey.

I think I spent about 1 hour and 20 minutes answering almost all of the questions. I'm probably just unusually slow. :P

Survey taken. By me, even.

Took the survey, and as others pointed out had some trouble with the questions about income (net? gross?) Also, is there any place where all the reading (fanfiction, books, blogs) hinted to in the survey are collected? I knew (and have read) some, but many I have never heard of, and would like to find out more.

Did it.

I have taken the survey

For the interests of identity obfuscation, I have rolled a random number between 1 and 100, and have waited for some time afterwards.

On a 1-49: I have taken the survey, and this post was made after a uniformly random period of up to 24 hours.

On a 50-98: I will take the survey after a uniformly random period of up to 72 hours.

On a 99-100: I have not actually taken the survey. Sorry about that, but this really has to be a possible outcome.

Have a 98% chance of an upvote.

I have taken the survey.

[-][anonymous]350

I've taken the survey.

Thanks Huluk for creating this subthread, very handy when reading others' comments about the survey itself.

I have taken the survey.

I took the survey.

Was taking it, and it crashed with a "This webpage is not available" error.

6namespace
We had some power outage related downtime for three hours or so, should be back up now.
6moridinamael
I'm a little unclear on how to proceed. I didn't establish a "save", so I can't really resume the survey. Does that mean I should start a new survey and pick up where I left off, or ... ?
4namespace
If you'd be willing to go through the trouble of doing it, yes that's exactly what you should do. I didn't think of that, thanks. Though from a data-consistency perspective people doing this would skew our response rate higher than it really is, I'd rather have the question data than an accurate response rate though so. shrug On the session timeout front, we're trying something out to make the sessions longer, which should cut down on that particular problem significantly.

Survey taken.

[-]Val310

Besides saying that I have taken the survey...

I would also like to mention that the predictions of probabilities of unobservable concepts was the hardest one for me. Of course, there are some in which i believe more than in some others, but still, any probability besides 0% or 100% seems really strange for me. For something like being in a simulation, if I would believe it but have some doubts, saying 99%, or if I would not believe but being open to it and saying 1%, these seem so arbitrary and odd for me. 1% is really huge in the scope of very probable or very improbable concepts which cannot be tested yet (and some may never ever be).

... before losing my sanity in trying to choose the percentages I would find plausible at least a few minutes later, I had to fill them based on my current gut feelings instead of Fermi estimation-like calculations.

I have taken the survey

Me, too! I've taken the survey and would like to receive some free internet points.

I have taken the survey.

I've taken the survey.

I've taken the survey.

I've taken the survey.

I completed the survey. Elo, thanks for organising this!

I have taken the survey.

I enjoyed the "yes, I worry about X, but only because I worry about everything" responses.

2taryneast
I really liked things like "option for people who aren't in the US and want an option to choose" plus I think I recall one like "I like clicking on options" :D
[-]ekr250

Survey has been taken.

Me! Me! I totally took the survey!

I've said it before and I've said it again - this is mild cult behavior.

... That being said, bring on the low cost gratification! I've taken the survey!

4hg00
Fun traditions might be undignified by the standards of academia, but they're perfectly normal in many other social contexts (small company, group house, etc.) You know what else exemplifies "mild cult behavior"? Burning Man! They give each other physical gifts instead of imaginary internet gifts. Even more problematic. If you are willing to define "cult" broadly enough, you can use the term to shut down any kind of cultural development. (Of course, cultural development that's already happened will get grandfathered in, the same way we don't call religions "cults" because they are too dignified and established.)
-2Lumifer
No, I don't think it does. Burning Man is an event and a community. I don't see any cultish tendencies around it.
3hg00
I was being sarcastic. My point was that the "cult" label can be hard to shake whether or not it's deserved--I analogized to Burning Man since it shares characteristics with LW, but was lucky enough to avoid getting labeled a "cult".
-2Lumifer
You really think it was just luck that BM didn't get a "cult" label and LW did..?
2hg00
I did not mean to say that it was "just" luck, but of course luck played a role (as it always does).

Took the survey before joining.

I have taken the survey

I have taken the survey.

I have taken the survey! Please reward my compliance.

I have taken the survey. It was fun, thanks!

Lo, I have taken the survey.

Taken.

I have taken the survey.

I have taken the survey. It was interesting, thanx to those who made it!

Oh right, I forgot this part. I have taken the survey (like two weeks ago)

I have taken the survey.

Newbie, done.

I took the survey for the 2nd year in a row. Can't wait to see the results.