Shouldn't you ask when the respondent thinks the Singularity will occur before mentioning the year 2100, to avoid anchoring?
I hate cognitive biases. I read your comment right before I went to take the test. "Ha!" I thought to myself, "clearly members of Less Wrong wouldn't be as effected. Why even bother mentioning it?" And then I clicked on the link while I thought about the singularity. "Hmm, 2100 is a decent year maybe it'll be 20 years before that though..." And I filled in my race/education/sex. "Hmm maybe it would be after that though, due to...oh god, it's the anchoring effect! Quick think of other numbers! 2090! 2110! Damnit. 1776! Wait that won't work..."
And as I slowly worked my way down, by brain tried in vain to come up with alternate years. Until I finally reached the problem. "Is this really what I think, or am I just putting this answer because of that comment in the thread?" But it didn't matter. The numbers were in the box, and I couldn't convince myself to change them.
There it stood: 2100.
PS. Yvain, any chance you could look at the mean/median/mode/standard deviation of that problem before and after you changed the questions around? I'd be very interested in seeing how people were effected by anchoring.
After reading the feedback I've made the following changes (after the first 104 entries so that anyone who has access to the data can check if there are significant differences before and after these changes):
Should anyone retake the survey? I'd be willing to if you can cancel the my first version-- I'll give the same answers on the Newton question.
Not as good as if someone can find a satisfactory IQ test, but could you add an SAT option for intelligence measurement?
I used percents for all my probabilities, including the one which was .5.
On the "Political" question: I identify with none of those. I understand the question is about which I identify with most, but all of the options have views on both social permissivity and economic redistribution. I am socially permissive, but have no belief one way or the other on redistribution/taxes. I simply have insufficient knowledge of that area to make a judgment. Perhaps it would be better to have two different questions - one for each of social views and economic views?
For "Religious views": I am an atheist but would not self-identify as either "spiritual" or "not spiritual". If a person asked me which I was, I would ask them what they meant by spiritual. I answered "Atheist but not spiritual", on the very weak grounds that I suspect I do not satisfy most other people's conceptions of spirituality; but really, the word is very ill-defined.
Thanks, Yvain! For the next survey, please consider country of residence and first language as questions.
I've been lurking on here for a long time, and just now registered to get a free karma point for taking the survey.
I did take the survey, however I found something I was unsure of what to put down and had to type in an explanation/question about:
It was for the question: "By what year do you think the Singularity will occur? Answer such that you think there is an even chance of the Singularity falling before or after that year. If you don't think a Singularity will ever happen, leave blank."
If I think the singularity is slightly less than 50% likely overall, what should I have put? It seemed off to leave it blank and imply I believed "I don't think a Singularity will ever happen" because that statement seemed to convey a great deal more certainty than 50+epsilon%. However, if I actually believed there was a less than 50% chance of it happening, I'm not going to reach an even chance of happening or not happening on any particular year.
As a side note, after taking that test, I realized that I don't feel very confident on a substantial number of things.
For the gender question it may make sense to have a generic "other" option. The monogamous/polygamous question should also maybe have a no preference option also.
Edit: And finished.
I think it is generally good to avoid "other" options as much as possible.
There are a few biases related to filling questionnaires. For example, many psychological tests ask you the same question twice, in opposite direction. (Question #13 "Do you think Singularity will happen?" Question #74: "Do you think Singularity will never happen?") This is because some people use heuristics "when unsure, say yes" and some other people use heuristics "when unsure, say no". So when you get two "yes" answers or two "no" answers to opposite forms of the question, you know that the person did not really answer the question.
Another bias is that when given three choices "yes", "no" and "maybe", some people will mostly choose "yes" or "no" answers, while others will prefer "maybe" answers. It does not necesarily mean that they have different opinions on the subject. It may possibly mean that they both think "yes, with 80% certainty", but for one of them this means "yes", and for the other one this means "maybe". So instead of measuring their ...
There are a number of types of snowflakes.
If you decide in advance that you aren't going to listen to anyone who doesn't fit your categories, you might be missing something.
You can have:
a) a survey, where everyone's individual differences are rounded into a few given categories;
b) a collection of blog articles, where everyone describes themselves exactly as they desire; or
c) a kind of survey, where some participants send a blog article instead of data.
Both (a) and (b) are valid options, each of them serves a different purpose. I would prefer to avoid (c), because it tries to do both things at the same time, and accomplishes neither. An answer "other" sometimes means "no answer is even approximately correct", but sometimes is just means "I prefer to send you a blog article instead of survey data". The first objection is valid, and is IMHO equivalent to simply not answering that question. The second objection seems more like refusing the idea of statistics. Statistics does not mean that people who gave the same answer are all perfectly alike, but ignoring the minor differences allows us to see the forest instead of the trees.
I guess the "special snowflake bias" is officially called "narcissism of small differences". The psychological foundation is that we have a need of identity, which is threatened by similar things, not different ones. So when something is similar to us, but not the same, we exaggerate the difference and downplay the similarity. From outside view we are probably less different than from inside view.
Would it not be useful for the “Degree” question to distinguish between the two no-degree cases of current undergraduate students and not-trying?
Thanks for doing this, I just took it. With the gender question, in addition to the transgender questions, it's maximally inclusive to include a non-binary "genderqueer" option.
Took the survey and finally created an account on here.
Looking at the comments, it seems like I am not the only one who used the survey as an impetus to create an account or a first post. I would be interested to see if there was a significant increase in the number of new accounts while the survey is running (as opposed to the average number of new accounts when there is no current survey).
...Also I took the IQ test posted in the comments.. Yeah, it has me as a good 15 points lower than what I was tested as in school also.
Survey now completed.
EDIT:
if you take it and post that you took it here, I will upvote you, and I hope other people will upvote you too
Let the record reflect that this comment currently has a negative score! :-(
EDIT2: No longer the case, obviously! :-)
I took it.
I think some of the "pick one" options were too broadly grouped, though any multiple-choice is going to be. I'd have preferred a "no preference" for "relationship style", for example, and more political options. Also I'm not sure what counts as "participates actively" in other groups--I've been a member of transhumanism-related groups for over a decade, for example, but am mostly a lurker; I did not check the box.
I would have been interested in seeing a question about involvement in offline activities like local meetups, or participation in IRC/other LW venues.
Thanks for running the survey!
Done. Definitely went through the whole "check the publication date"--whoop of victory--worry I was underconfident routine. Except silently because there's a sleeping person less than a foot away.
I'm amazed at the range of possibilities I considered for some of those probabilities. I definitely do not have a solid grasp of reality.
Filled out the survey. The cryonics-question could use an option "I would be signed up if it was possible where I live."
Yvain, one very important question that I think you missed: Do you currently have an account on Lesswrong?
I personally don't, and glancing through the number of 'first post' comments here, I believe that the ratio of lurkers to active users may be significant. (This is a throwaway account, and I am making an exception this once because there would be no other way to get information from the lurkers.)
Issues with the survey:
EDIT: Overall, it's pretty good.
Took it.
My family is of mixed religious background, so I just arbitrarily used my mother's religious background for those questions. You might want to make the answer choices a little more flexible.
I know "male, female, FTM, MTF, other" is a standard gender/sex question, but I don't know why. A problem is that it implies that "FTM" is a distinct category from, rather than a subset of, "male" (ditto for female). This would be better if other questions had answers that were subsets of other answers, but you seem to try hard not to do that. This could be fixed by phrasing it as "cis male", but then you'd get people complaining about "cis" and "trans" not being a perfect dichotomy and complaining about the confusing word and so on. This could also be fixed by splitting the question into "gender (male/female/other)" and "Are you trans? (yes/no)", but then you'd get other complaints.
I wouldn't have been too far off on the Newton question if I had been able to remember the mapping between century numbering and year numbering. I ended up two centuries off. Fortunately I took that into account when calibrating.
Also, for the record: I'm not "considering cryonics". I'm cryocrastinating. Cryonics is obviously the best choice, and I should be signing up for it in the next five seconds. I will probably die while not signed up for cryonics, and that will be death by stupidity, and you will all get to point and laugh at my corpse.
What kind of person asks about a baby's junk?
Most of them, by implication if nothing else. The minute they can't do so subtly, things get nasty.
As for me, I was surprised it asked about my racial background and my family's religion but not what country I grew up in or live in.
I completed the survey. Thanks, Yvain, for doing it!
The option "Atheist but spiritual" gave me a pause. What does it actually mean?
"Atheist" refers to the lack of a belief in gods. "Spiritual" includes all sorts of other supernatural notions, like ghosts, non-physical minds, souls, magic, animistic spirits, mystical energies, etc. Also, "spiritual" can refer to a way of looking at the world exemplified by religions that some atheists consider a vital part of the human experience.
This is great! I hope there's a big response.
It seems likely you're going to get skewed answers for the IQ question. Mostly it's the really intelligent and the below average who get (professional) IQ tests - average people seem less likely to get them.
I predict high average IQ, but low response rate on the IQ question, which will give bad results. Can you tell us how many people respond to that question this time? (no. of responses isn't registered on the previous survey)
I think it would be more informative to ask people to take one specific online test, now, and report their score. With everyone taking the same test, even if it's miscalibrated, people could at least see how they compare to other LWers. Asking people to remember a score they were given years ago is just going to produce a ridiculous amount of bias.
I think it would be more informative to ask people to take one specific online test, now, and report their score.
Are there any free, non-spam-causlng, online IQ tests that produce reasonable results (i.e. correlate strongly to standard IQ tests)?
Here's one that closely imitates Raven's Progressive Matrices and claims to have been calibrated with a sample of 250,000 people: http://www.iqtest.dk/
Here's another one: http://sifter.org/iqtest/ . I can't find any mention of where the questions came from or how it's calibrated, but it's shorter and doesn't require Flash.
Neither one asks for an e-mail address or any identifying information. They might be too easy for some on LW, but harder ones tend to cost money. As Viliam_Bur pointed out, any free online test's validity is questionable, but the first one is basically a direct copy of a "real" test, and neither one has any apparent ulterior motive. Anecdotally, they were both within 10 points of each other and my "real" score.
Yes - I'm quoting an IQ test I did as a kid which had a suspiciously high score, I'm pretty confident I'd get a much less spectacular score if I did one today.
Awesome. Definitely don't do another one then. (Unless you need to diagnose something of course!)
(how many people have strong feelings on Three Worlds Collide these days?)
(Of course to actually get the answer, you would presumably have to...take a survey. :-) )
Thanks for putting this together, Yvain! Recommendation to the Powers That Be: promote this to the main page so that more people notice it.
I'm not sure what it is about a survey that gets me to stop lurking at a community and actually create an account, but there you have it. Maybe it's just the chance to tell my 'story' anonymously.
I took it a few hours ago, and only just then realized that I apparently can get karma from saying so.
Posted. It wasn't clear whether the IQ calibration question was whether your IQ would be higher than the reported IQ of respondents or the actual IQ of respondents, and also whether that included respondents that didn't answer the IQ question.
Everyone should take the survey before reading any more comments, in case they contain anchors etc.
I took the survey. My estimates will be very poorly calibrated (I haven't done much in the way of calibration/estimation exercises) but I'm hoping they'll at least be good enough for wisdom-of-the-crowds purposes and more useful than just leaving blank.
Minor quibble: shouldn't "p(xrisk)" be "p(NOT xrisk)"? Just worried about people in a hurry not reading the question properly.
I took the survey, I found the "Moral Views" question very hard to answer to, folding "moral views" in one of 4 broad categories is surhuman effort for me ;) but I did my best.
Also, not wanting to enter a political debate here and now, but your definition of "communism" seems a strawman to me.
I've encountered people online who would want an "Other" option for the Gender question.
Also, my only possible answer to "Relationship Style" is "I don't know."
Edit: Survey filled, though. Left Relationship Style blank.
Just took it.
About the probability questions: I thought you were supposed to answer them instantly for your intuitive stance at the moment, without additional research, though I see some of responders apparently did research. Perhaps it should be better specified what is meant.
I just took it. My issue, which I haven't seen mentioned yet, is with the use of "agnostic" as a midpoint on the scale between theism and atheism. I realize that's a common colloquial use now but I don't get how it's a meaningful category -- unless it's meant to refer to negative atheism, and the "atheism" answers refer to positive atheism? And in the historical use of "agnostic" I think it's a separate category altogether that could overlap with both atheism and theism.
Overall I found the questions very interesting though, and I'm curious to see the results.
Filled out the survey. Neat!
I didn't know those versions of morality. There wasn't an option for "don't know" but I guess leaving it blank is the same thing.
To modify an example from the Stanford Encyclopedia of Philosophy: a Good Samaritan is widely agreed to be a good person, but the reasons vary:
The cryonics question is broken! I couldn't answer it without suspecting it would be misleading. My p would be incredibly low but only because my p for the human species surviving is low. This is a technically correct way to answer the question but I am not at all confident that everyone else would answer literally, including the obvious consideration "if everyone else is dead, yeah, you die too". Or, even if everyone did, I am not confident that the appropriate math would be done on a per-participant level in the results for the p(cryo) to be meaningful.
I took the survey. I'd really have liked an "other/no affiliation" option on the politics question, though, or a finer-grained scale. I suppose I could just have left it blank, but that seems not to transmit the right information.
Took it.
(Regarding the phrase "ontologically basic mental entity"; in my head, I always hear it in the voice of Raz from Psychonauts.)
Took the survey.
Thought you might have included an option for "reactionary" on the political orientation question. The distinction between reactionary, and libertarian or conservative is substantial even given the fact that the match isn't supposed to be perfect.
The global warming question might be more discriminating if the question were whether someone thinks that the mainstream view on AGW is scientifically valid within reason. The question as it stands is vague, hinging on the interpretation of "significant".
Otherwise a good survey!
Just took the survey. It was odd how only the word "Other" was translated into the Norwegian "Andre"... and everything else was in English.
Liberal, for example the US Democratic Party or the UK Labour Party: socially permissive, more taxes, more redistribution of wealth
Socialist, for example Scandinavian countries: socially permissive, high taxes, major redistribution of wealth
Only an American could have written something like that... Political "ideologies" apparently do not translate between countries in any way. It's like asking Muslims if they feel closer to Catholics or Lutherans.
The test has also a problem with extremely low "probability" events like "God existing". There's really no meaningful number between a vague "theoretically possibly just extremely unlikely" (and number of 0s you put there doesn't really mean anything) and "literally impossible 0%" here.
I took the survey. Sorry I asked you to keep my data private, but I precommitted to doing so in order to improve the quality of my responses.
Good idea, and a good set of questions. However, while I might say I'm fairly knowledgeable about a few topics anywhere else, the feeling of going far out of my depth is one I associate strongly with LW. As an example, I would expect the list of those who could hold a heavy AI discussion with LW's resident experts to be about 5 people.
Also, "exists" when referring to the entire observable universe, makes me a bit tense. In our past light cone? In our future light cone? In a spacelike interval? It makes a big difference.
I think the phrasing there will probably cause weird effects. For example, it seems most LWers have only vague ideas of biology and medicine, and I can talk confidently with a biology researcher or physician of average ability, so I felt happy checking that box. If everyone reasons like me, we’ll see lots of checks in that box, not because people here are expert in biology and medicine, but because we aren’t.
The "Anti-Agathics" question is ambiguous:
What is the probability that any person living at this moment will reach an age of one thousand years?
Two possible meanings (which, at least for me, would result in very different numbers):
Given a randomly selected person living at this moment, what is the probability that they will reach an age of one thousand years?
What is the probability that at least one person living at this moment will reach an age of one thousand years?
The final straw was noticing a comment referring to "the most recent survey I know of" and realizing it was from May 2009. I think it is well past time for another survey, so here is one now.
Click here to take the survey
I've tried to keep the structure of the last survey intact so it will be easy to compare results and see changes over time, but there were a few problems with the last survey that required changes, and a few questions from the last survey that just didn't apply as much anymore (how many people have strong feelings on Three Worlds Collide these days?)
Please try to give serious answers that are easy to process by computer (see the introduction). And please let me know as soon as possible if there are any security problems (people other than me who can access the data) or any absolutely awful questions.
I will probably run the survey for about a month unless new people stop responding well before that. Like the last survey, I'll try to calculate some results myself and release the raw data (minus the people who want to keep theirs private) for anyone else who wants to examine it.
Like the last survey, if you take it and post that you took it here, I will upvote you, and I hope other people will upvote you too.