It looks as if the teacher's password here is to answer "Yes", yet most of these questions call for a considered "No" or "Does not compute". To give a few concrete examples:
Do you, in every situation, endeavour to have an accurate map of the territory?
Seeking accuracy in every situation is extremely bad time management.
Do you regularly ask, ...
Depending on the sense of "regularly", explicitly reciting questions of doubtful practical use doesn't seem like a reasonable course of action. Some of these questions should be cemented into habits of thought, at which point you don't explicitly ask them, even as you benefit from their purpose.
Do you stop reading when a source becomes irrelevant?
Irrelevant how? Maybe it's pleasant reading.
Do you ever call topics or ideas boring, shallow, crazy, beneath you, or confusing (or other words that close off thought)?
Numerology, astrology, time cubism?..
This is great feedback, thanks! Let me try address some of it.
Seeking accuracy in every situation is extremely bad time management.
I agree; that's not what I meant to suggest. Not sure if I'll be able to convey this better. I'm trying to ask whether the person legitimately wants an accurate map (where it matters to them).
Depending on the sense of "regularly",...
I agree. Every question with a "regularly" or "frequently" or similar may just not work, period. How do I ask a question that determines if someone actually asks about the causes of their beliefs, whether consciously or habitually? It certainly requires self-awareness and self-honesty on their part. It could be changed to, "In the last 24 hours have you ..."
Irrelevant how? Maybe it's pleasant reading.
Irrelevant to the current investigation. If you're genuinely curious about what you're investigating, you'll drop irrelevant information in favor of what's higher value. I'll try rewording this.
Numerology, astrology, time cubism?..
I expect you don't spend time on astrology because you've deemed it false, not because you ignored it for being "shallow" etc.? In other words, I'm trying to ask "do you avoid thinking about topics for the wrong reasons?" This might not be possible. Also, I should probably drop the word 'boring'.
(Actually, the first thing I did after reading that was think, "time cubism, what?" googled it, and read about it for a few minutes.)
I'm uncomfortable with the idea of measuring aspects of rationality with checklists of specific mental habits, especially when the benefits of these habits are unproven.
is "unproven" the right yardstick? I don't know if you're making this mistake, but in our discussions of the efficacy of one or other part of rationality training it sometimes feels as if some people are asking "does the evidence permit us to believe" while others ask "does the evidence compel us to believe". The whole business of training in modern rationality is very new. It's unlikely anyone is very good at it yet, and it would be astonishing if we had really compelling evidence that someone was. It's good to ask how we can learn more about whether this is any good, but it's also OK to assess it on the balance of probabilities.
How do you assess the balance of probabilities in this case? If you go around asking people which mental habits help them in their day-to-day lives, you'll probably find that prayer is more effective than the OP's checklist. Or is there some more tangible evidence?
Can we flesh out what makes people uncomfortable about it? Is it primarily that the benefits of a specific skill/question are unclear/unproven?
AFAIK a list of proven habits doesn't really exist. As these do become clear, the checklist could be updated, improved, items added, items removed, etc.? I see potential value in that. Thoughts?
Do you have specific habits for getting curious when you notice you're not curious about something important?
How about something even more specific, like:
Can you think of the last time you noticed you weren't curious about something important? If so, when, and about what?
Did you take action about getting more curious in that case? If so, what action?
(To keep the checklist format, you could have a list of highly visible boxes to check, with some space for optional write-ins at the end of the questions - the specific question is just to help trigger the mind / make sure it's something specific)
You could even extend that to "can you list three examples", or allow scale ratings (that makes the checklist more cumbersome and less of a checklist, but more valuable as a long-term tracking tool).
Web app idea: I'm posting this comment immediately and without editing so I don't forget the idea before I get a chance to write it down/work it out more, as I have to leave my computer soon.
No priming by reading questions before passages. Writing their thoughts before seeing the question will hopefully keep people honest. Saving text box with answers allows answer auditing. Each passage's irrationality may be more or less obvious depending on a person's background. Same with desired/undesired thinking examples with questions (that's what we're measuring with this though, isn't it?).
Positive example question: Yes = +1, No = 0 Negative example question: Yes = -1, No = 0
With even split between positive/negative example questions, rationalists should score = 1/2 # of questions asked. More questions answered = more confidence in estimate. Wider range of topics addressed in questions = more confidence in estimate.
Edited to add: I created a storyboard for the app's testing process here and have started a list of example passages with desired/undesired responses here.
Is there a good reason we use "I desire to believe" rather than the more normal-sounding "I want to believe"? I understand it was originally done as a parody of religious creeds, but why are we still doing it?
How about trying to recall recent situations from one's life as a test of rationality? For example, you could make a checklist for a conversation about ideas you recently engaged in, or a checklist for an article on the Internet that you read. Participants could fill out checklist at random times, thinking of the last situation that matched the question.
I feel like it is a good idea to measure specifics, but that there are way too many questions here. Especially if, as you say, there are going to be other checklists for different skills. I couldn't even get through reading it, without applying serious effort.
I would feel overwhelmed if I was asked to actually ANSWER all of those questions. Even if I started out putting serious thought and effort into my answers, by the end I think that I would just be flying through them, answering anything without thinking about it, just to get it done. This means that the accuracy of the answers will go down.
My suggestion is to limit it to about 5 questions that you think give you the best estimation of what you are trying to measure (curiousity). For example, you can probably cut out number 15 (especially as a yes/no question); Anyone who says they NEVER call a topic boring or confusing, etc, is probably lying either to themselves or you. (I would guess that everyone finds SOME topic boring)
If it is short enough that people can maintain attention and effort throughout the whole survey, then you will end up with more accurate, thoughtful, and useful answers. You can still keep the longform set of questions as a useful guide for people to use personally.
If you DON'T make it any shorter, than I would say you should definitely break the filling-out process into many smaller chunks (i.e. do 5 before each meal or something.)
I am sure there are people on here that think that filling out such a survey is easy for them, and therefore should be easy for everyone else, but unless you are only sending it to hardcore rationalists and LessWrongians, I would say the majority of the population would find it difficultly long.
Good feedback, I went through the questions and had no such overwhelmed experience. I enjoyed it, actually, it reminded me of some of the areas where I've improved and some where I haven't but want to. I agree with your puzzlingness about the absolutes in some of these questions.
Also maybe a final question for fun: did these questions cause you to think and wonder what the answer is or did you enter a answer-for-the-sake-of-completing-the-survey mode? :)
I'm helping CMR create a 'rationality checklist'. The basic idea is this:
Sample application: before and after a minicamp. Target market: LWers.
I made a checklist for Curiosity (below). Others are in the works. A question like "Am I a curious person?", while it is useful to know, is too vague for this checklist and too susceptible to bias.
Any and all feedback is greatly appreciated. It is easy to answer yes/no? Is anything missing? Anything unclear? Would you find this useful?
Curiosity