The idea is potentially useful but also potentially a giant minefield. Anything that can be hijacked and turned into a stupid rat race for status will be. I spent most of my time as a martial artist wishing I was a black belt, rather than wishing I was good at martial arts. Which is probably why I never got a black belt.
If we want to do this, far better it apply to lessons rather than people. It should be usable only the way you used it on the top of this post: as a way of stating a difficulty level. This is denotatively the same (if level 1.5 posts are appropriate to me, I must be a level 1.5 rationalist) but connotatively very different. There should be a holy unbreakable taboo upon ever mentioning levels in connection with a person.
But before I actually started putting the numbers on posts, I would want a much more precise description of what they mean. If you listed ten or twenty diverse OB posts along with the level you would assign each, that would help.
This is a minefield, and the historical experience is bad. The idea of 'Shock Levels' comes to mind. Post dependencies and suggested background reading seem vastly superior. I strongly oppose numerical levels.
Ok, here's a list, if you want more, email me at patrick.robotham2@gmail.com
The Third Alternative level 1
Procrastination level 0
Reasonable Disagreement level 1
Can Counterfactuals be True? level 2
Knowing about biases can hurt you level 1.2
Rationalization level 1.7
Modesty in a Disagreement level 2
A Fable of Science and Politics level 0
Explain/Worship/Ignore level 1
Foxes vs Hedgehogs level 2
I think we should be very careful about collapsing rationality into one dimension. Similar to Tolstoy's thought that "happy families are all alike; every unhappy family is unhappy in its own way", we know there is only one way to be rational, but there are lots of ways to be irrational.
You could imagine rationalists being arranged on a pyramid. Omega is at the pinnacle, and it fans out as you go lower. I would bet knowing your horizontal position is much more important than knowing your vertical position.
Even then, this ignores possible non-monotonicities in rationality. Being half a rationalist can make you worse off because you know how to explain away the opinions of others without relinquishing your own.
And how could we classify Robert Aumann? He has produced awesome work, but makes what most of us would call a low-level mistake. I agree work needs to be done to make our community more accessible, but I don't think we can gain much through simple classification.
These levels might represent levels in 'awesomeness' but they don't appear to represent levels of rationality.
It is certainly possible to be remarkably rational and not produce a great marvel of understanding. I do not think that producing such a work even proves that one is especially rational. It indicance they are Brilliant, persistent, well resourced and in the right place at the right time while somewhat correlating with rationality.
That said, I am comfortable with the word 'rationalist' largely to the extent that it isn't used often. Ascribing meaning to it beyond 'one who strives for rational thoughts or behavior' just doesn't appeal to me.
A charitable reading would be that mathematical expertise is necessary but not sufficient for rationality.
There is no level of developed skill at which it is no longer possible to compartmentalize. It's not a matter of technique, it's a matter of motivation and choice.
If there's going to be some attempt at quantifying rationality, it must address whether methods for countering innate human biases are actually being used, not merely known.
Intended for Levels 0-1.9 Related to: Playing Video Games In Shuffle Mode
Taking the advice of talisman, I thought it would be useful to compile a list of levels of general rationalist skill. I think these levels loosely correllate to how effective one is as a rationalist, but they are not the same thing, a level 2 might still compartmentalize. This list is more a specification of the difficulty of material that a person will be able to follow, as well as providing a list of levels above mine, that I can aspire to. Of course, skill is more of a continuum than a discrete set, I've leveled up once before, but I don't remember any single moment in which this occured, just like I don't remember any single moment where I suddenly matured.
The first level, level 0, is the rationality skill of an average 20 year old with internet access, which is the skill of the vast majority of people in the world, if not higher. They've probably forgotten what math they knew beyond a little algebra and arithmetic, and haven't gotten in to the habit of checking facts. These are the kind of people who think Spock when you mention logic. This level of skill is something any rationalist has to go through, so the majority of people who arrive here without coming from OB will probably fall in to this category. With any luck, these people might discover debate forums, and seeing someone shred an argument to pieces introduces them to the idea of a logical fallacy, and begins their transition in to level 1.
The transition from level 0 to level 1 is fairly painless. With a good reading list, it should probably take about three months. Level 1 is the stage where people start self-identifying as rationalists. The usual transition to this stage comes from thinking about rationality as rhetoric, many rationalists become rationalists after a long time spent arguing with creationists and other crackpots, and slowly developing a list of silly arguments and misconcpetions and why they're so silly, until they've got a big enough list that it becomes a matter of turning on the "bullshit detector". After a bit of practice doing this, people become more willing to think hard about their beliefs after their arguments are torn to shreds in the same way that they tore other people's arguments down.
Knowing (to the core of one's being) the fundamentals of logic: that a good argument should go from shared assumptions to a conclusion with clear and explicit reasoning every step of the way - marks a level 1 or higher. People at level 1 also generally know about the scientific method and the scientific ideas that crackpots talk about, some standard fallacies and cognitive biases, and understand the law of truly large numbers: that with a big enough sample space, all sorts of weird coincidences can happen. It's much easier to apply the techniques of rationality to propositions than it is to empirical expectations, which is why most books on the subject cover those, and why you should try to translate the latter in to the former when possible. Learning how to be rational about empirical expectations is the driving force behind transitioning to level 2.
The transition from level 1 to 2 is a rarer and more lengthly process. Level 2 is also much more specialized. Whereas level 1 consists mainly of reasoning about propositions, level 2 involves higher levels of abstraction i.e. thinking about the rules that are required to reason about propositions in a particular domain. To do this without committing serious mistakes requires a great deal of care and rigor, as well as knowledge of the relevant sciences and philosophical principles. When you realize that all those papers on cognitive biases and common fallacies also apply to you, you become immediately suspicious of anything that isn't absolutely explicit and precise. People at level 2 are also much more likely to admit when they don't know something, because the pattern of defining and using a precise and explicit model is nigh impossible when you don't know what you're talking about. Hence formalization is a very important technique at this level, which requires that people at level 2 know advanced math.
Becoming a level 2 isn't easy, it requires learning an advanced model of how to think about problems in a particular domain that may clash with your naive intuitions, being able to trust the math, and more importantly do the math is an essential skill, and one that is difficult to master. Programming and teaching are good practice techniques at this level, because they both require being explicit about your unconscious reasoning. The process of using a formal model to predict and explain things tends to give people at level 2 much better intuitions about problems in their domain of knowledge, as well as a sense of aesthetics about abstracted models. This sense of aesthetics is probably very important in helping a person trying to reach level 3 and come up with their own.
Level 3 is the level where you are able to produce awesome where you not only have an inutitive understanding of the mathematical models in your field, you also created them. At level 3 you don't have a guiding textbook to help you, you are the one that writes them. Very, very few people reach this level, and those who do are heralded as geniuses, such as Newton, Darwin, Godel, Russel or Einstein. I don't know how to become a level 3, I know Eliezer has had some experience trying, but these are the people who change the world.