I'm using all the spoilers I can find, and still find it a challenging game. Feel free to mock me. :)
A "spoiled" game of NetHack means you have precise numerical values of the upsides and downsides of various actions, e.g. rubbing a magic lamp. Or reading a scroll that you know from a shopkeeper's offer can be one of N scrolls, some of which have beneficial effects and others harmful. That definitely requires probabilistic decision-making - indulge in wishful thinking and you'll die often; play too cautiously (ignoring positive EVs of some actions with nasty downsides) and, well, you'll die often.
I suppose playing it "unspoiled" is even better, as you'd have to infer the frequencies from observation as opposed to having them delivered on a silver platter, as it were.
(ETA May 10th: finally Ascended as a Knight.)
Oh, no, I have no problems with people spoiling themselves for Nethack. That's pretty much the only way to actually win. But if your aim is to improve rationality, rather than to do as well as possible within the game, it might be better to play it unspoiled. After all, Morendil mentioned "hypothesis testing" as something that was taught by Nethack: The spoilt version doesn't really test that.
What I've found that the spoilt version of Nethack tests, more than anything else, is patience. Nethack spoilt isn't about scholarship, really. You don't study. You have a situation, and you look up things that are relevant to that situation. There is a small bit of study at the beginning, generally when you look up stuff like how to begin, what a newbie-friendly class/race is, and how to not die on the second floor.
But really, it's patience. I once did an experiment where players who were relatively new to Nethack were encouraged to spoil themselves as early and often as possible, and request advice frequently from better players. Really, to do anything short of having someone else play the game for you was not only allowed, but actively encouraged. Since I usually put a limiter on how willing I am to spoil myself on roguelikes, I thought this might be fun. (Namely, I'm unwilling to ask for any advice in tactical situations, only strategic ones: Which area should I go to next, instead of "How do I kill this ogre?")
Conventional wisdom for Nethack states that upon reaching the halfway point of the game, you should win from there if you play correctly. I got about three-quarters of the way there, on my third run, having never gotten past the second floor on my runs prior to those three. I died to a misclick, not to lack of knowledge or poor tactics. So, patience is the true virtue of Nethack: It's surprisingly easy to win as long as you spoil yourself, get advice, and don't screw up.
Sadly, the experiment only had the one participant actually try it, namely me, so the evidence shall remain anecdotal.
You can definitely only notice the rationality lessons if you already know them explicitly. I wonder how great the effect is when you're playing the game but don't know about the lessons. My guess: not very great. People can figure it out by instinct. It makes them use the hypothesis-testing skills they already have, not improve them. Most of the difficulty (for me at least) is not in the hypothesis testing, but in multitasking enough to find out what multiple balls do at once, then tracking them all as they bounce around the screen until you can tag them all.
Also, it's a short game, one you're unlikely to spend enough time doing for it to mold you. If you really want to get good at hypothesis testing, I recommend taking up programming. Debugging often stretches my hypothesis testing skills like nothing else I can think of.
I doubt it will much improve anyone's rationality. It does nicely illustrate a few issues on how science is done, and could be a fun way of explaining for the layman.
Harriet (and xerxes acting like harriet) is bugged to always move away from me (at least when I run the game). I have to beat levels containing her by guessing :P
The Blob Family is a simple game made by Leon Arnott. At heart, it's a game about testing hypotheses and getting the right answer with the least amount of evidence you can.
The mechanics work like so: Balls bounce around the screen randomly and you control a character who needs to avoid them. You can aim the mouse anywhere and activate a sonar. On the right side are rules for how various balls will react to this, and your goal is to figure out which ball is which. As you use the sonar more, the balls speed up, so it becomes more difficult to stay alive, thus giving an incentive to test your hypothesis in as few clicks as possible.
It very nicely illustrates the principle that, to test a hypothesis, you must design tests to falisfy your intuitions rather than to confirm them. For example, in one level, when you use the sonar:
I found myself mistakenly clicking in the center of the screen to test hypothesis 1, but this is insufficient. To design the proper tests, you need to keep the mouse out of the center, keep it away from you, and depending on the position of the balls keep it off a straight line from you.
It could also demonstrate the ability of a fast brain to test hypotheses quickly. For many levels, if you could slow time down and set up a very good test, you could solve the problem with a single click. But we humans aren't usually so attentive.
Just thought the LW crowd might enjoy it.