So my theory is that much of compartmentalization is simply because the search space is so large that people don't end up seeing that there might be a connection between two domains. Even if they do see the potential, or if it's explicitly pointed out to them, they might still not know enough about the domain in question (such as in the example of heavy boots), or they might find the proposed connection implausible.

If a person's knowledge is highly compartmentalized, and consists of these three facts:

1. A human being walked across the moon.

2. There are small rocks on the surface.

3. The moon is a planetary body.

without any educational background, would they choose the right answer?

I believe that their is a high probability that basic intuition would lead to an accurate answer.

So what went wrong, in your case? I don't think that you can attribute it to a failure of compartmentalization. It wasn't that you didn't make connections to your prior knowledge; the problem was that you made too many and that you hadn't organized your priors into a confidence hierarchy.

Confusion occurs when tenuous connections are made and lead to an over-analysis of the question. You differ from the person in the hypothetical, because you had prior knowledge of the forces involved. Connections are only helpful when they are made from strong foundational knowledge to new applications. When you are making many connections from a condition of uncertainty, to a new problem, your intuition fails. It results in the assignment of a low confidence level, to each of many connections, while ignoring basic observations or truisms.

It seems, you were confident in the areas of physics, most applicable in this situation; enumerating the atmosphere, gravity, and mass as the most influential. You attempted to remember how these forces interacted and recognized that they had dimensions to them, that you had forgotten. The connections caused your intuition to be replaced by a humbleness, and the go to answer was a balanced combination of forces. Thus the pen would float.

It is clear that you came to the problem with much more information than my hypothetical person, armed with three foundational facts. You too had those three facts, if that was all that was in your moon compartment the intuition would have been clearer.

(I apologize for the presumptions I made in referring to your thought process. This is a situation, in which, we find ourselves frequently.)

Rediscovering is not as prestigious as discovering, because it is not as difficult and does not signal intellectual greatness.

There is a difference between rediscovering and old idea, and adapting an old idea to a new situation. Simply rediscovering an old idea does not grant much prestige. Austrians are constantly coming across Hayek quotes and parading them around as definitive solutions to current problems. The problem is that these ideas are every bit as untestable as they were on the day Hayek wrote them. A confirmation bias leads Austrians to see them as Truth, while Keysians remain skeptical.

When old ideas are adapted into a testable form they endow a great deal of prestige. There are all sorts of anecdotes about this happening, such as Henry Ford taking the idea of an assembly line from Oldsmobile and mixing it with his observations from a meat factory, to create the moving assembly line. The difference is that this is a testable idea that creates immediate results.

progress means replacing one pile of sludge with another fashionable sludge-pile of similar quality.

The methods available to test these various hypotheses seem to have more of an impact on their prominence, than any objective measure of truth. Classical mechanics conformed to observations and could be confirmed by various tests. This led to widespread adoption until, observations were be made that did not fit the theories. Often the theories are available and cover various possible outcomes, all justified by the intuition offered by the current, yet untestable, theories.

This is where the social sciences run into difficulty. Predictions made by the social sciences are confirmed or disproved by the available methods of verification, at the time the predictions are made. These methods of verification evolve at a slower rate than the theories, and are always limited by the dynamic nature of human actors in large groups. Even if we could determine the utility function for everyone in the world, by the time that utility function had been applied and used to test various SS theories, they would have already changed.

It is unlikely that the LHC will produce results, not yet predicted by various physicists. When it does produce results, some thoeries will be proved and some will be disproved. The confirmation of the correct theory, however, is more valuable than 100 potentially correct yet untestable theories.

Mathematics has evolved quickly, for the same reasons that language has evolved, it is testable in its immediate ability to express and be understood. It has a very clean and objective measurement of success.