Like, any explanation of the form "psychology is harder for reason X, so physics had more impressive work earlier on, which drew in smarter people..." feels like a just-so story; it seems at least as plausible that more competent people will be drawn to more difficult problems. (Note that both the hypotheses put forward in the OP are of this form, so this is also a response to the OP.)

Thanks, John. I want to clarify how our hypotheses differ from the "just-so story" pattern you described.

Our hypotheses don't claim "psychology is harder for reason X, which led to physics attracting smarter people." Rather, we propose structural factors that may contribute to the *perception* of different disciplines' difficulty, independent of the researchers' capabilities.

I share your skepticism of just-so stories, which typically:

• Highlight compatibility between evidence and a hypothesis
• Fail to consider alternative explanations
• Don't generate novel, testable predictions
• Often lack explicit calibration about confidence

We've tried to avoid these pitfalls in several ways:

• First, we're explicit about our confidence levels. We present these as partial explanations among many factors that likely contribute to perceived disciplinary difficulty, not as comprehensive accounts.
• Second, our hypotheses can generate testable predictions. For example:
  • The Rigid Demands hypothesis makes it more likely that self-reported pre-commitment to specific questions should correlate with lower R² values across fields
  • The Fruit in the Hand hypothesis makes it more likely that the something like the Kolmogorov complexity of algorithms that solve "impressive" tasks in evolved social domains (like facial recognition) should be greater than those for non-evolved physical domains (like calculating rocket trajectories)
• Third, we've formalized our reasoning mathematically, which helps expose assumptions and clarify the scope of our claims.

Clearly, these are speculative conjectures regarding very hard questions. That said, we think our formal approach can help move us toward more structured hypotheses about a fascinating question—something we hope is marginally better than just-so stories. :)

This seems a crazy comparison to make.

Perhaps. I appreciate the prompt to think more about this.

Here's a picture that underpins our perspective and might provide a crux: 

The world is full of hard questions. Under some suitable measure over questions, we might find that almost all questions are too hard to answer. Many large scale systems exhibit chaotic behavior, most parts of the universe are unreachable to us, most nonlinear systems of differential equations have no analytic solution. Some prediction tasks are theoretically unsolvable (e.g., an analytic solution to the three-body problem), some are just practically unsolvable given current technology (e.g., knowledge of sufficiently far off domains in our forward light cone).

Are there any actual predictions past physics was trying to make which we still can't make and don't even care about? None that I can think of.

Here are a few prediction tasks regarding the physical world that we have not answered. What is the exact direction of movement of a single atom in the next instant? How do we achieve nuclear fusion at room temperature? What constitutes a measurement in QM? Why do we observe electric charges but not magnetic monopoles? Why did the Pioneer 10 and 11 spacecraft experience an unexplained deceleration? What is the long-run behavior of any real-world chaotic system? Is the sound of knuckles cracking caused by the formation or collapse of air bubbles? How many grains of sand are there on a given beach on third planet on the fourth star of Alpha Centauri?

But this list is deceptive in that it hides the vast number of problems we have not answered but don't remember because we were not pre-committed to their solution—we simply picked them up only to realize they were too hard or unpromising and we put them back down again on the search for more tractable and promising problems. 

As you say, you can't think of any problems in physics we couldn't solve and no longer care about! In labs across the globe, PIs, post-docs, grad students, and researchers of all stripes are formulating vast numbers of questions, the majority of which are abandoned. 

Scientists in the hard sciences routinely dismiss equivocal test results as unsatisfactory and opt to pursue alternative lines of inquiry that promise unequivocal findings. In many cases, the strength of typical inferences even makes statistical analysis unnecessary. The biologist Pamela Reinagel captures this common attitude concisely in this talk: "If you needed to do a statistical test, you just did a bad experiment," and "If you needed statistics, you are studying something so trifling it doesn't matter." What is happening is that we are redefining our problems until we find regularities that are sufficiently strong so as to be worth pursuing.

In contrast, the education researcher and the development economist are stuck with only a few outcome variables they're allowed to care about, and so have to dig through the crud of tiny $R^2$ values to find anything of interest. When I give you such a problem—explain the cause of market crashes or effective interventions to improve educational outcomes—you might just be out of luck for how much of it you can explain with a few crisp variables and elegant relations. The social scientist in such cases doesn't get to scour the expanse of the space of regularities in physical systems until they find a promising vein of inquiry heretofore unimagined. 

Of course, no scientist is fully unrestricted in her capacity to redefine her problems, but we suspect that there are differences in degree here between the sciences that make a difference.

Tell me if the following vignette is elucidating. 

You and your colleague are given the following tasks: you have 10 years in which to work on cancer come back with your greatest success, your colleague also has 10 years in which to work on cancer but they are also allowed to try to make progress on any other disease. Which do you expect will achieve the greatest success?

This case is simple in the sense that one domain of inquiry is a strict superset of another, which makes the value of larger search space more clear, but the difference in sizes of search space will be there in cases that are less obvious as well.

Our hypotheses suggest that the social scientists may be like the first researcher. They were pre-committed to smaller domain with less flexibility to go find promising veins of tractability in problem space. This, and variations on the general cardinality reasoning mentioned in footnote 7, are the core of our results.

Thanks! Yeah! That's just how we are thinking about it.

I like that observation, and it sounds just. Marxism, Keynesian economics, and various psychotherapeutic paradigms provide striking examples of theories that substantively influence the behavior they are meant to describe. And, as you say, the nature of their influence can be subtle and multifaceted—ranging from informing people’s expectations of others’ behavior and their own behavior to providing Schelling points for social coordination and introducing possible actions and strategies not previously salient or even imagined.

The best reference I know for a discussion of something like this is by the sociologist of science, Robert Merton in his 1948 "The Self-Fulling Prophecy." In it, he considers mechanisms by which economic, psychological, and sociological theories become self-reinforcing or self-negating.

That’s exactly right. Results showing that low-rationality agents don’t always converge to a Nash equilibrium (NE) do not provide a compelling argument against the thesis that high-rationality agents do or should converge to NE. As you suggest, to address this question, one should directly model high-rationality agents and analyze their behavior.

We’d love to write another post on the high-rationality road at some point and would greatly appreciate your input!

Aumann & Brandenburger (1995), “The Epistemic Conditions for Nash Equilibrium,” and Stalnaker (1996), “Knowledge, Belief, and Counterfactual Reasoning in Games,” provide good analyses of the conditions for NE play in strategic games of complete and perfect information.

For games of incomplete information, Kalai and Lehrer (1993), “Rational Learning Leads to Nash Equilibrium,” demonstrate that when rational agents are uncertain about one another’s types, but their priors are mutually absolutely continuous, Bayesian learning guarantees in-the-limit convergence to Nash play in repeated games. These results establish a generous range of conditions—mutual knowledge of rationality and mutual absolute continuity of priors—that ensure convergence to a Nash equilibrium.

However, there are subtle limitations to this result. Foster & Young (2001), “On the Impossibility of Predicting the Behavior of Rational Agents,” show that in near-zero-sum games with imperfect information, agents cannot learn to predict one another’s actions and, as a result, do not converge to Nash play. In such games, mutual absolute continuity of priors cannot be satisfied.