What attributes make a task useful for rationality verification?
After thinking it over I believe I have identified three main components. The first is that the task should be as grounded in the real world as is possible. The second is that the task should be involve a wide variety of subtasks, preferably ones that involve decision making and/or forecasting. This will help insure that the effect is from general rationality, rather than from the rationality training helping with domain specific skills. The third is that there should be clear measure of successes for the task.
As I am not personally involved with the field I could be missing something important, but it seems like founding a successful startup would fulfill all three components. I propose that investigating the effect of giving startup founders rationality training would be a good basis for an experiment. Unfortunately, I do not know if it would be feasible to run such an experiment in real life. Thus, I am turning to the LW community to see if the people reading this have any suggestions.
I didn't go into details about exact exprimental methods for a couple of reasons. Partially because I assumed, apparently incorrectly, that it was obvious that any experiment for testing rationality would be conducted with the best experimental protocols that we could manage. But mostly, because I thought that it would be good to get feed back on the basic idea of rationally verification + startups ?= good before spending time going into detail about things like control groups, random assignment ect.
I welcome suggestions along those lines, and given the attention this has received will try to go back and add some of my own ideas when I have time, but wanted to make cleat that I wasn't intending this post as a detailed experimental design.
Also does anyone have any idea why the first part of this post has different spacing from the second? It's not intentional on my part.