While I agree completely with you about what you call the "caveats", "loose ends", and "cognitive load" imposed by them, it's ENORMOUSLY frustrating to read this in a public forum. I was already well aware of these problems years ago when I was unexpectedly scouted and hired by Metaculus and given a brief to fix them. But they turned out to be completely, wildly incompetent as an organization, like just dogshit incompetent. They had given contradictory instructions to other people and ultimately just didn't understand what they were even telling people to do, and they were extremely quick to blame their employees for problems without doing any actual study, just going with whoever complained about the other person first or whoever had the better resume. So they ended up firing me within a few months and continuing on to approve these garbage questions.
There is no reason why crowdsourced forecasting oracles actually need to be full of under- and over-specified "gotcha" questions. It's just that there has literally never once been one that was intensively curated by people who weren't idiots. If someone wanted to do that, they could. It is a problem of competence and of these platforms often being created by ideologues who aren't really forecasters at all.
While I agree completely with you about what you call the "caveats", "loose ends", and "cognitive load" imposed by them, it's ENORMOUSLY frustrating to read this in a public forum. I was already well aware of these problems years ago when I was unexpectedly scouted and hired by Metaculus and given a brief to fix them. But they turned out to be completely, wildly incompetent as an organization, like just dogshit incompetent. They had given contradictory instructions to other people and ultimately just didn't understand what they were even telling people to do, and they were extremely quick to blame their employees for problems without doing any actual study, just going with whoever complained about the other person first or whoever had the better resume. So they ended up firing me within a few months and continuing on to approve these garbage questions.
There is no reason why crowdsourced forecasting oracles actually need to be full of under- and over-specified "gotcha" questions. It's just that there has literally never once been one that was intensively curated by people who weren't idiots. If someone wanted to do that, they could. It is a problem of competence and of these platforms often being created by ideologues who aren't really forecasters at all.