After reading the Arbital postmortem, I remembered some old ideas regarding a tool for claim and prediction aggregation.
First, the tool would have the basic features. There would be a list of claims. Each claim is a clear and concise statements that could be true or false, perhaps with a short explanation. For each claim, the users could vote on its likelihood. All these votes would be aggregated into a single number for each claim.
Second, the tool would allow the creation of composite claims by combining two existing claims. In particular, a conditional claim IF B THEN A would represent the conditional probability P(A|B). For every claim, it should be easy to find the conditionals it participates in, or the claims it is composed of. Conditionals are voted on same as simple claims (I would even consider a version where only conditionals are voted on).
Third, the tool would understand the basic probability laws and use this to direct the users' attention. For example, if three claims don't satisfy the law P(A|B) P(B) < P(A), users might be alerted about this error. On the other hand, if P(A|B) = P(B|A) = 1, the two claims might be merged or one could be discarded, to reduce the clutter.
Fourth, given a claim the tool might collect every other claim that supports it, follow every chain of argument and assemble them into a single graph, or even a semi-readable text, with the strongest arguments and counterarguments most visible.
Let's consider a possible workflow. Suppose you browse the list of claims, and find a ridiculous claim X assigned a high likelihood. You could just vote to decrease the likelihood and perhaps leave an offensive comment, however this is unlikely to have much effect. Instead you could find a convincing counterargument Y, then add both P(Y) = 1 and P(X|Y) = 0 to the list of claims. Now other users would be notified of the resulting inconsistency and would reply by voting on one of these claims, changing their vote on X, or by creating additional arguments that contradict Y or support X. In turn you would attack these new arguments, eventually creating a large graph of reasoning. Perhaps at some point there would be enough general claims that people from different debates could reuse some, instead of duplicating them, and only create new conditional claims, making the graph dense.
I took some time to write a small prototype. It is initialized with some AI related claims, and it implements first, second, and a tiny bit of third paragraph. Now for some meta. The prototype is a single page app, using angular1, backed by nodejs and mongodb. The actual features took between 1 and 2 hours to write. The backend and deployment took a couple more hours, largely because I hadn't done that in a while. Therefore I think that it's quite feasible to make similar prototypes for other ideas. Is there any value in it though?