This is kind of a tangent to the subject, but seeing someone bring up Critical Rationalism on Less Wrong still brings up some pretty massive negative associations for me. By far the majority of all the mentions of Critical Rationalism on this site, due to monomania and prolific rate of posting, come from the author of the most downvoted post ever to appear on Main, and possibly the most disruptive user ever to frequent the site.

I'm thinking that the website should be strongly devoted to neutrality or objectivity, as is Wikipedia.

In Wikipedia part of being objective means to accurately report what's generally known about a topic and not engaging in original research. I think you will have a hard time to judge fallacies without people engaging in something like original research. Fox news has it's own brand of fair & balanced with isn't exactly the same thing that most people think of when they hear the phrase.

What kind of objectivity do you want for your website?

The status quo bias of one person is another person's Chesterton's fence.

Monolithic vs subjective As pointed out it's hard to gather everyones input to a single result. Rather than have a single fallacy / not fallacy rating have each user be able to express (and own) whether a statement is fallacious. In a usual case it would have the result of "95.4% people think this is a false dictonomy". However there is valuable information on cross-correlating what arguments pass which evaluator. You could have functionality to "ignore all evaluators that think this is a fair argument". People could also profilate themselfs as being quality evaluators. There is a problem/feature where the standard of the evaluator need not be rigour. You could for example have a profilic evaluator for each major political leaning. Or you could aggregate the information by cross-referencing proclaimed political identity ie "65% of self-identified democrats think this argument is fair"

Applicability vs context Being able to target already produced texts means there would be wide applicability. However I am a little concerned on selection effects on what makes it as a "thing to scrutinize". This kind of thing would be effective about small isolated arguments. However politicians that fit their arguments to fit the situation they are presented in could be wrongly presented in being judged outside of that speech situation. Maybe they know that there are better / more valid arguments for their position but choose to utter those they know their audience can relate to. Bringing those arguments under a close scrutiny would be to partly miss the point. I guess part of the idea would be to apply pressure to always use arguments that could pass harsher standards? However I can see many downsides to that. I would rather have all the arguments to be processed to be explicitly (re)created in the context of the website. Then it would be clear that everybody involved respects the clean play attitude and that the arguments are meant to be elaborate and precise. This could mean that only the core and essential points would be covered. That is, it would not be a witch hunt to harass other medias but be an internal matter.

explicitness vs summary score I would have each argument input in a special language/notation that forces every argument to be explicit and computer readable. The arguments would not be prose but collections and networks of semantic tokens. This would provide human language independence ie french and english users would render the tokens in their language but they would be manipulating the same exact ones when one makes a claim in french it would be accessible to the english user too. With the guarantee of computer readableness you could things like compare the axioms of two users and point where contradict, at such a point a discussion is possible. You could then track how often did those discussion shift opinions and which arguments were effective at which populations / belief bases. This could easily be rendered a tool for anti-knowledge seeking testing which manipulations work the best. If such a reduction is not done the meaning of any end result will be a bit nebulous. Its meaning would depend on the process by which it is produced and it would mask approval of a group in the guise of numeric inarguable data. If the vision of what the "clean play" consist off it could be useful but I doubt there is a single axis that would be so critically important to track. I would rather have metrics that tell stuff but don't give a conclusion than reach a conclusion I am not sure what it tells.

Such a website would probably be a good idea-- you don't need to have everyone using it, you just need a good-sized audience. Snopes and polifact are doing alright.

To my mind, the interesting question is choosing a manageable task that will make it easy for the site to grow.

This idea really isn't that original. Philosopher, journalists, and others have done this for quite some time, and this has, I'm confident, had significant positive effects on the public debate. Religious people were forced to improve their arguments as a result of philosophical criticism from Hume and others, for instance. Also, even if you might think that the political debate is bad, imagine how bad it would have been if there were no journalists reporting relatively objectively on politics.

My suggestion is thus just to make existing efforts at objective reporting and criticism more systematic and comprehensive: it's not a qualitative leap, but just a quantitative shift. I don't see why that would be impossible in principle (though it may be hard in practice) or why that would not have any effect, given how much our present institutions for objective criticism actually has achieved (in my view).

I have a very good friend who has taught collegiate level debate for forty years. Just before he retired, he did an experiment where he and his students would actually do what you are proposing here and point out, list, highlight and rebut the various forms of argumentative cheating on Facebook and Twitter and see what happened. The result? Of the thirty students in one class, seventeen were banned from groups and had their friends list drop to below 50 people. The remaining students found that there followers dropped, that they received less over all views, and that they became targets of more and more abusive comments with threads that would begin as discussions and quickly de-escalate to that all time internet favorite the argumentum ad hominum. That pattern held good across eight classes and 240 students (give or take). I grant that one professors off the cuff experiment does not equal solid research but it is perhaps a bit disheartening.

And that leaves out both cognitive dissonance and the backfire effect. No one wants to be proven wrong, and many will fight for their worldviews and the facts that fit them, even facts that are not facts.

And given that people also frequently admit to not using the (biased but) existent research tools that are already out there why would they bother to search for and use a research tool designed for something like this? I can see such a site rapidly being used as a weapon by one side of an argument and being shouted down as "fake news" by the other.

Also, just as an aside, I am a student at Duke University- School of Medicine and we are absolutely FORBIDDEN to use Wikipedia or any other crowd-sourced knowledge base as a source in our own researches. As one of the administrators pointed out, at least eleven of the professorial staff and hundreds of the students contributed to Wiki at any given time and it was a common hobby to go rewrite a rival's work.