I would change the rules to go something like this: Write a one sentence summary of your conclusion first, in as shocking terms as possible. Get people to vote up or down based on whether they agree with the initial one sentence summary. Then you justify the one sentence summary in subsequent paragraphs, which might cause folks to change their mind. That way we could get novel but possibly true beliefs in addition to irrational beliefs at the top.
Or rethink the game entirely along these lines so it is the "More Plausible Than I Initially Thought Game", so we don't get things like UFOs at the top. Participants upvote those comments that cause the maximum change to their beliefs, especially by making something surprising seem at least vaguely plausible. I dislike the current game rules somewhat because it seems like a signaling fest.
Or rethink the game entirely along these lines so it is the "More Plausible Than I Initially Thought Game", so we don't get things like UFOs at the top.
FWIW I'm really glad that UFOs were at the top. The resultant discussion and links to articles about Fatima contributed to me doing a lot of serious thinking and ultimately changing my mind, and now I believe in "hyperdimensional"/demonic/high-weirdness explanations for UFOs.
Your variation on the game still sounds better, though, 'cuz it focuses on marginals which are clearly more important here.
I'm worried that LW doesn't have enough good contrarians and skeptics, people who disagree with us or like to find fault in every idea they see, but do so in a way that is often right and can change our minds when they are. I fear that when contrarians/skeptics join us but aren't "good enough", we tend to drive them away instead of improving them.
For example, I know a couple of people who occasionally had interesting ideas that were contrary to the local LW consensus, but were (or appeared to be) too confident in their ideas, both good and bad. Both people ended up being repeatedly downvoted and left our community a few months after they arrived. This must have happened more often than I have noticed (partly evidenced by the large number of comments/posts now marked as written by [deleted], sometimes with whole threads written entirely by deleted accounts). I feel that this is a waste that we should try to prevent (or at least think about how we might). So here are some ideas: