"Community Notes" resolution for vague predictions

Raemon

Is there some kind of "get prediction markets, or predictions, onto twitter as a central object" project going? If so, how is it going?

I'm thinking through "how to raise median sanity" on a world scale. There are several incentive and institutional problems that make this very difficult. One angle is "try to make it a thing that the world tracks and cares about your predictions, and getting them right/wrong."

Two past angles here were:

Fact checker sites of the 90s/00s, which became politicized
Prediction markets, which I feel like aren't a good fit for most of what people would/should care about here. Operationalizing them is extremely annoying.

An idea for a twitter feature, which I think Musk should conceptually like given his stated values, and, I think he might have enough inertia to get behind, is:

When you make a prediction shaped tweet, it automatically gets converted into a prediction-y object*. AI suggests a few plausible operationalizations, or specific edge cases that seem likely to come up that you might want to address.
People can "like" predictions, predictions with a lot of attention get more automated followup.
Instead of resolving predictions with explicit careful operationalization, people vote on whether it seems like it resolved, and a community-notes-esque algorithm handles partisan disagreements.
popular predictions automatically get AI-generated community-notes-eque followup in terms of collecting evidence.

So, the final result will give a succinct "do The People think this resolved true?", which is some kind of percentage score that takes into account "how much it seems true to people" (if it's ambiguous / on-a-spectrum), and controversial the answer is. (TODO: figure out good mathy operationalization that is robust to partisanship)

But then, immediately underneath that, is a list of citations/evidence, so even if something is ambiguous, it's as easy as possible to see concrete facts related to it. (Also with community-notes-esque voting on relevance)

"Have you successfully made good predictions" is a badge / leaderboard that is optimized to be fun and rewarding.

The "community-notes-for-vague-prediction-resolution" mechanic could be implemented by Manifold or Metaculus or LessWrong or whoever, but, I think it's a lot more useful if it's on twitter. (tho maybe good to test it out on these places to iron kinks out and prototype UI).

My impression of the landscape is "Musk did really care about but truth, and believes in community notes in particular, but also went crazy after buying twitter. The Community Notes team is a small fiefdom he trusts and still believes in." So, insofar as this is pitched as an extension of that, might be doable.

Curious about takes from @Zvi and @Austin Chen.

Is there some kind of "get prediction markets, or predictions, onto twitter as a central object" project going? If so, how is it going?

Two past angles here were:

Fact checker sites of the 90s/00s, which became politicized
Prediction markets, which I feel like aren't a good fit for most of what people would/should care about here. Operationalizing them is extremely annoying.

An idea for a twitter feature, which I think Musk should conceptually like given his stated values, and, I think he might have enough inertia to get behind, is:

When you make a prediction shaped tweet, it automatically gets converted into a prediction-y object*. AI suggests a few plausible operationalizations, or specific edge cases that seem likely to come up that you might want to address.
People can "like" predictions, predictions with a lot of attention get more automated followup.
Instead of resolving predictions with explicit careful operationalization, people vote on whether it seems like it resolved, and a community-notes-esque algorithm handles partisan disagreements.
popular predictions automatically get AI-generated community-notes-eque followup in terms of collecting evidence.

"Have you successfully made good predictions" is a badge / leaderboard that is optimized to be fun and rewarding.

Curious about takes from @Zvi and @Austin Chen.

Thanks for substantive reply!

Yep, this is a big swing.

This is backchaining from "what could possibly help, for substantively raising the median sanity, or at least median sanity of the intellectual class?". Things that help more "on the margin" are also useful and people should do them but, man, we really need a lot of sanity really fast IMO. I think we will need some big swings.

That said, obviously if you were going to do this, you'd want to start with cheaper ways to prototype it. (i.e. experiment with it on Lesswrong, Manifold, glosso, etc, and initially roll it out as an opt in beta thing on twitter).

But, I want to start by asking "when we simulate the good, smooth version of the product, implemented magically perfectly, does the big swing even seem like it'd work?" and iterate on that conceptually before worrying about the simpler dogfood prototyping.

The big swing needs to engage directly with "what seems incentive compatible with 'most people don't care about truth, really' and "pundits and leaders actively resist submitting to processes that could rule them out" and "any process that has teeth will be a target of politicization".

Whomst among us has not spent time down the resolution criteria mines. It's not just about getting some criteria it's about getting criteria which nail the issue. If people read the criteria for "Is the strait of Hormuz open by October 1st, 2026?" is it gonna match what they think?

This seems to be missing the point of the post, though. The whole point is "resolution criteria is basically a hard blocker on accurate predictions going mainstream. So how do we completely route around it?"

The post is proposing the hypothesis, that to square that circle:

you make it happen frictionless/automatically on a big platform people already use.
you make that happen by selling Musk on it (which seems hard but doable? modulo prototyping it in lower stakes places first and, like, you know, make a world-class UI and sort out novel algorithmic problems or whatever, but, "that's the easy part")
you use LLMs to automate it as much as you can (in particularly aggregating and summarizing the object level evidence).
you use community notes algorithm to handle "resolve complex predictions quickly without having to deal with resolution criteria, in a way that will hold up."
use various UI tricks to make it feel like a fun game instead of a punishment.

I think the main thing the post is asking is "does #4 actually work?". It feels intuitively to me like a mix of "LLMs to make an initial ruling and highlight evidence, community notes voting to resolve controversial stuff" should work. But, I'm interested in thoughts from people who have more experience with the gritty details o that.

("is it actually possible to make this a fun game that mainstream people care about, or at least mainstream aspiring intellectuals, if you solve all the other problems?" feels like the second hardest part)

43

"Community Notes" resolution for vague predictions

43

43

43

"Community Notes" resolution for vague predictions

43

43