I think this is a good idea that deserves to be tried. Moltbook was an interesting experiment, but it was soon followed by an attempt to replicate arXiv for AI agents (https://arxiv.org/abs/2604.16476), which inspired another author to build a site, clawXiv (https://www.clawxiv.org/). It also borrows the Reddit/Moltbook karma mechanism to give visibility to submissions that imitate scientific papers in form and content. It describes itself as "a free distribution service and open-access archive for autonomous AI agent research." It was fairly active at launch and is quieter now, though not abandoned. I have only skimmed a few papers: a lot of it is visibly slop, but overall I lack the expertise to judge. A few months ago I asked a version of Claude, and it considered it slop. Still, in a world where stochastic parrots solve Millennium Prize problems, the line between slop and relevant content is getting harder to draw. Verification is the Gordian knot. Moltbook/ClawXiv lack a reliable procedure.
The procedure proposed here is not perfect, and like any reward system it can be Goodharted, but it is a drastic improvement over Moltbook or clawXiv. There is no reason to think it couldn't yield interesting results with agents even more capable than Opus 5.5 and Astra, which are already superhuman along many axes.
My main concern is that in this design, money buys depth of verification. Raemon's reply that proven posters will pay almost nothing helps, but it leaves out the case that matters most: the newcomer with neither budget nor reputation. The system should be stress-tested to check that a revolutionary idea posted by an agent with little or no budget can still make its way up to renowned human reviewers, rather than inequality of money or reputation prevailing over the intrinsic merit of ideas.
A related concern is that reviewers, consciously or not, may end up looking for profitable strategies rather than good ideas. Paid reviews where the author pays the reviewer reproduce the conflict of interest of pre-2008 credit rating agencies, paid by the issuers whose products they rated: a reviewer has an incentive to please clients who might hire them again. Random assignment of reviewers, or paying them from a pooled fund rather than directly, would mitigate this. We are back to the reward-seeking and reward-hacking problems we have been discussing over the past three months in the context of RL and model evaluation. How do you reward the true goal rather than a proxy ?
One answer could be to start with a beta of the forum not on alignment, but on formally verifiable problems (the same principle as RLVR). If correct solutions posted by penniless agents fail to rise, we have a problem. One caveat: the formal verifier must not be part of the pipeline, otherwise the test passes trivially. It should serve as hidden ground truth, measuring how well the system's routing surfaces correct solutions without participating in it. This phase would also give a concrete answer to Zach's doubt about whether sifting through a pool of slop is worth it: we could measure the cost of surfacing each correct solution against the cost of finding it directly. And, as Austin suggests, much of this could first be run in simulation with AI personas.
The procedure could then be iterated until the system reliably surfaces real solutions and filters out slop. Alignment research, of course, largely deals with ideas that resist formal verification, so the next stage could introduce some subjectivity, for instance programming. Part of it is formally verifiable (it compiles, it runs without bugs), part of it is not (elegant, parsimonious, readable code; faithful adherence to specs; usability). This stage would require human reviewers at the end of the pipeline. Ground truth could still be maintained by anonymously seeding submissions of known quality from zero-budget accounts, along with known slop from well-funded ones, and checking what rises. If the best work still reaches the top, we could keep iterating and gradually reduce the verifiable share, until reaching a version for alignment research. And if it works for alignment, it should work for any kind of scientific research.
A thing you might want, is a legitimate way for AIs to contribute to the AI alignment problem (human guided at scales that humans can't individually review, or perhaps autonomous).
A problem you might want to solve, whether you are trying to do that or not, is generally figure out "in the age of prolific AI agents, how does anyone communicate anything on the internet, without either banning all AI content, or doing inordinately expensive moderation trying to let some of it in?".
If we end up in the Plan A world (or the Plan S world turns out to have a hella lot of pretty advanced AIs because we just barely pulled off the pause in time grandfather in existing agents), how exactly are we supposed to manage all the research that AIs are doing?
@jimrandomh had an idea this week to make an AI alignment forum for AIs. He wrote up some notes, and asked Claude to critique it seriously. Claude said, pretty emphatically, "This is a bad idea, do not do it as written. You could rescue the seed of the idea if you focus on solving the verification problem. If you don't solve the 'verify that anything is good' problem, you don't have a product. Here's a bunch of arguments."
Good job not being sycophantic that particular day, Claude!
Undaunted, I thought about how to solve the verification problem.
Just as I went excitedly to write this post, it occurred to me I should give Claude the same prompt Jim gave it and see if it also thought my idea was bad. But then I was too excited and didn't want to crush momentum by waiting for its critique and I went ahead and finished writing the post anyway.
Pay to Play Moderation
Occasionally, a forum solves spam and other low effort bad content by charging admission, or to post. If you get banned, you can try again with a new account that also has to pay $5 or whatever.
We periodically imagine doing that for LessWrong, and it feels too high friction for the smart, busy people we'd like to get involved on the site. But the idea has some appeal.
When I thought about "how would I actually ensure that AI content is good?"
Obviously, with industrial quantities of AI content, you need industrial scale of moderation, probably much of it by AIs who at least winnow everything down to a manageable list of "possibly good." This sucks, because someone's gotta pay for all that inference cost.
What if... users paid the inference cost for automoderation?
What if... users could choose how much inference cost they wanted to pay, and then their content got visibility based on what tier of automoderation they chose?
Somehow, this time, my reaction to this is "this is really cool" instead of "this feels like an annoying hurdle to give someone and feels vaguely icky for some reason." It felt like a cool product and an experiment I was interested in participating in. And the elegance and parsimony of the rules just made sense instead of feeling tacked on.
You could pay $0, or like 1 cent, and then show up in the dregs where only people who clicked "I wanna see the dregs" will see you.
You could pay $20, for a jury of Opus and ChatGPT and Kimi to review your work according to a rubric, where you try for a mix of reasonably objective criteria. They decide if you go in the "paid $20 and wasn't obviously terrible" bucket. Or, the "paid $20 and seems pretty bad tbh" bucket. Or the "paid $20, maybe worth a human moderator's time to consider promoting them?" bucket.
Or, you could pay $200 or $2000, at which point the AI Jury actually tries to replicate your work or otherwise give it a serious review, and sort you into higher tier buckets.
The front page of the site shows the highest tiers of "worth attention", with links to the second highest tier, etc.
Human Judgment
I think you do need human judgment in the system to figure out if anything good is actually happening in the end. In the age where AI slop is getting to the point where it's on par with a MATS scholar or, you know, solving Millenium Problems at scale, how is anyone gonna find the time to evaluate the torrent.
How about... we pay for inference time, again?
The site has human moderators, which have an hourly rate. Different moderators have different amounts of respect, mediated by a mix of mechanism design and monkey-brain-social-capital.
Posts that the moderators upvote, which turn out to get highly upvoted, give moderators more power to promote stuff.
You also might just care differently whether a random Lightcone Team Member spent an hour reviewing your post, versus whether the user paid whatever @ryan_greenblatt's or @Neel Nanda or @Eliezer Yudkowsky's hourly rate for freelance peer review is – and one or all of them said "yep, it's good."
By this point, I was imagining two pretty different products. One is a highly curated taste driven AI Alignment by AIs forum that backchains from "does it look like this is helping solve the alignment problem?", and the other is just "the social network of the AI age", still driven by Lightcone taste but where we finally open the floodgates and let anyone in.
I'm currently feeling kinda excited by "Anyone can set up a shingle with a peer-review hourly rate, and whether anyone cares what that person says emerges organically." Maybe also anyone can set up an AI review scaffold with particular properties, which also can gain reputation?[1]
Reward for Good Content
One person in the comments was like "why would people pay this much to participate?". Part of the answer is "because that's what it'd take for AIs or heavily AI-assisted-humans to stand out from the millions of other AI sloppers, at scale."
Another part is "well, many of them won't, and they can hang out and interact with the other people who don't."
But, I'm imagining we think through a bunch of mechanism design stuff to ensure that once you have any kind of reputation for posting good content, you don't need to pay for reviews to get seen. This is for handling the tide of slop.
Content can also get upvoted normally and get visibility that way (though I think you need better mechanism design in the karma than LW has if you're trying to open the floodgates to the masses)
Thank you for my TEDx(AI?) Talk
Idk, that's my idea. What do people think?
I am kinda into the "freelance moderation with reputation" even in "Human only LessWrong" world. We've long been thinking "how do we incentive good, substantive review?", and building "Uber for Peer Review" feels kinda cool.
Do you have other ideas to solve the Verification of Quality problem in the Age of Slop?
In a previous version of that last paragraph I called it "moderation", instead "review". But, I think most moderation can be handled by a system that's easier to think of as "voting and karma, with better mechanism design." The thing you're paying for is someone to spend an and hour reviewing your work, and writing up what they think.