the announcement post introduced the journal's core premise and its three key innovations: paid attributed review, published review summaries,[1] and LLM-assisted workflow;
The journal is being built to serve the alignment research community. This post’s purpose is to solicit feedback and encourage you to sign up on alignmentjournal.org if you want to participate. Our plans will be continually revised.
Goals
The primary function of a journal is to bring together exceptional researchers and focus their collective attention on identifying and elevating the best work. To reprise, our goals in concert with this are:
Improve and disseminate research on alignment that emphasizes advances in scientific and conceptual understanding.[2]
Combine the best features from traditional journals, ML conferences, internet forums, and social media while avoiding their pathologies: existing academic venues can be slow, opaque, myopic, and waste expert effort, while forums lack sufficient depth, filtering, and legible certification.
More accurately match papers to the best reviewers based on expertise and interest, making review more rigorous and carefully rationing scarce expert attention.
Bring researchers from traditional academia, frontier labs, and independent organizations together by providing a rigorously vetted, high-standards venue incentivizing deep contributions.
Filter and make legible the top alignment research results to outsiders, including to researchers in adjacent fields and to funders.
The innovations in review process, incentive design, and adaptation to AI tools are in service of the core proposition: leveraging the attention of people to accelerate progress in alignment. To that end, we must learn from the strengths and weaknesses of traditional journals, ML conferences, internet forums, and social media, drawing lessons from previous and ongoing efforts to improve academic review, as carried out by, among others, TMLR, The Unjournal, Distill, and NeurIPS.
Theory of change
The publication pipeline for academics and for AI safety researchers is ripe for disruption; there are many roadblocks to effective research that can be addressed with a well-designed intervention.
Prestige and information externalities in scholarly publication
Journals generally provide two goods: curation and public dissemination of research, and career capital for researchers. A third — belonging to and sustaining a community of peers — is important and explains a lot of why academics participate, but it complements the other two rather than competing with them, so we set it aside. The incentives governing the first two functions are potentially in conflict but can be brought into better alignment through improved institutional design.
From the perspective of the community of research users, the output of research — ideas, results, methods — is non-rival and imperfectly excludable, and its social value is maximized by broad dissemination and reuse. Research consumers need research outputs that are vetted for quality, cheap to access, easy to copy, and widely distributed.
From the perspective of the researcher, the relevant goods look different. Publication in a journal is best understood as an excludable and congestible good.[3] The journal provides ratification of one’s results, scholarly and public attention toward them, and consequent prestige and career capital. A new journal in a field that lacks one offers the prospect of generating these benefits for kinds of work that existing venues underweight. As long as the pool of researchers seeking to publish is small, these goods might be only weakly rivalrous. As the number of submissions increases, they become more strongly rivalrous. A new journal has only a limited store of attention, prestige, and career capital that it can distribute without diluting these goods by accepting work below the quality bar. Peer review and rejection are congestion control devices, intended to preserve the value of these goods.
There is therefore a presumptive tension between how journals can serve their two constituencies. Authors benefit from exclusivity which confers attention, career capital, and prestige. Readers benefit from establishing a baseline of validity but not further gating research beyond that. In the face of meager response by the dominant commercial journals to address this conflict, preprint servers have increasingly assumed a role as a demand-side solution, satisfying the distributional interests of research consumers.[4] But while the marginal cost of adding an additional reader to a digital artifact is close to zero, the marginal social benefit depends substantially on the step that preprint servers omit: validation of the research.
Separate validation from acclaim
A journal can ease this tension by separating validation from recognition. It needs to do three things. First, validate: provide the review step that preprint servers do not, and publish only work that has passed such validation. Second, distribute openly: make every published paper as cheap to access, link and cite as an arXiv preprint. Together these serve readers. They do little for authors, though. Being certified as sound earns some career capital indirectly, through the attention and impact that validation unlocks, but is unlikely to unlock the clout of acceptance in some flagship organ. The third step, then, is to supply the scarce good separately: choose a small number of the most notable of the papers already published and give them awards that bring prestige and attention.
Overlay journals already take the first two steps. They sit on top of a preprint server, run peer review, and certify accepted papers without hosting copies of their own. Some go further and add extra certifications for standout work. We plan to do all three, with a set of prizes for remarkable work at the top. If done right, these could jointly deliver the weight that publication in a highly selective journal carries in other fields.
This separation also solves a review-side problem. At a selective venue, reviewers are asked not only whether research is sound and worthwhile, but whether it deserves one of a limited number of slots: a rivalrous good that they themselves compete for, at the same time or at other times. It is unsurprising that this does not always produce constructive reviews. It is also an unreliable way to allocate acclaim, since reviewers generally do not see the field against which any given paper is competing.
Our acceptance bar is a serious one, but it is not a ranking. A paper must be correct, must advance our principled understanding, and must be the most valuable thing for at least a few researchers to read now. Each paper is judged against the bar rather than against the other submissions, so acceptance is not rivalrous in the sense above.[5] We plan to let reviewers nominate a paper for one of a small, fixed set of awards and make the case for it. Those decisions are made by a separate committee, chosen to minimize conflicts of interest, that sees the whole field of candidates. Acceptance thus signals that a paper has cleared a demanding bar, while the scarcer prestige is concentrated in the awards.
Separating validation from acclaim is not a new idea. PLOS ONE has reviewed for soundness alone since 2006, leaving importance for readers to judge after publication. eLife dropped accept/reject decisions in 2023. It now publishes reviewed preprints with an assessment that rates significance and strength of evidence on a fixed vocabulary, a cousin of our review summaries. Closest to our design, TMLR accepts papers on correctness and audience interest and recognizes significance through separate certifications. Its Outstanding certification can be awarded a year or more after publication. Its experience also points to the main risk of this approach: despite offering its own certifications, TMLR has faced a perceived “prestige gap” relative to the major ML conferences. We are betting that a more demanding acceptance bar will let acceptance carry weight in its own right, but that remains to be shown.
Remunerate reviewer labor
Peer review runs, mostly, on unpaid labor. In research communities with relatively homogeneous career objectives, review reciprocity (reviewing in exchange for being reviewed) can incentivize participation. This is common in ML conferences, e.g., ICLR, where reviewing is a condition of submitting. In alignment, many of the people best placed to review sit in commercial companies or nonprofit organizations, and may be indifferent to or actively discouraged from publication. They have nothing to get back. Reciprocity and community spirit alone are unlikely to sustain the collective epistemic enterprise of an alignment journal. We will bridge that incentive gap by paying reviewers.
Paid review has the additional advantage that payment can more straightforwardly be tied to desirable outcomes, such as review quality and insight, to avoid minimal-effort reviews that merely satisfy reciprocity conditions.
Paying for work people might have done out of goodwill can backfire. Payment can displace the motives it was meant to supplement, a risk we discussed in our features and policies post.[6] One well-known finding is that small payments can do worse than none at all (Gneezy & Rustichini 2000). This observation informs our design:
Payments are substantial, currently $500‒$1,700, so they read as a professional fee rather than a tip; we are explicitly valuing the work of reviewing.
Money is not the only reward. Reviews are attributed, and the review summary is a public, credited piece of work. Payment is an experiment we can scale back or drop if it misfires.
The evidence so far is encouraging but thin.[7] We will track review quality, invitation acceptance rates and who agrees to review, and survey reviewers on why they took part.
Desired outcomes
With these adjustments, we think that alignment research can develop a culture of peer review that it has, to date, often lacked. The requirement to pass peer review incentivizes serious engagement with existing literature, and anticipation of and engagement with objections before publication rather than after. Despite its faults, peer review can elevate the assessments of a few reviewers who deeply engage with the work rather than aggregating up many surface-level responses as is often the case on social media, especially for work that has yet to attract widespread attention. Peer review can also create new opportunities to connect work to neighboring fields.
Bringing peer review and a systematic, widely-indexed publication process to alignment research can furthermore make it more legible to prospective funders and policy makers outside of the alignment research community, who might balk at treating online forums or personal blogs as a source of research ground truth, and at relying on karma for peer review.[8]
Of course, many formal approaches to publication have led to systemic problems, such as contributing to a “publish or perish” mentality in which prestigious publications are seen as an end in themselves, instead of a means to advance a collective epistemic project. While there is no decisive way to avoid this risk, we can mitigate it by keeping acceptance a standard rather than a quota, and then allocating acclaim based not only on editor taste but on the observed usefulness of (and response to) a piece of research. The model should be prizes awarded ex post, in regular review cycles after publication, once a paper’s usefulness and reception can be observed, rather than decisions made entirely at acceptance.
Objections
We also acknowledge that online forums combined with a tightly-knit research community allow for a kind of rapid iteration and experimentation that peer review inevitably slows down. But the Alignment Journal would not seek to replace the forums and blog posts where ideas are often first worked out. Instead, it would provide a venue of record where the fully-developed versions of those ideas could be validated and distributed to the broader research community. We have prepared a downside risk pre-mortem on these and other issues.
Key performance indicators
These are some of the indicators that we will use to estimate the success of the journal, both subjective and objective. We will use these to diagnose problems, not as targets to optimize.
Who submits and reviews
Where authors and reviewers come from: the mix by sector (academia, frontier companies, independent) and by home discipline.
Do researchers submit their best research to us? “Best” is subjective of course, but conditional on whose taste you trust this is pretty measurable.
Review process (matching, rigor, speed)
Time between submission and publication. We expect things to start slow as we work out the kinks in the process, but our long-term goal is 1-2 months.
Time to first review, time in discussion and time to decision, tracked separately.[9] We want everything to go fast.
Review quality as rated by action editors, plus reviewers’ self-rated expertise as a check on matching.
Authors’ rating of how useful the reviews were, including authors of rejected papers.
Do people engage with, cite, and discuss the review summaries?
Legibility and prestige (outsiders, funders, career capital)
Article-level citations, including the share from outside the alignment community.
(Eventually) Impact factor and related traditional journal metrics.[10]
Downside risk pre-mortem
How could this journal have negative impact?
Low-quality research.We could lower the status of the field by publishing low-quality work in the “official” journal of alignment research.
Poor management.If we end up with “market power” but run the journal poorly, we could bring the field into disrepute and make it difficult for a better journal to get started.
Wrong focus. The boundaries and value of different types of alignment research are hotly disputed. We could overly reward or promote research that proves fruitless, misallocating researchers' effort and misleading outsiders on progress. Mitigation requires carefully defined criteria to make transparent decisions.
Narrow editorial taste becoming the field's definition of “real” alignment research. The editorial board is broad, but cannot span the whole field, and many calls are taste-based and made with limited information. Moreover, different sub-domains of alignment have differing appetites for a journal, some areas being already well-served by the flagship conferences, so we face selection bias.
Lack of sustainability/persistence.If we become the venue of record for important work and then lose funding or otherwise fold, our site and its contents could vanish, memory-holing the research we published. The more successful we are, the worse this would be.[11][12]
Publishing tools used by capabilities research. If we successfully improve academic publishing mechanisms, we could accelerate dangerous research such as the gain of offense-dominant capabilities or publish and elevate alignment work with capability externalities.
Waste of time. We might simply induce alignment researchers to divert some of their effort into journal activities without sufficient upside.
Goodharting. If awards/acclaim provided by the Alignment Journal come to have significant material benefits for recipients, we could end up incentivizing optimizing for acclaim as distinct from advancing the collective epistemic project of making powerful AI go well.
Reputation-washing. Frontier companies advancing AI capabilities could use the reputation and prestige of a publication in the journal to obscure harmful practices that do not live up to the standards advertised in their published work.
Crowding out intrinsic motivation. Paying reviewers could displace the goodwill that sustains peer review elsewhere. It could attract reviewers who are mainly there for the fee, or make unpaid reviewing in the wider field feel like a lesser obligation. We will watch review quality and the makeup of our reviewer pool, and change the payment scheme if we see either effect.
Funder dependence or capture. The journal’s business model depends upon philanthropic funding, thereby providing a mechanism through which the interests of funders may be imposed upon the journal. We seek to preserve our independence by relying upon a well-regarded editorial board to make decisions independently of such interests, staking reputation upon it. We welcome suggestions for further safeguards.
Extensions
If the journal is successful, it could become a good platform for several possible extensions:
Replication bounties. Offer payments to third parties to do post-publication replication for certain papers that the editors judge to be notable/important and are somewhat doubted by the community.
(For the time being, full replication of a paper is probably too costly, even with recent AI tools, to be done routinely, which is why we think it should not be part of the normal review process.)
This is especially attractive in the light of potential sharing of expertise or infrastructure with similar organizations, e.g., The Unjournal.
Public forecasts (or even prediction markets) about which publications will later achieve some objective correlate of importance or veracity, like citation count, replication, or mention in model cards or legislation. Some combination of editors, reviewers, or readers could participate.
Preregistration and registered reports. We could encourage authors to preregister their hypotheses and analyses, and ask reviewers to check the paper against the registration.[13] A strong version is the Registered Reports format, now offered by over 300 journals.
Retrospective review of existing published papers, either
With author participation (similar to traditional peer review)
Without author participation (i.e., third-party commentary)
An in-person conference.
Journal-to-conference track. In agreement with one or more conferences, we could “certify” some high-quality subset of papers we publish to get automatic inclusion in a track at the conferences.
See TMLR’s blog post on doing this for their journal with the NeurIPS, ICML, and ICLR conferences.
Conference-to-journal track. Is there virtue to anointing select papers with a DOI and a seal of approval? Note that we already permit substantial expansion of conference papers.
Review articles and pedagogy, perhaps commissioned, i.e., author payments. These might feature stronger editorial input and perhaps interactive content.
Appendix: Prior Art
This section contains our thoughts on a few particular existing projects related to reforming journals: Distill, the Journal of Open Source Software, the Unjournal, and Transactions on Machine Learning Research. We hope it gives a sense of how we think about these things.
Distill
Distill was a highly regarded, widely read, and unusual ML journal which focused on clear communication and pedagogy. It went on indefinite hiatus after less than five years of operations, and their post-mortem is strongly recommended reading. In our assessment, Distill had two main value-adds which, by the editors’ own assessment, were extremely labor-intensive:
intense (50+ hour) mentorship to authors; and
web-native interactive and dynamic visualizations.
We believe these are worthy goals, and we hope that Distill or another journal finds a way to take up this mantle again in a sustainable way; indeed, LLMs potentially greatly reduce the burden of building dynamic content. However, the primary goal of the Alignment Journal is distinct from Distill’s pedagogic goals: efficient and legibleassessment of research. We will focus instead on optimizing the use of reviewer time and effort.
If the Alignment Journal overall is successful, we could imagine branching out to more pedagogical work with stronger editorial input and interactive content (among many other possibilities; see Extensions), but that would only be done with caution.
Journal of Open Source Software
The Journal of Open Source Software (JOSS) is a diamond open-access journal for research software demonstrating a wide range of innovative editorial and review strategies which have informed the Alignment Journal's thinking.
Authors submit short papers (under 1,000 words, in Markdown) and the journal reviews both the paper and the underlying code. It launched in May 2016 and by 2023 was publishing roughly one paper per day, having just hit its 2,000th paper — all with no article processing charges. For the year 2019, while publishing about 350 papers, they put their cash costs at around $800 (about $21,000 counting donated services but excluding volunteer effort from editors and reviewers).
JOSS solves a different problem — giving credit for software — than we aim to solve in the Alignment Journal. Rather than the specific details, we take from them the lesson that it is possible to do open-access peer review at scale, with a fully open-source stack, which they have done for a decade.
Automation as philosophy
JOSS automates remorselessly for cost and editor/reviewer efficiency. In practice this means proof generation from Markdown, reference DOI checking via the Crossref API, repository analysis (license detection, language detection, lines-of-code count), Crossref XML generation, DOI minting, paper hosting, reminder emails for overdue reviews, and final publication are all handled by software. The only human steps are scope decisions, reviewer matching, reading the paper, and a final proofread before acceptance.
Chatops-driven publishing
Each review takes place in its own GitHub issue, with a JOSS bot participating. Assigning reviewers, generating proofs, checking references, and accepting papers are all done by typing @editorialbot <command> in that thread. This operational pattern is called “chatops-driven publishing” by founding editor Arfon Smith. This design has several beneficial consequences:
New editors learn by reading existing reviews. The commands are self-documenting; you do not need a training manual, just a few completed threads to read.
The audit trail is complete by construction. Every state transition — reviewer assigned, checklist item checked, proof generated, acceptance triggered — is a timestamped event.
Operational dashboards are easy. All state lives in structured, queryable form; thus, showing editor workload, time-in-review, and bottlenecks is just a database query over existing data.
The underlying technology (a Ruby framework called Buffy deployed on GitHub Issues) does not transfer to the Alignment Journal’s needs, which require semi-confidential review, per-user access controls, and paid workflow rather than public volunteer review. But the principle seems to contain design patterns worth investigating: keep editorial tooling inside the review interface rather than in a separate system, and design every workflow around minimizing the number of clicks, context switches, and decisions an editor has to make.
Open-source infrastructure
JOSS's stack is four loosely-coupled components:
a Rails web app for submission intake and dashboards;
a Pandoc-based document production pipeline (Inara) that produces PDF, HTML, JATS XML, Crossref XML, and CFF from Markdown source; and
a GitHub repository for archival storage with CI workflows for compilation and DOI deposit.
Each component can be replaced independently. Initial build cost was roughly $50K (0.25 FTE for a year). Their blog posts on cost models, scaling, and operations at 2,000 papers are worth reading for anyone designing a journal's technical infrastructure.
Where JOSS has struggled
Challenges that JOSS has faced are also worth noting for the Alignment Journal:
Review time creeps upward. Average time spent in review went from under three months (pre-COVID) to about four months by 2023. JOSS remains uncertain why; likely contributors include reviewer fatigue as the pool gets overused, submission growth without proportional reviewer growth, and the general post-COVID decline in volunteer engagement. One lesson from this is to instrument review timelines from day one, tracking time-to-first-review, time-in-discussion, and time-to-decision separately. Once a slowdown becomes visible it is much easier to diagnose where the bottleneck actually is.
Scope disputes consume disproportionate editorial time. JOSS initially handled scope questions ad hoc on a mailing list, which consumed substantial editor effort. They eventually moved to automated flagging (submissions under 1,000 lines of code get flagged, under 300 get desk-rejected), a mandatory “Statement of Need” section, and a weekly batch review of flagged submissions by the editorial team. For the Alignment Journal, scope decisions seem likely to be at least as contested. A documented, regular, low-overhead process for scope decisions from the start (rather than handling each case ad hoc) seems substantially preferable.
The Unjournal
The Unjournal offers an early-stage working example of an effort to unbundle expert evaluation from traditional journal publication. It is not itself a journal, and there are no accept/reject decisions. Instead, it overlays public evaluations (aka reviews), quantitative ratings, author responses, and manager syntheses on top of preprints and working papers, while remaining non-exclusive.[14] They focus on research with “potential for global impact” in economics, social science, policy, and forecasting rather than technical AI or ML.[15] Some aspects of their process are relevant here.
Substantive Innovations
Public evaluations/reviews/ratings, editorial synthesis
The Unjournal publishes the full “evaluation packages” with DOIs for every research object they consider. This creates a rich public artifact for later readers, funders, and policy makers, and it makes the labor of review legible instead of discarding it after an editorial decision. We're unlikely to fully go down this route: our journal still uses optionally-confidential referee reports and editorial selection. However, our proposed review summaries are motivated by a similar goal: reviewer attention should produce something more informative than a one-bit accept/reject, and a summary of those findings should be prepared with the reader in mind. Their “prize-winning evaluations” provide some strong examples of value add by creating a richer signal.
Structured evaluation and quantified ratings
As The Unjournal does not use an accept/reject filter, they emphasize quantitative ratings and predictions as a measure of research prestige and credibility. We're unlikely to ask for this level of structured rating for a typical publication, although we plan to give a small number of awards.
Prioritizing impact, links to impact-focused practitioners and stakeholders
The Unjournal's field specialist teams explicitly prioritize research for its potential real-world impact (along the lines discussed here). They also actively engage funders and practitioners as stakeholders, particularly through their Pivotal Questions projects and workshops. The Alignment Journal's editorial team might also consider impact in editorial selection and triage, and could take steps toward making evaluations legible to the broader alignment ecosystem, including companies, funders, and governance bodies.
Procedural innovations and tools
Compensating reviewers
The Unjournal pays evaluators (see their current policies), including promptness incentives and substantial cash prizes for the strongest evaluations. Their rationale cites evidence that financial incentives shorten turnaround times. The Alignment Journal plans a similar approach, paying reviewers and editors where existing norms leave their work under-incentivized and unpaid. Neither organization has yet published data on how payment affects reviews; we intend to.
Publisher-independent hosting and editorial management tools
The Unjournal has explored and adapted a stack of tools and platforms for managing their processes (research prioritization, identifying expertise, managing review, etc.), hosting their output, integrating it with the scholarly/bibliometric ecosystem (DOIs etc.), and engaging academics and practitioners. Their tools are generally linked to the open-access journal, open science, and “publish, review, curate” initiatives and communities. They are increasingly moving toward structured automations and code and database solutions, facilitated by AI coding tools.
Automation and the use of LLM tools to enhance reviewer/evaluator work
The Unjournal is exploring the use of LLMs to enhance evaluator work, including automated auditing of quantitative claims and structured checklists.[16] Their emerging policy requires evaluators to stand by any content in their report as their own judgment, and to disclose how AI tools were used, ideally providing links to chats or explanations. We share this general orientation: the goal is to build systems that facilitate responsible AI use and transparent reporting.
Transactions on Machine Learning Research
In machine learning, conferences are the journals. This is background knowledge for ML practitioners, but can be surprising to other scientists. The venues of record are a few giant annual conferences — chiefly NeurIPS, ICML, and ICLR — not journals. A paper accepted there is peer-reviewed, archival, citable, and is what counts for hiring and tenure. Thus there is a single fixed annual deadline per venue (so tens of thousands of papers arrive at once), review compressed into a few intense weeks by reviewers who just submitted their own work, and accept/reject decisions with only a short rebuttal and little of the revise-and-resubmit dialogue normal in a journal. Acceptance is gated on novelty and impact rather than mere correctness. Transactions on Machine Learning Research (TMLR) was built as a response to various strains upon this system.
Its parent journal, JMLR, was itself a protest.JMLR was founded in 2000 by MIT's Leslie Pack Kaelbling to make ML research free to read on the web, at a time when the field's main journal, Kluwer's Machine Learning, was paywalled and retained copyright control of submitted articles. In 2001 Michael I. Jordan coordinated a mass resignation — roughly 39–40 editors, about two-thirds of the board — who redirected their effort to JMLR. JMLR is now an umbrella covering the PMLR proceedings series, the MLOSS software track, and TMLR.
TMLR was announced on 12 December 2021 by inaugural Editors-in-Chief Hugo Larochelle, Raia Hadsell, and Kyunghyun Cho, with funding from CIFAR, Mila, the Vector Institute, and DeepMind. The founders cited the challenges of the conference system as motivation: “disenchantment with the exclusivity of our top conferences,” “slow turnaround,” “high-stakes pressure and stress around a handful of fixed deadlines,” and a perceived decline in the quality of the peer review process at the flagship conferences. TMLR is an overtly experimental solution, positioned somewhere between the conferences and original-flavor JMLR in its design choices. Peer review is conducted on OpenReview rather than JMLR's in-house system. And unlike JMLR, TMLR does not accept extended versions of previously published conference papers.
Design decisions:
Acceptance is based on correctness, not novelty or impact. TMLR’s acceptance criteria reduce the whole accept/reject decision to a handful of questions: (1) are the paper's claims “supported by accurate and convincing evidence”?, (2a) would “some individuals in TMLR’s audience be interested in the findings of this paper”, and (2b) does the paper communicate them clearly? Novelty, significance, and impact are explicitly not criteria, and reviewers are forbidden from rejecting on those grounds; authors whose claims overreach are asked to add evidence or “adjust (reduce) their claims” to fit what they have shown. A correct, modestly-scoped result is publishable on its merits.
Significance is judged separately. Significance judgments are made in optional, rarely-awarded certifications — Outstanding (≈ best-paper, ~once a year), Featured (≈ oral/spotlight), plus Reproducibility, Survey, and a journal-to-conference (J2C) certification that lets ~10% of accepted papers be featured at NeurIPS, ICML, or ICLR.
Rolling submission with no deadlines; each paper runs on its own timeline.
Fast turnaround backed by capped workload.Reviews are due within 2 weeks (4 for submissions over 12 pages) and decisions are targeted for ~9 weeks after submission, supported by a reviewer cap of roughly one paper at a time and ~6 reviews a year. Practical turnarounds can be longer and TMLR “does not make any guarantees.”
Per-paper action editors (equivalent to conference area chairs) who own a paper end to end, and a fully open review process on OpenReview.
Operational record and open tensions. TMLR’s annual reports put its median time from submission to decision for conference-length papers at about three months. It was 76 days in its first year (2022 report), and 91 days in both 2023 and 2025 (2025 report). That is faster than NeurIPS (126 days from deadline to notification in 2025) but no faster than ICML (about 91). TMLR has held that pace while submissions grew from about 100 a month in 2023 to about 250 in 2025, and they roughly tripled again in the year to mid-2026. The strain shows. The editors report that they “repeatedly run low on reviewers throughout the year”, and their median is well above the 60–70 days it would take if everyone met their deadlines.
Questions remain about the career value of TMLR papers. Because a plain acceptance deliberately says nothing about significance, commentators note a “perceived prestige gap,” and early-career researchers are “incentivized to shoot for conference papers above all else”, i.e., not TMLR. TMLR’s 2023 survey is more encouraging but self-selected. About three-fifths of submitters rated its accepted papers comparable to those at NeurIPS/ICML/ICLR, a fifth rated them better and a fifth worse. Reviewers and action editors were less positive. TMLR has sought conventional markers of standing: it obtained Scopus indexing in 2025 because “at certain institutions, publishing in journals indexed by these services is an important component for career advancement.”
The subjectivity of the acceptance criteria is still debated: once novelty and impact are removed, the second acceptance question — “would some individuals in TMLR's audience be interested in the findings?” — is subjective. This is phrased permissively — some individuals, not many, and with no judgment of how interested — so a rejection on this ground has to argue that essentially nobody would care, which is far harder to sustain than a conference reviewer's familiar “not novel enough.”
Lessons for the Alignment Journal
We share some features with TMLR: rolling submission, acceptance that does not necessarily demand strict novelty, and significance signaled separately through rare certifications, some awarded after publication. We depart from it on our acceptance questions. TMLR asks only whether some readers would be interested, a threshold most sound papers meet: in 2025 it accepted 70.6% of submissions, excluding withdrawals and desk rejections (2025 report). Our Value criterion asks instead whether a paper is the most valuable thing for at least some alignment researchers to read now, which is somewhat narrower. This makes acceptance itself a stronger signal, at the cost of a harder and more subjective judgment for reviewers, so we are wading back into a more delicate trade-off, which TMLR chose to avoid. Whether this is the correct posture remains to be seen.
Our working name for these at the time of that post was “reviewer abstracts”, to emphasize the parallels with the author-written abstracts, but we've changed to calling them “review summaries” as this seems to reduce confusion in practice.
By contrast, for many machine-learning conferences, there exists a perception that achieving SOTA on some benchmark is a speed-run to publication, and that the implied selection pressure might favor pure capabilities research. For a steelmanning of the case for benchmark-led science, we recommend Moritz Hardt’s Emerging Science of Machine Learning Benchmarks. Regardless, such targets are not those of the journal.
Potts et al. (2016) model a journal as a club in the economic sense, in which members “form groups in order to confer externalities on each other, subject to congestion.” The positive externalities are readers, citations and referees; the negative ones come from crowding, such as a lower chance of being published. Club goods are often summarized as excludable but non-rival, but club theory has always been concerned with congestion, and congestion is what matters for our argument in the main text: places in top traditional journals are severely limited, with many more submissions than acceptances, and the prestige of being one of the “top papers” presumably declines in .
In the happy case that we are overwhelmed by more high-quality papers than our finite review budget can assess, it might eventually become rival, in which case we would need to revisit this design.
In a randomized experiment with about 1,500 referees at the Journal of Public Economics, a $100 payment for meeting a four-week deadline cut median review times by a further eight days beyond the shorter deadline alone. Editors followed the referees’ recommendations at unchanged rates, and paid referees did not slow down once the payments stopped, which suggests little crowding out of intrinsic motivation (Chetty, Saez & Sándor 2014; summary). At Critical Care Medicine, offering $250 to 414 of 715 invited reviewers raised the share who completed a review from 42% to 50% (the rise in agreement to review, from 48% to 53%, was not statistically significant). Paid reviews came back about a day sooner, with no reported difference in quality (Inside Higher Ed).
Finke & Casper (2026) do this in their study of whether models downplay controversies involving their own developers: they preregistered their hypotheses and tests.
The Unjournal's evaluation process is also unusually structured. In addition to detailed reports, evaluators provide claim assessment and quantitative ratings across multiple dimensions, with confidence bounds. Their syntheses of these offer a layer between raw reviews and the paper itself: a reader can quickly learn what informed judges found important, fragile, contested, or decision-relevant. See their dashboard and data blogs on this.
They cover areas including global health, animal welfare, scientific innovations and the social implications of technological progress, including some work on AI risks and governance, and the economics of AI scaling.
The Alignment Journal is now open for submissions!
This is the fifth post in a series:
In this post we go deeper: presenting our theory of change — including the status and information economics of the journal; a survey of previous related projects; and the high-level conclusions that follow from this analysis. We list some of the key performance indicators we'll track, the downside scenarios we must avoid, and possible future extensions of the journal.
The journal is being built to serve the alignment research community. This post’s purpose is to solicit feedback and encourage you to sign up on alignmentjournal.org if you want to participate. Our plans will be continually revised.
Goals
The primary function of a journal is to bring together exceptional researchers and focus their collective attention on identifying and elevating the best work. To reprise, our goals in concert with this are:
The innovations in review process, incentive design, and adaptation to AI tools are in service of the core proposition: leveraging the attention of people to accelerate progress in alignment. To that end, we must learn from the strengths and weaknesses of traditional journals, ML conferences, internet forums, and social media, drawing lessons from previous and ongoing efforts to improve academic review, as carried out by, among others, TMLR, The Unjournal, Distill, and NeurIPS.
Theory of change
The publication pipeline for academics and for AI safety researchers is ripe for disruption; there are many roadblocks to effective research that can be addressed with a well-designed intervention.
Prestige and information externalities in scholarly publication
Journals generally provide two goods: curation and public dissemination of research, and career capital for researchers. A third — belonging to and sustaining a community of peers — is important and explains a lot of why academics participate, but it complements the other two rather than competing with them, so we set it aside. The incentives governing the first two functions are potentially in conflict but can be brought into better alignment through improved institutional design.
From the perspective of the community of research users, the output of research — ideas, results, methods — is non-rival and imperfectly excludable, and its social value is maximized by broad dissemination and reuse. Research consumers need research outputs that are vetted for quality, cheap to access, easy to copy, and widely distributed.
From the perspective of the researcher, the relevant goods look different. Publication in a journal is best understood as an excludable and congestible good.[3] The journal provides ratification of one’s results, scholarly and public attention toward them, and consequent prestige and career capital. A new journal in a field that lacks one offers the prospect of generating these benefits for kinds of work that existing venues underweight. As long as the pool of researchers seeking to publish is small, these goods might be only weakly rivalrous. As the number of submissions increases, they become more strongly rivalrous. A new journal has only a limited store of attention, prestige, and career capital that it can distribute without diluting these goods by accepting work below the quality bar. Peer review and rejection are congestion control devices, intended to preserve the value of these goods.
There is therefore a presumptive tension between how journals can serve their two constituencies. Authors benefit from exclusivity which confers attention, career capital, and prestige. Readers benefit from establishing a baseline of validity but not further gating research beyond that. In the face of meager response by the dominant commercial journals to address this conflict, preprint servers have increasingly assumed a role as a demand-side solution, satisfying the distributional interests of research consumers.[4] But while the marginal cost of adding an additional reader to a digital artifact is close to zero, the marginal social benefit depends substantially on the step that preprint servers omit: validation of the research.
Separate validation from acclaim
A journal can ease this tension by separating validation from recognition. It needs to do three things. First, validate: provide the review step that preprint servers do not, and publish only work that has passed such validation. Second, distribute openly: make every published paper as cheap to access, link and cite as an arXiv preprint. Together these serve readers. They do little for authors, though. Being certified as sound earns some career capital indirectly, through the attention and impact that validation unlocks, but is unlikely to unlock the clout of acceptance in some flagship organ. The third step, then, is to supply the scarce good separately: choose a small number of the most notable of the papers already published and give them awards that bring prestige and attention.
Overlay journals already take the first two steps. They sit on top of a preprint server, run peer review, and certify accepted papers without hosting copies of their own. Some go further and add extra certifications for standout work. We plan to do all three, with a set of prizes for remarkable work at the top. If done right, these could jointly deliver the weight that publication in a highly selective journal carries in other fields.
This separation also solves a review-side problem. At a selective venue, reviewers are asked not only whether research is sound and worthwhile, but whether it deserves one of a limited number of slots: a rivalrous good that they themselves compete for, at the same time or at other times. It is unsurprising that this does not always produce constructive reviews. It is also an unreliable way to allocate acclaim, since reviewers generally do not see the field against which any given paper is competing.
Our acceptance bar is a serious one, but it is not a ranking. A paper must be correct, must advance our principled understanding, and must be the most valuable thing for at least a few researchers to read now. Each paper is judged against the bar rather than against the other submissions, so acceptance is not rivalrous in the sense above.[5] We plan to let reviewers nominate a paper for one of a small, fixed set of awards and make the case for it. Those decisions are made by a separate committee, chosen to minimize conflicts of interest, that sees the whole field of candidates. Acceptance thus signals that a paper has cleared a demanding bar, while the scarcer prestige is concentrated in the awards.
Separating validation from acclaim is not a new idea. PLOS ONE has reviewed for soundness alone since 2006, leaving importance for readers to judge after publication. eLife dropped accept/reject decisions in 2023. It now publishes reviewed preprints with an assessment that rates significance and strength of evidence on a fixed vocabulary, a cousin of our review summaries. Closest to our design, TMLR accepts papers on correctness and audience interest and recognizes significance through separate certifications. Its Outstanding certification can be awarded a year or more after publication. Its experience also points to the main risk of this approach: despite offering its own certifications, TMLR has faced a perceived “prestige gap” relative to the major ML conferences. We are betting that a more demanding acceptance bar will let acceptance carry weight in its own right, but that remains to be shown.
Remunerate reviewer labor
Peer review runs, mostly, on unpaid labor. In research communities with relatively homogeneous career objectives, review reciprocity (reviewing in exchange for being reviewed) can incentivize participation. This is common in ML conferences, e.g., ICLR, where reviewing is a condition of submitting. In alignment, many of the people best placed to review sit in commercial companies or nonprofit organizations, and may be indifferent to or actively discouraged from publication. They have nothing to get back. Reciprocity and community spirit alone are unlikely to sustain the collective epistemic enterprise of an alignment journal. We will bridge that incentive gap by paying reviewers.
Paid review has the additional advantage that payment can more straightforwardly be tied to desirable outcomes, such as review quality and insight, to avoid minimal-effort reviews that merely satisfy reciprocity conditions.
Paying for work people might have done out of goodwill can backfire. Payment can displace the motives it was meant to supplement, a risk we discussed in our features and policies post.[6] One well-known finding is that small payments can do worse than none at all (Gneezy & Rustichini 2000). This observation informs our design:
The evidence so far is encouraging but thin.[7] We will track review quality, invitation acceptance rates and who agrees to review, and survey reviewers on why they took part.
Desired outcomes
With these adjustments, we think that alignment research can develop a culture of peer review that it has, to date, often lacked. The requirement to pass peer review incentivizes serious engagement with existing literature, and anticipation of and engagement with objections before publication rather than after. Despite its faults, peer review can elevate the assessments of a few reviewers who deeply engage with the work rather than aggregating up many surface-level responses as is often the case on social media, especially for work that has yet to attract widespread attention. Peer review can also create new opportunities to connect work to neighboring fields.
Bringing peer review and a systematic, widely-indexed publication process to alignment research can furthermore make it more legible to prospective funders and policy makers outside of the alignment research community, who might balk at treating online forums or personal blogs as a source of research ground truth, and at relying on karma for peer review.[8]
Of course, many formal approaches to publication have led to systemic problems, such as contributing to a “publish or perish” mentality in which prestigious publications are seen as an end in themselves, instead of a means to advance a collective epistemic project. While there is no decisive way to avoid this risk, we can mitigate it by keeping acceptance a standard rather than a quota, and then allocating acclaim based not only on editor taste but on the observed usefulness of (and response to) a piece of research. The model should be prizes awarded ex post, in regular review cycles after publication, once a paper’s usefulness and reception can be observed, rather than decisions made entirely at acceptance.
Objections
We also acknowledge that online forums combined with a tightly-knit research community allow for a kind of rapid iteration and experimentation that peer review inevitably slows down. But the Alignment Journal would not seek to replace the forums and blog posts where ideas are often first worked out. Instead, it would provide a venue of record where the fully-developed versions of those ideas could be validated and distributed to the broader research community. We have prepared a downside risk pre-mortem on these and other issues.
Key performance indicators
These are some of the indicators that we will use to estimate the success of the journal, both subjective and objective. We will use these to diagnose problems, not as targets to optimize.
Who submits and reviews
Review process (matching, rigor, speed)
Legibility and prestige (outsiders, funders, career capital)
Downside risk pre-mortem
How could this journal have negative impact?
Extensions
If the journal is successful, it could become a good platform for several possible extensions:
Appendix: Prior Art
This section contains our thoughts on a few particular existing projects related to reforming journals: Distill, the Journal of Open Source Software, the Unjournal, and Transactions on Machine Learning Research. We hope it gives a sense of how we think about these things.
Distill
Distill was a highly regarded, widely read, and unusual ML journal which focused on clear communication and pedagogy. It went on indefinite hiatus after less than five years of operations, and their post-mortem is strongly recommended reading. In our assessment, Distill had two main value-adds which, by the editors’ own assessment, were extremely labor-intensive:
We believe these are worthy goals, and we hope that Distill or another journal finds a way to take up this mantle again in a sustainable way; indeed, LLMs potentially greatly reduce the burden of building dynamic content. However, the primary goal of the Alignment Journal is distinct from Distill’s pedagogic goals: efficient and legible assessment of research. We will focus instead on optimizing the use of reviewer time and effort.
If the Alignment Journal overall is successful, we could imagine branching out to more pedagogical work with stronger editorial input and interactive content (among many other possibilities; see Extensions), but that would only be done with caution.
Journal of Open Source Software
The Journal of Open Source Software (JOSS) is a diamond open-access journal for research software demonstrating a wide range of innovative editorial and review strategies which have informed the Alignment Journal's thinking.
Authors submit short papers (under 1,000 words, in Markdown) and the journal reviews both the paper and the underlying code. It launched in May 2016 and by 2023 was publishing roughly one paper per day, having just hit its 2,000th paper — all with no article processing charges. For the year 2019, while publishing about 350 papers, they put their cash costs at around $800 (about $21,000 counting donated services but excluding volunteer effort from editors and reviewers).JOSS solves a different problem — giving credit for software — than we aim to solve in the Alignment Journal. Rather than the specific details, we take from them the lesson that it is possible to do open-access peer review at scale, with a fully open-source stack, which they have done for a decade.
Automation as philosophy
JOSS automates remorselessly for cost and editor/reviewer efficiency. In practice this means proof generation from Markdown, reference DOI checking via the Crossref API, repository analysis (license detection, language detection, lines-of-code count), Crossref XML generation, DOI minting, paper hosting, reminder emails for overdue reviews, and final publication are all handled by software. The only human steps are scope decisions, reviewer matching, reading the paper, and a final proofread before acceptance.
Chatops-driven publishing
Each review takes place in its own GitHub issue, with a JOSS bot participating. Assigning reviewers, generating proofs, checking references, and accepting papers are all done by typing
@editorialbot <command>in that thread. This operational pattern is called “chatops-driven publishing” by founding editor Arfon Smith. This design has several beneficial consequences:The underlying technology (a Ruby framework called Buffy deployed on GitHub Issues) does not transfer to the Alignment Journal’s needs, which require semi-confidential review, per-user access controls, and paid workflow rather than public volunteer review. But the principle seems to contain design patterns worth investigating: keep editorial tooling inside the review interface rather than in a separate system, and design every workflow around minimizing the number of clicks, context switches, and decisions an editor has to make.
Open-source infrastructure
JOSS's stack is four loosely-coupled components:
Each component can be replaced independently. Initial build cost was roughly $50K (0.25 FTE for a year). Their blog posts on cost models, scaling, and operations at 2,000 papers are worth reading for anyone designing a journal's technical infrastructure.
Where JOSS has struggled
Challenges that JOSS has faced are also worth noting for the Alignment Journal:
Review time creeps upward. Average time spent in review went from under three months (pre-COVID) to about four months by 2023. JOSS remains uncertain why; likely contributors include reviewer fatigue as the pool gets overused, submission growth without proportional reviewer growth, and the general post-COVID decline in volunteer engagement. One lesson from this is to instrument review timelines from day one, tracking time-to-first-review, time-in-discussion, and time-to-decision separately. Once a slowdown becomes visible it is much easier to diagnose where the bottleneck actually is.
Scope disputes consume disproportionate editorial time. JOSS initially handled scope questions ad hoc on a mailing list, which consumed substantial editor effort. They eventually moved to automated flagging (submissions under 1,000 lines of code get flagged, under 300 get desk-rejected), a mandatory “Statement of Need” section, and a weekly batch review of flagged submissions by the editorial team. For the Alignment Journal, scope decisions seem likely to be at least as contested. A documented, regular, low-overhead process for scope decisions from the start (rather than handling each case ad hoc) seems substantially preferable.
The Unjournal
The Unjournal offers an early-stage working example of an effort to unbundle expert evaluation from traditional journal publication. It is not itself a journal, and there are no accept/reject decisions. Instead, it overlays public evaluations (aka reviews), quantitative ratings, author responses, and manager syntheses on top of preprints and working papers, while remaining non-exclusive.[14] They focus on research with “potential for global impact” in economics, social science, policy, and forecasting rather than technical AI or ML.[15] Some aspects of their process are relevant here.
Substantive Innovations
Public evaluations/reviews/ratings, editorial synthesis
The Unjournal publishes the full “evaluation packages” with DOIs for every research object they consider. This creates a rich public artifact for later readers, funders, and policy makers, and it makes the labor of review legible instead of discarding it after an editorial decision. We're unlikely to fully go down this route: our journal still uses optionally-confidential referee reports and editorial selection. However, our proposed review summaries are motivated by a similar goal: reviewer attention should produce something more informative than a one-bit accept/reject, and a summary of those findings should be prepared with the reader in mind. Their “prize-winning evaluations” provide some strong examples of value add by creating a richer signal.
Structured evaluation and quantified ratings
As The Unjournal does not use an accept/reject filter, they emphasize quantitative ratings and predictions as a measure of research prestige and credibility. We're unlikely to ask for this level of structured rating for a typical publication, although we plan to give a small number of awards.
Prioritizing impact, links to impact-focused practitioners and stakeholders
The Unjournal's field specialist teams explicitly prioritize research for its potential real-world impact (along the lines discussed here). They also actively engage funders and practitioners as stakeholders, particularly through their Pivotal Questions projects and workshops. The Alignment Journal's editorial team might also consider impact in editorial selection and triage, and could take steps toward making evaluations legible to the broader alignment ecosystem, including companies, funders, and governance bodies.
Procedural innovations and tools
Compensating reviewers
The Unjournal pays evaluators (see their current policies), including promptness incentives and substantial cash prizes for the strongest evaluations. Their rationale cites evidence that financial incentives shorten turnaround times. The Alignment Journal plans a similar approach, paying reviewers and editors where existing norms leave their work under-incentivized and unpaid. Neither organization has yet published data on how payment affects reviews; we intend to.
Publisher-independent hosting and editorial management tools
The Unjournal has explored and adapted a stack of tools and platforms for managing their processes (research prioritization, identifying expertise, managing review, etc.), hosting their output, integrating it with the scholarly/bibliometric ecosystem (DOIs etc.), and engaging academics and practitioners. Their tools are generally linked to the open-access journal, open science, and “publish, review, curate” initiatives and communities. They are increasingly moving toward structured automations and code and database solutions, facilitated by AI coding tools.
Automation and the use of LLM tools to enhance reviewer/evaluator work
The Unjournal is exploring the use of LLMs to enhance evaluator work, including automated auditing of quantitative claims and structured checklists.[16] Their emerging policy requires evaluators to stand by any content in their report as their own judgment, and to disclose how AI tools were used, ideally providing links to chats or explanations. We share this general orientation: the goal is to build systems that facilitate responsible AI use and transparent reporting.
Transactions on Machine Learning Research
In machine learning, conferences are the journals. This is background knowledge for ML practitioners, but can be surprising to other scientists. The venues of record are a few giant annual conferences — chiefly NeurIPS, ICML, and ICLR — not journals. A paper accepted there is peer-reviewed, archival, citable, and is what counts for hiring and tenure. Thus there is a single fixed annual deadline per venue (so tens of thousands of papers arrive at once), review compressed into a few intense weeks by reviewers who just submitted their own work, and accept/reject decisions with only a short rebuttal and little of the revise-and-resubmit dialogue normal in a journal. Acceptance is gated on novelty and impact rather than mere correctness. Transactions on Machine Learning Research (TMLR) was built as a response to various strains upon this system.
Its parent journal, JMLR, was itself a protest. JMLR was founded in 2000 by MIT's Leslie Pack Kaelbling to make ML research free to read on the web, at a time when the field's main journal, Kluwer's Machine Learning, was paywalled and retained copyright control of submitted articles. In 2001 Michael I. Jordan coordinated a mass resignation — roughly 39–40 editors, about two-thirds of the board — who redirected their effort to JMLR. JMLR is now an umbrella covering the PMLR proceedings series, the MLOSS software track, and TMLR.
TMLR was announced on 12 December 2021 by inaugural Editors-in-Chief Hugo Larochelle, Raia Hadsell, and Kyunghyun Cho, with funding from CIFAR, Mila, the Vector Institute, and DeepMind. The founders cited the challenges of the conference system as motivation: “disenchantment with the exclusivity of our top conferences,” “slow turnaround,” “high-stakes pressure and stress around a handful of fixed deadlines,” and a perceived decline in the quality of the peer review process at the flagship conferences. TMLR is an overtly experimental solution, positioned somewhere between the conferences and original-flavor JMLR in its design choices. Peer review is conducted on OpenReview rather than JMLR's in-house system. And unlike JMLR, TMLR does not accept extended versions of previously published conference papers.
Design decisions:
Operational record and open tensions. TMLR’s annual reports put its median time from submission to decision for conference-length papers at about three months. It was 76 days in its first year (2022 report), and 91 days in both 2023 and 2025 (2025 report). That is faster than NeurIPS (126 days from deadline to notification in 2025) but no faster than ICML (about 91). TMLR has held that pace while submissions grew from about 100 a month in 2023 to about 250 in 2025, and they roughly tripled again in the year to mid-2026. The strain shows. The editors report that they “repeatedly run low on reviewers throughout the year”, and their median is well above the 60–70 days it would take if everyone met their deadlines.
Questions remain about the career value of TMLR papers. Because a plain acceptance deliberately says nothing about significance, commentators note a “perceived prestige gap,” and early-career researchers are “incentivized to shoot for conference papers above all else”, i.e., not TMLR. TMLR’s 2023 survey is more encouraging but self-selected. About three-fifths of submitters rated its accepted papers comparable to those at NeurIPS/ICML/ICLR, a fifth rated them better and a fifth worse. Reviewers and action editors were less positive. TMLR has sought conventional markers of standing: it obtained Scopus indexing in 2025 because “at certain institutions, publishing in journals indexed by these services is an important component for career advancement.”
The subjectivity of the acceptance criteria is still debated: once novelty and impact are removed, the second acceptance question — “would some individuals in TMLR's audience be interested in the findings?” — is subjective. This is phrased permissively — some individuals, not many, and with no judgment of how interested — so a rejection on this ground has to argue that essentially nobody would care, which is far harder to sustain than a conference reviewer's familiar “not novel enough.”
Lessons for the Alignment Journal
We share some features with TMLR: rolling submission, acceptance that does not necessarily demand strict novelty, and significance signaled separately through rare certifications, some awarded after publication. We depart from it on our acceptance questions. TMLR asks only whether some readers would be interested, a threshold most sound papers meet: in 2025 it accepted 70.6% of submissions, excluding withdrawals and desk rejections (2025 report). Our Value criterion asks instead whether a paper is the most valuable thing for at least some alignment researchers to read now, which is somewhat narrower. This makes acceptance itself a stronger signal, at the cost of a harder and more subjective judgment for reviewers, so we are wading back into a more delicate trade-off, which TMLR chose to avoid. Whether this is the correct posture remains to be seen.
Credits and thanks
This post has been informed by gracious contribution and feedback from Gautam Kamath, @Leon Lang, Konstantinos Voudouris, Geoffrey Irving, @Edmund Lau, @Yonatan Cale, @Alexander Gietelink Oldenziel, and @Marcus Hutter. All responsibility for errors resides with the authors.
Illustration: Rembrandt van Rijn, The phoenix or the statue overthrown (1658), Rijksmuseum.
Our working name for these at the time of that post was “reviewer abstracts”, to emphasize the parallels with the author-written abstracts, but we've changed to calling them “review summaries” as this seems to reduce confusion in practice.
By contrast, for many machine-learning conferences, there exists a perception that achieving SOTA on some benchmark is a speed-run to publication, and that the implied selection pressure might favor pure capabilities research. For a steelmanning of the case for benchmark-led science, we recommend Moritz Hardt’s Emerging Science of Machine Learning Benchmarks. Regardless, such targets are not those of the journal.
Potts et al. (2016) model a journal as a club in the economic sense, in which members “form groups in order to confer externalities on each other, subject to congestion.” The positive externalities are readers, citations and referees; the negative ones come from crowding, such as a lower chance of being published. Club goods are often summarized as excludable but non-rival, but club theory has always been concerned with congestion, and congestion is what matters for our argument in the main text: places in top traditional journals are severely limited, with many more submissions than acceptances, and the prestige of being one of the “top
arXiv (1991) grew out of the physics preprint-circulation culture, mainly as an attempt to solve for speed. Some journals, especially outside the oligopoly, have made serious attempts on the access side: PLOS, diamond OA, and JMLR for example.
In the happy case that we are overwhelmed by more high-quality papers than our finite review budget can assess, it might eventually become rival, in which case we would need to revisit this design.
For a survey, see Frey & Jegen (2001).
In a randomized experiment with about 1,500 referees at the Journal of Public Economics, a $100 payment for meeting a four-week deadline cut median review times by a further eight days beyond the shorter deadline alone. Editors followed the referees’ recommendations at unchanged rates, and paid referees did not slow down once the payments stopped, which suggests little crowding out of intrinsic motivation (Chetty, Saez & Sándor 2014; summary). At Critical Care Medicine, offering $250 to 414 of 715 invited reviewers raised the share who completed a review from 42% to 50% (the rise in agreement to review, from 48% to 53%, was not statistically significant). Paid reviews came back about a day sooner, with no reported difference in quality (Inside Higher Ed).
See LessWrong’s own Annual Review for another example of additional certification/award on top of the raw karma rankings.
This was a takeaway from the JOSS debrief.
Journals do not get an impact factor until they are in their third year at the earliest, since it needs two “prior years” as a baseline. Cf. the DORA critique of journal-level metrics.
In “Open is not forever”, Laakso, Matthias & Jahn (2021) found 174 open-access journals that disappeared from the web between 2000 and 2019.
We will be choosing an archival means such as CLOCKSS/Portico/Zenodo when we commence full operations.
Finke & Casper (2026) do this in their study of whether models downplay controversies involving their own developers: they preregistered their hypotheses and tests.
The Unjournal's evaluation process is also unusually structured. In addition to detailed reports, evaluators provide claim assessment and quantitative ratings across multiple dimensions, with confidence bounds. Their syntheses of these offer a layer between raw reviews and the paper itself: a reader can quickly learn what informed judges found important, fragile, contested, or decision-relevant. See their dashboard and data blogs on this.
They cover areas including global health, animal welfare, scientific innovations and the social implications of technological progress, including some work on AI risks and governance, and the economics of AI scaling.
Also see their related LLM-vs-human evaluation benchmarking project.