Replying toHow Well Does RL Scale?

The Unjournal (unjournal.org) is considering commissioning this post for expert evaluation in our applied stream.

Looking for any feedback (here or privately) on whether this would be high-value, how to go about it (what particular issues/expertise), whether other research in this domain is higher value, etc.

Replying toHow Well Does RL Scale?

david reinstein2mo

How Well Does RL Scale?

I asked some LLM models/agents to consider this post, in preparation for considering this for some form of Unjournal.org evaluation. FWIW:

1. Conversation started with GPTPro

2. RoastmyPoast.org "epistemic audit" (result: C+, 68/100, which is a bit below average iirc)

3. Claude 4.5 Opus

My take: they saw this post as plausible, no major errors, and generating some useful insights, but with some important limitations, and the main claims are not all 'obviously demonstrated'

Below. Some overall syntheses/pulled quotes that seemed relevant to me. All folded content is LLM

GPTPro

What holds up (probability ~0.6–0.8):

Public evidence supports that inference budgets buy big gains on current reasoning benchmarks, and RL post-training scaling appears meaningfully less compute-efficient (often by ~2 extra

... (read 684 more words →)

Can we do useful meta-analysis? Unjournal evaluations of "Meaningfully reducing consumption of meat... is an unsolved problem..."

david reinstein

3mo

The Unjournal commissioned two evaluations of "Meaningfully reducing consumption of meat and animal products is an unsolved problem: A meta-analysis" by Seth Ariel Green, Benny Smith, and Maya B Mathur. See our evaluation package here.

My take: the research was ambitious and useful, but it seems to have important limitations, as noted in the critical evaluations; Matthew Janés evaluation provided constructive and actionable insights and suggestions.

I'd like to encourage follow-up research on this same question, starting with this paper's example and its shared database (demonstrating commendable transparency), taking these suggestions on board, and building something even more comprehensive and rigorous.

Do you agree? I come back to some 'cruxes' below:

Is meta-analysis even useful in

... (read more)

Replying toA Defense of Peer Review

david reinstein4mo

A Defense of Peer Review

I agree but I would not frame this as review in terms of thumbs up/thumbs down -- we can do better. In economics, for example, most people post their research in a fairly polished format online long before it makes it through the journal peer-review process. People can host their work in a variety of interesting and useful formats that actually go beyond what you can put in a frozen PDF of course.

Then we can have continuous public evaluation of this work, both crowdsourced and managed - at unjournal.org we do the latter, we pay experts to write detailed reports explaining the strengths, weaknesses, credibility, and usefulness of the research, and to... (read more)

Replying toUnjournal evaluation of "Towards best practices in AGI safety & governance" (2023), quick take

david reinstein6mo

Unjournal evaluation of "Towards best practices in AGI safety & governance" (2023), quick take

Naturally this paper is several years old. But it still seems like the most prominent work on this, with 61 citations etc.

My own take: we need more work in this area... perhaps follow-up work doing a similar survey, taking sample selection and question design more seriously.

I hope we can identify & evaluate such work in a timely fashion.

E.g., there is some overlap with
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5021463

which "focuses on measures to mitigate systemic risks associated with general-purpose AI models, rather than addressing the AGI scenario considered in this paper".

I'm eager to hear other suggestions for relevant work to consider and evaluate.

Replying toUnjournal evaluation of "Towards best practices in AGI safety & governance" (2023), quick take

david reinstein6mo

Unjournal evaluation of "Towards best practices in AGI safety & governance" (2023), quick take

My own take on this is that this suggests we need more work in this area. We need a follow-up with some follow-up work doing a similar survey, taking sample selection and question design more seriously.

Representative quotes from the evaluations

Sampling bias/selection issues

Potential sampling bias – particularly over-representation of safety-minded respondents
There’s a risk that the sample reflects the views of those already more inclined to endorse stringent safety norms. This is particularly important in light of the sample composition, as over 40% of respondents are affiliated with AGI labs

Question selection/design, need for sharper questions

As the authors note, this [agreement] may be partly due to the high-level and generally uncontroversial framing of the statements

david reinstein6mo

GovAI: Towards best practices in AGI safety and governance: A survey of expert opinion

By the way, just flagging that The Unjournal did an evaluation of this (post/discussion here -- I'll extend this with some more opinionated comments now). Overall it was taken to be a strong step but with important limitations and need for further caveats and further work.

Replying toGovAI: Towards best practices in AGI safety and governance: A survey of expert opinion

david reinstein6mo

GovAI: Towards best practices in AGI safety and governance: A survey of expert opinion

By "this is now the canonical collection" do you mean the ideas surveyed in the paper? Do you think it's still canonical or is it now ~out-of-date?

Unjournal evaluation of "Towards best practices in AGI safety & governance" (2023), quick take

david reinstein

6mo

The Unjournal commissioned two evaluations of the 2023 paper "Towards best practices in AGI safety & governance" from 2023. (Linkposted here)

The evaluators were generally positive, but identified some important limitations. My own take on this is that this suggests we need more work in this area. We need a follow-up with some follow-up work doing a similar survey, taking sample selection and question design more seriously.

You can read the full evaluation package at the link above (following internal links), including ratings and a discussion from the evaluation manager.

Representative quotes from the evaluations

Sampling bias/selection issues

Potential sampling bias – particularly over-representation of safety-minded respondents
There’s a risk that the sample reflects the views of those

... (read 214 more words →)

GPT-5 is out

david reinstein

6mo

Here's the system card.

Some quick takeaways (from a non-expert)

They merged the models and deprecated old ones.
It seems to choose whether it wants to use a reasoning model for you.
Seems to be a lot faster and have a lot less hallucination
You can now use voice mode with reasoning models/the frontier.
And new personalities (Cynic, Robot, Listener, Nerd)
Seems like big gains for developers (integrating GPT into their apps etc.)

From Wired:

[Altman called it] “a significant step along the path to AGI”

Two new model variants: a lightweight gpt-5-mini and an even faster, cheaper, gpt-5-nano (which is only in the API).

OpenAI says that the $200 a month Pro tier offers unlimited GPT-5 access, along with

david reinstein7mo

The Unjournal's "Pivotal Questions" project

Unjournal’s first Pivotal Question, focusing on the viability of cultured meat — This post also gives concrete details of our process and proposed approach.

The Unjournal's "Pivotal Questions" project

david reinstein

7mo

This EA Forum sequence explains The Unjournal's Pivotal Questions Project, and tracks our progress. (Unjournal: in a nutshell.)

The "Pivotal Questions" project process (piloting):

Elicit ‘target questions’ from global-impact-focused organizations, aiming at high VOI for funding and policy choices
Select the most useful target questions, get feedback, and collaboratively refine them to make them more precise, tractable, and decision-relevant
Elicit stakeholder and expert beliefs over these questions, including through (Metaculus/Minitaculus) prediction markets
Source and prioritize research informing the target questions
Commission expert evaluations of this research, focused on how the research informs the target questions, and eliciting beliefs over these
Get feedback from research authors and the target organization(s), and updated beliefs about the target questions
Prepare a ‘synthesis report’, quantifying our

... (read 182 more words →)

Replying toAnnouncing the Double Crux Bot

david reinstein7mo

Announcing the Double Crux Bot

I was indeed looking for something that could be used in a live conversation.

Replying toAnnouncing the Double Crux Bot

david reinstein11mo

Announcing the Double Crux Bot

Is there a version of this bot (or something similar) that one can use in an LLM model or website? I want to use this on a podcast without having to link this to a Slack

'Chat with impactful research & evaluations' (Unjournal NotebookLMs)

david reinstein

Post status: First-pass, looking for feedback, aiming to build and share something more polished and comprehensive

The Unjournal: background, progress, push for communications and impact

The Unjournal is a nonprofit that publicly evaluates and rates research, focusing on impact.

We now have about 30 "evaluation packages" posted (here, indexing in scholarly ecosystem). Each package links the (open-access) papers, and contains 1-3 expert evaluations and ratings of this work, as well as a synthesis and the evaluation manager's report. Some also have author responses. We're working to make this content more visible and more useful, including through accessible public summaries.

Also see:

our Pivotal Questions initiative
our regularly updated 'research with potential for impact' database

Notebook/chatbot exploration (esp. NotebookLM)

We're considering/building... (read 416 more words →)

Calibration training for 'percentile rankings'?

david reinstein

Does this tool/site exist? What's the best/easiest way to build it?

I'm looking to find or build a tool for calibration training for 'percentile rankings'.

E.g., give me a list of dog photos to rate out of a common pool, and ask me to give percentile rankings for each ('as a percentage, where does this dog fall in the disribution of dog quality, among all the dog photos in our database?).

The tool should graph my percentile rankings against a uniform distribution. It should provide a metric for 'the probability that my rankings were drawn from a uniform distribution.'

It might be ideal to have these spaced out to keep me from ~cheating by remembering... (read 332 more words →)

Checking public figures on whether they "answered the question" quick analysis from Harris/Trump debate, and a proposal

david reinstein

Public figures often avoid answering a question; going off in a different direction without recognizing that they didn't answer it. (Sometimes they even say 'I answered your question' even though they didn't.

I wish they still said 'no comment'.

Non-answers without acknowledgement seems bad for public epistemics as well as for good governance and public choice.

In the linkpost, I report on a quick Anthropic/Claude analysis of the extent to which Harris and Trump actually answered the questions they were asked in last night's debate. (TLDR: neither did, but Trump did substantially worse.)

I suspect it would be easy to make a fairly useable tool to judge 'was the question answered?' in real time. I think bringing this into debates and interviews could add a lot of value.

A possible check against motivated reasoning using elicit.org

david reinstein

Are you worried you may be engaging in motivated reasoning, rationalization ... or committing other reasoning fallacies?

I propose the following epistemic check using Elicit.org's "reason from one claim to another" tool

Whenever you have a theory that $A \to B$ ,
Take your theory negating one side or the other (or the contrapositive of either negation), and feed it into this tool.

Feed it
$A \to \neg B$

and/or
$\neg A \to B$ ,

and see if any of the arguments it presents seem equally plausible to your arguments for $A \to B$ .

If they seem similarly plausive, believe your original arguments and conclusion less.

Caveat: the tool is not working great yet, and often requires a few rounds of iteration, selecting the better arguments and tell.ing it "show me more like this", or feeding... (read more)

LESSWRONG
LW

LESSWRONG
LW

david reinstein

Checking public figures on whether they "answered the question" quick analysis from Harris/Trump debate, and a proposal

Unjournal evaluation of "Towards best practices in AGI safety & governance" (2023), quick take

The Unjournal's "Pivotal Questions" project

'Chat with impactful research & evaluations' (Unjournal NotebookLMs)

david reinstein

Can we do useful meta-analysis? Unjournal evaluations of "Meaningfully reducing consumption of meat... is an unsolved problem..."

Unjournal evaluation of "Towards best practices in AGI safety & governance" (2023), quick take

GPT-5 is out

The Unjournal's "Pivotal Questions" project

'Chat with impactful research & evaluations' (Unjournal NotebookLMs)

Calibration training for 'percentile rankings'?

Checking public figures on whether they "answered the question" quick analysis from Harris/Trump debate, and a proposal

david reinstein

Checking public figures on whether they "answered the question" quick analysis from Harris/Trump debate, and a proposal

Unjournal evaluation of "Towards best practices in AGI safety & governance" (2023), quick take

The Unjournal's "Pivotal Questions" project

'Chat with impactful research & evaluations' (Unjournal NotebookLMs)

david reinstein

Can we do useful meta-analysis? Unjournal evaluations of "Meaningfully reducing consumption of meat... is an unsolved problem..."

Unjournal evaluation of "Towards best practices in AGI safety & governance" (2023), quick take

GPT-5 is out

The Unjournal's "Pivotal Questions" project

'Chat with impactful research & evaluations' (Unjournal NotebookLMs)

Calibration training for 'percentile rankings'?

Checking public figures on whether they "answered the question" quick analysis from Harris/Trump debate, and a proposal

GPTPro

Representative quotes from the evaluations

Sampling bias/selection issues

Question selection/design, need for sharper questions

Representative quotes from the evaluations

Sampling bias/selection issues

The Unjournal: background, progress, push for communications and impact

Notebook/chatbot exploration (esp. NotebookLM)

Does this tool/site exist? What's the best/easiest way to build it?