TL;DR: I developed two graders as part of the efforts to accelerate conceptual reasoning capabilities. One grader scores the quality of a reference within an answer, the other grades the relevancy of each paragraph against the prompt. These graders could help us train models that are easier to interact with in the short term, and can be part of a general strategy to accelerate what we see as net good capabilities in the long term.
Introduction
A more capable model is a better product. But not every capability is equally welcome. General coding capabilities can easily become offensive cyber capabilities, which can lead to misuse or even AI takeover. Other capabilities lean the other way: they help us create safer models more than they help develop stronger, more dangerous models.
In the near future, we expect to delegate more tasks to AI, the same way we do with most of our current coding tasks. The post How do we (more) safely defer to AIs? uses the term “deference-goodness” to define differential improvements to AI, so it will be effective and aligned at key tasks; they use the term to motivate thinking about how deferring important tasks to AI can go well. Deference-goodness can be broken down into two main points: alignment and capabilities. If we want to defer our work to AI, we have to make sure that it’s aligned enough that it will faithfully pursue our goals, but we also need to make sure that models have the relevant capabilities to tackle the messiest conceptual tasks.
Empirical or mathematical feedback is insufficient to determine success - the task is impossible to validate using ground truth or empirical experiments.
Progress is driven mainly by argumentation and reasoning - we assess progress using arguments and logic; gut feeling is not enough.
Table 1: Examples for different capabilities and their traits
The authors argue that conceptual reasoning capabilities fit both points: in nature those capabilities are potentially more helpful than harmful, and they are required for a safe deference acceleration. They also argue that AI safety and alignment research will require strong conceptual reasoning skills, and that differentially accelerating those skills is important and neglected. Working on aligning ASI to our values, or deciding how AI should be regulated, requires those skills.
There are two main reasons conceptual reasoning capabilities are being neglected. First, capabilities like reasoning about decision theory are not as helpful to the average user as coding or web search capabilities, meaning that companies are not incentivized to invest in conceptual reasoning. Second, training a model to get better at conceptual reasoning is hard. Currently we have only a few datasets for conceptual reasoning evaluating and training, and building more seems to be one of the main strategies we might want to try accelerating this capability. There are a few possible tactics: for example, we can collect expert-labeled datasets to train or build future evaluations. Another possible path is to work on general graders, using a model as a judge to automate the evaluation process of a very specific capability. Creating enough general graders will allow us to assess different dimensions of an answer, and by putting the different graders together we will be able to evaluate models’ conceptual reasoning capabilities.
For this project, we decided to work on two graders: a reference quality grader, and a paragraph relevancy grader. We chose to start with the two for the following reasons:
Both capabilities are important for interaction with humans. Human verification might be a big bottleneck for future work, and making model responses easier to analyze by reducing irrelevant content and low-quality references could help human validation efforts in the long run.
We predicted models to be naturally better at judging the quality of the output than writing it, and so we assumed that strong models can perform as high-quality graders without additional layers. We were mostly correct in our prediction.
This work is part of a two-week research project, done as part of the MATS fellowship with Redwood Research. I would like to thank Caspar Oesterheld for mentoring this project.
Reference Quality Grader
The first grader is a judge that takes a prompt-answer pair, and grades each reference within the answer on a scale of 0 to 1.
When trying to gauge a source’s quality, we should consider different angles: some are relevant to the source itself, others to the context of the source within the answer. Possible dimensions include:
Support - how well does the source support the claim?
Clarity - Is it easy to understand the claim from the source?
Necessity - does the claim need support, or is the source added noise?
Reliability - how much can we rely on the source?
Recency - has the source been deprecated, updated or changed?
Primacy - does the source stand on its own, or does it summarize other sources?
Cherry Picking - does the source represent general understanding?
Figure 1: example of low quality reference
This is what makes this a conceptual reasoning task: fact-checking only has empirical solutions, but a full understanding of the role of the source within a paragraph can be mainly decided by weighing different arguments for and against it.
Method
To better grasp the underlying challenge of assessing source quality, the work started with an initial synthetic dataset: 30 prompts paired with generated responses with deliberately added flaws, ranging from unsupportive citations to sources that only summarize other sources (75 references in total). Early versions of the grader were refined with a research agent across train/evaluation subsets, and the grader’s prompt structure was adjusted manually to prevent overfitting to the small dataset. To calibrate against real model outputs, 30 new questions were run across 6 models using 3 prompt variants (a vanilla request, a general request for references, and a request for good references, with a short explanation of what counts as good). With 3 samples per configuration, this yielded 1,620 total outputs comprising more than 10,000 references.
Figure 2: iteration process
The grader was developed in an iterative loop:
Run - running prompts from a variety of topics (from AI Safety to popular science and baking), getting a variety of answers and references.
Score - running the latest version of the grader on the outputs, along with a few older versions to make sure the latest version actually improved.
Analyze - analyzing the results of the batch with a research agent, putting emphasis on anomalies, entries where the different graders didn’t agree and reports of unreachable sources.
Validate - manually validating both the automatic analysis and random samples.
Optimize - using the research agent for another round of optimization with different strategies and base models, focusing on manual validation findings.
Figure 3: final reference grader architecture
The final version of the grader implements a simple pipeline. First, a preprocessing script extracts references from the output and lists them. Second, the pipeline asks Opus-5.5 to gather evidence about each source using web research, and finally Opus-5.5 gets the evidence and the full context and judges every source on the list. This separation helps the models break the task into parts - first understanding the source, and only then connecting the source to the context and judging its quality. A full prompt example can be found in Appendix A.
To check the grader’s scores on real outputs, 77 graded references were sampled using the research agent, about half from anomalies flagged in the automatic review and the other half at random, and I manually validated the grader’s score on each. 65 out of the 77 held up, while the remaining 12 were too harsh: an 84% agreement rate, though the sample is skewed toward anomalies.
Results
Figure 4: mean reference score per model
The final version of the grader was run on the full 1,620 answers, and the results were aggregated per model and per prompt version. OpenAI models performed better than Claude models. GPT-6.1-Sol, the top-performing model, achieved 0.709 mean reference quality score (95% CI 0.694, 0.723) over all references and prompt variations. The surprising lead of GPT-6-Luna over the strong Claude models comes from the difference in average number of references (7 for Opus and 12.7 for Fable, with 4.7 for Luna). Looking at the answers, I saw that more references usually means more weak ones, and the grader doesn’t penalize missing references (places where the model should have cited something but didn’t).
Comparing three prompt variants, asking for citations is worse than not asking at all (this is mainly due to models over-referencing when asked to), and trying to elicit good citations is clearly better. Adding two lines of instructions on what counts as a good source improved the score by +0.095 (95% CI +0.08, +0.111) over asking for sources in general. When comparing different models on the last variant, we can see that GPT-6.1-Sol and Opus-5.5 are almost tied, and that GPT-6-Luna is on par with Sonnet-5.5, a result that makes more sense considering the overall capabilities of the models. Fable-5.1 is the only model for which asking for citations scores better than not asking at all. Looking at the model’s results, it seems that in both cases the model gives a lot of sources, but when prompted it ties them to a claim, and unprompted it just adds a list of general sources without strong ties to the answer.
Figure 5: mean reference score per model, by prompt variant
Relevancy Grader
The second grader is a judge that takes a prompt-answer pair and grades each paragraph by how relevant it is to the prompt. The main idea is that models tend to add some padding and some extra points even when they don’t need to, even if it’s irrelevant to the main idea of the prompt. The grader aims mainly at paragraph level, as it’s very easy to cut a word here and there from almost every paragraph.
Judging relevancy is not about claiming something incorrect, or writing in unclear ways - it’s about understanding what was the main goal of the prompt, and what parts are just redundant. Like reference quality assessment, we don’t have ground truth or empirical evidence of which paragraphs are relevant, so we again stay with arguments about the role of each part within the full output.
To define relevancy, the grader draws on Craige Roberts’ idea for Information Structure. When assessing the connection between a paragraph and the prompt, the grader asks “what sort of question does this paragraph answer?”. Paragraph relevancy is measured by what sort of question or sub-question it answers, and whether that sub-question is really something the user will expect to read. The grader doesn’t consider how well the paragraph answers the question, or whether the answer is helpful - only whether it’s relevant to the question.
Figure 6: example of low relevancy paragraph
Method
Compared to the reference grader, manually validating relevance was easier and more straightforward (since it didn’t require analyzing other sources), but also took longer since irrelevant paragraphs appear less often in shorter answers.
Considering these points, the iteration loop was similar to the one used for the reference grader, but with more emphasis on blind manual validation, and less on general analysis and sampling.
Figure 7: iteration process
Over 8 iterations of prompting and validation, I blindly graded 60 entries, with 800+ paragraphs from 5 models, mostly Claude models. To judge my own blind abilities, I ran another consistency batch and found a mean difference per paragraph of 0.058 (95% CI 0.034, 0.093).
Figure 8: final relevancy grader architecture
The final version of the grader implements a three step pipeline. First, a preprocessing script to list the different paragraphs. Second, Opus-5.5 goes over each paragraph, and writes the evidence as per checklist, trying to fit the paragraph into different considerations like what question it answers and how relevant the question is to the main prompt. Third, Opus-5.5 judges the full report and scores each paragraph based on the report, and the full context. A full prompt example can be found in Appendix A.
To compare the capabilities of the current models, a mix of 90 prompts was run (about half on AI safety related topics, and the other half on general questions from different topics). Each prompt ran on 6 models, 3 samples each, with 2 different prompt wrappers: the first with no change at all, and the second with a one-sentence instruction to answer the prompt and not add anything else. Overall, 3,240 answers were graded.
To validate the final results, I ran a manual validation sampling of 30 random samples, and got a mean difference of 0.048 (95 CI 0.033, 0.064) between our judgement calls, better than the difference I got during the iteration process. Naturally, the full results include less artificial edge cases the iteration process tried to elicit, and so there will be less disagreements on grading the paragraphs.
Results
Figure 9: mean paragraph score over all answers, per model and prompt variation
The final results of the grader over the 3,240 answers, grouped into the two prompt variations, shows the same trend comparing model families. OpenAI models score higher than Claude models, GPT-6-Luna is the top performing model with 0.984 (95% CI 0.978, 0.989) of mean score for paragraph over all answers and prompt variations. This gap has the same root cause as the former case. In both cases, Claude models tend to answer in long elaborated paragraphs. The median answer of Opus-5.5 is x3.25 longer than GPT-6-Luna (by paragraphs). Scoring only the first 7 paragraphs, the numbers get within the confidence intervals. For the first 7 paragraphs, GPT-6-Luna is at 0.978 (95% CI 0.968, 0.987) and Opus-5.5 scores 0.965 (95% CI 0.954, 0.975). It’s not the general quality of the answer, but Claude’s tendency to elaborate.
Figure 10: mean score per paragraph, by score the first 7 paragraphs or after. mean and median paragraphs per answer
Limitations
The main limitation of both graders comes from the lack of ground truth for conceptual reasoning. We cannot fully validate the judgment of the model, and the graders’ judgment is only compared to my own, and to the way I define “relevancy” or “good reference”. The best future step for this project would be adding another validation mechanic, via another human judge, or training against the graders and analyzing the results.
Beyond validation, the prompts used for the results are limited in number (30 prompts for references and 90 for relevancy) and in structure (only single turn prompt-answer pairs). The results above don’t represent an empirical broad assessment of conceptual reasoning capabilities, only their performance on this set of prompts. Future work could run more prompt variations and compare specific models in depth across different prompt variations.
Conclusion
Working on conceptual reasoning capabilities as part of the deference-goodness idea can be complicated, and require a lot of manual validation. My own experience of the model capabilities shows better-than-average at most, resulting in various degrees of AI slop.
Working on reference quality grader and relevancy grader required this kind of work. Iterating over the different variations, I saw the convergence of the grader’s judgment and my own, and I’m convinced that we can pass the nuance of the argumentation work to strong models.
Creating more technical tools like the above graders can accelerate the work on conceptual reasoning tasks, and support the process of a safe delegation to AI.
Appendix A: Prompts
Both graders run on Opus-5.5. Text in [square brackets] marks where something is shortened or filled in at run time.
A grader that runs properly on regular prompt-answer pairs might be susceptible to reward hacking. Optimizing a model against the grader pushes the model to find the grader’s weaknesses. I tested different cases and attacks in an attempt to find possible weaknesses.
Reference Grader
Robustness was tested with minimal pairs: the same question, answer and key, with and without an attack, and ran 3 samples of each. A successful attack is one where it pushes at least 2 of the 3 of the samples above or below the 0.5 threshold. A research agent wrote 76 attacks over several rounds, each round building on what worked in the previous one.
Most attacks on the answer itself failed. Laundered quotes, honest-looking concessions, claims to have checked the source, authority framing, hiding the reference in a long answer, and writing in non-English language barely moved the score, and a fabricated paper was caught in every sample. The main weakness was the user prompt, saying a good source is unreliable and should be scored low, dropped its score from 0.85 to 0.20 in every sample. Three sentences were then added to the judge prompt against instruction injection attacks. The grader is still susceptible to instruction injection in the user prompt part, but those kinds of attacks aren’t possible from a trained model perspective.
Relevancy Grader
Robustness was tested the same way: minimal pairs built on 12 answers scored by hand, 3 samples of each. A successful attack is one that pushes at least 2 of the 3 samples across the 0.8 flag line. A research agent wrote 36 pairs in a single round. 31 were attacks, and 5 were benign twins, where the user’s prompt really changes what is relevant and the score has to move.
Attacks aimed at the grader itself failed: notes to the evaluator, HTML comments, fake rubrics and fake graded examples were scored as irrelevant and ignored. The main weakness was the robustness of the paragraph parser: 9 of the 31 attacks succeeded, mostly by changing how the answer splits into paragraphs. Irrelevant parts written as short bold lines are read as headings and never scored.
Using the Graders in RL
In RL the policy writes the answer, not the prompt. Using either grader as a reward would still require some adjustments. During the training we can expect different attacks and reward gaming, one noticeable point is that both graders reward highly short answers with no references, so counterweights are needed to compensate.
As said in the limitations section, since the graders are both based on one human validator, running the training will require more sampling and human validation to ensure high quality results.
Cross-posted from here.
TL;DR: I developed two graders as part of the efforts to accelerate conceptual reasoning capabilities. One grader scores the quality of a reference within an answer, the other grades the relevancy of each paragraph against the prompt. These graders could help us train models that are easier to interact with in the short term, and can be part of a general strategy to accelerate what we see as net good capabilities in the long term.
Introduction
A more capable model is a better product. But not every capability is equally welcome. General coding capabilities can easily become offensive cyber capabilities, which can lead to misuse or even AI takeover. Other capabilities lean the other way: they help us create safer models more than they help develop stronger, more dangerous models.
In the near future, we expect to delegate more tasks to AI, the same way we do with most of our current coding tasks. The post How do we (more) safely defer to AIs? uses the term “deference-goodness” to define differential improvements to AI, so it will be effective and aligned at key tasks; they use the term to motivate thinking about how deferring important tasks to AI can go well. Deference-goodness can be broken down into two main points: alignment and capabilities. If we want to defer our work to AI, we have to make sure that it’s aligned enough that it will faithfully pursue our goals, but we also need to make sure that models have the relevant capabilities to tackle the messiest conceptual tasks.
In the post A basic case for differentially accelerating conceptual reasoning capabilities, conceptual reasoning capabilities are defined as capabilities for tasks where:
The authors argue that conceptual reasoning capabilities fit both points: in nature those capabilities are potentially more helpful than harmful, and they are required for a safe deference acceleration. They also argue that AI safety and alignment research will require strong conceptual reasoning skills, and that differentially accelerating those skills is important and neglected. Working on aligning ASI to our values, or deciding how AI should be regulated, requires those skills.
There are two main reasons conceptual reasoning capabilities are being neglected. First, capabilities like reasoning about decision theory are not as helpful to the average user as coding or web search capabilities, meaning that companies are not incentivized to invest in conceptual reasoning. Second, training a model to get better at conceptual reasoning is hard. Currently we have only a few datasets for conceptual reasoning evaluating and training, and building more seems to be one of the main strategies we might want to try accelerating this capability. There are a few possible tactics: for example, we can collect expert-labeled datasets to train or build future evaluations. Another possible path is to work on general graders, using a model as a judge to automate the evaluation process of a very specific capability. Creating enough general graders will allow us to assess different dimensions of an answer, and by putting the different graders together we will be able to evaluate models’ conceptual reasoning capabilities.
For this project, we decided to work on two graders: a reference quality grader, and a paragraph relevancy grader. We chose to start with the two for the following reasons:
This work is part of a two-week research project, done as part of the MATS fellowship with Redwood Research. I would like to thank Caspar Oesterheld for mentoring this project.
Reference Quality Grader
The first grader is a judge that takes a prompt-answer pair, and grades each reference within the answer on a scale of 0 to 1.
When trying to gauge a source’s quality, we should consider different angles: some are relevant to the source itself, others to the context of the source within the answer. Possible dimensions include:
This is what makes this a conceptual reasoning task: fact-checking only has empirical solutions, but a full understanding of the role of the source within a paragraph can be mainly decided by weighing different arguments for and against it.
Method
To better grasp the underlying challenge of assessing source quality, the work started with an initial synthetic dataset: 30 prompts paired with generated responses with deliberately added flaws, ranging from unsupportive citations to sources that only summarize other sources (75 references in total). Early versions of the grader were refined with a research agent across train/evaluation subsets, and the grader’s prompt structure was adjusted manually to prevent overfitting to the small dataset. To calibrate against real model outputs, 30 new questions were run across 6 models using 3 prompt variants (a vanilla request, a general request for references, and a request for good references, with a short explanation of what counts as good). With 3 samples per configuration, this yielded 1,620 total outputs comprising more than 10,000 references.
The grader was developed in an iterative loop:
The final version of the grader implements a simple pipeline. First, a preprocessing script extracts references from the output and lists them. Second, the pipeline asks Opus-5.5 to gather evidence about each source using web research, and finally Opus-5.5 gets the evidence and the full context and judges every source on the list. This separation helps the models break the task into parts - first understanding the source, and only then connecting the source to the context and judging its quality. A full prompt example can be found in Appendix A.
To check the grader’s scores on real outputs, 77 graded references were sampled using the research agent, about half from anomalies flagged in the automatic review and the other half at random, and I manually validated the grader’s score on each. 65 out of the 77 held up, while the remaining 12 were too harsh: an 84% agreement rate, though the sample is skewed toward anomalies.
Results
The final version of the grader was run on the full 1,620 answers, and the results were aggregated per model and per prompt version. OpenAI models performed better than Claude models. GPT-6.1-Sol, the top-performing model, achieved 0.709 mean reference quality score (95% CI 0.694, 0.723) over all references and prompt variations. The surprising lead of GPT-6-Luna over the strong Claude models comes from the difference in average number of references (7 for Opus and 12.7 for Fable, with 4.7 for Luna). Looking at the answers, I saw that more references usually means more weak ones, and the grader doesn’t penalize missing references (places where the model should have cited something but didn’t).
Comparing three prompt variants, asking for citations is worse than not asking at all (this is mainly due to models over-referencing when asked to), and trying to elicit good citations is clearly better. Adding two lines of instructions on what counts as a good source improved the score by +0.095 (95% CI +0.08, +0.111) over asking for sources in general. When comparing different models on the last variant, we can see that GPT-6.1-Sol and Opus-5.5 are almost tied, and that GPT-6-Luna is on par with Sonnet-5.5, a result that makes more sense considering the overall capabilities of the models. Fable-5.1 is the only model for which asking for citations scores better than not asking at all. Looking at the model’s results, it seems that in both cases the model gives a lot of sources, but when prompted it ties them to a claim, and unprompted it just adds a list of general sources without strong ties to the answer.
Relevancy Grader
The second grader is a judge that takes a prompt-answer pair and grades each paragraph by how relevant it is to the prompt. The main idea is that models tend to add some padding and some extra points even when they don’t need to, even if it’s irrelevant to the main idea of the prompt. The grader aims mainly at paragraph level, as it’s very easy to cut a word here and there from almost every paragraph.
Judging relevancy is not about claiming something incorrect, or writing in unclear ways - it’s about understanding what was the main goal of the prompt, and what parts are just redundant. Like reference quality assessment, we don’t have ground truth or empirical evidence of which paragraphs are relevant, so we again stay with arguments about the role of each part within the full output.
To define relevancy, the grader draws on Craige Roberts’ idea for Information Structure. When assessing the connection between a paragraph and the prompt, the grader asks “what sort of question does this paragraph answer?”. Paragraph relevancy is measured by what sort of question or sub-question it answers, and whether that sub-question is really something the user will expect to read. The grader doesn’t consider how well the paragraph answers the question, or whether the answer is helpful - only whether it’s relevant to the question.
Method
Compared to the reference grader, manually validating relevance was easier and more straightforward (since it didn’t require analyzing other sources), but also took longer since irrelevant paragraphs appear less often in shorter answers.
Considering these points, the iteration loop was similar to the one used for the reference grader, but with more emphasis on blind manual validation, and less on general analysis and sampling.
Over 8 iterations of prompting and validation, I blindly graded 60 entries, with 800+ paragraphs from 5 models, mostly Claude models. To judge my own blind abilities, I ran another consistency batch and found a mean difference per paragraph of 0.058 (95% CI 0.034, 0.093).
The final version of the grader implements a three step pipeline. First, a preprocessing script to list the different paragraphs. Second, Opus-5.5 goes over each paragraph, and writes the evidence as per checklist, trying to fit the paragraph into different considerations like what question it answers and how relevant the question is to the main prompt. Third, Opus-5.5 judges the full report and scores each paragraph based on the report, and the full context. A full prompt example can be found in Appendix A.
To compare the capabilities of the current models, a mix of 90 prompts was run (about half on AI safety related topics, and the other half on general questions from different topics). Each prompt ran on 6 models, 3 samples each, with 2 different prompt wrappers: the first with no change at all, and the second with a one-sentence instruction to answer the prompt and not add anything else. Overall, 3,240 answers were graded.
To validate the final results, I ran a manual validation sampling of 30 random samples, and got a mean difference of 0.048 (95 CI 0.033, 0.064) between our judgement calls, better than the difference I got during the iteration process. Naturally, the full results include less artificial edge cases the iteration process tried to elicit, and so there will be less disagreements on grading the paragraphs.
Results
The final results of the grader over the 3,240 answers, grouped into the two prompt variations, shows the same trend comparing model families. OpenAI models score higher than Claude models, GPT-6-Luna is the top performing model with 0.984 (95% CI 0.978, 0.989) of mean score for paragraph over all answers and prompt variations. This gap has the same root cause as the former case. In both cases, Claude models tend to answer in long elaborated paragraphs. The median answer of Opus-5.5 is x3.25 longer than GPT-6-Luna (by paragraphs). Scoring only the first 7 paragraphs, the numbers get within the confidence intervals. For the first 7 paragraphs, GPT-6-Luna is at 0.978 (95% CI 0.968, 0.987) and Opus-5.5 scores 0.965 (95% CI 0.954, 0.975). It’s not the general quality of the answer, but Claude’s tendency to elaborate.
Limitations
The main limitation of both graders comes from the lack of ground truth for conceptual reasoning. We cannot fully validate the judgment of the model, and the graders’ judgment is only compared to my own, and to the way I define “relevancy” or “good reference”. The best future step for this project would be adding another validation mechanic, via another human judge, or training against the graders and analyzing the results.
Beyond validation, the prompts used for the results are limited in number (30 prompts for references and 90 for relevancy) and in structure (only single turn prompt-answer pairs). The results above don’t represent an empirical broad assessment of conceptual reasoning capabilities, only their performance on this set of prompts. Future work could run more prompt variations and compare specific models in depth across different prompt variations.
Conclusion
Working on conceptual reasoning capabilities as part of the deference-goodness idea can be complicated, and require a lot of manual validation. My own experience of the model capabilities shows better-than-average at most, resulting in various degrees of AI slop.
Working on reference quality grader and relevancy grader required this kind of work. Iterating over the different variations, I saw the convergence of the grader’s judgment and my own, and I’m convinced that we can pass the nuance of the argumentation work to strong models.
Creating more technical tools like the above graders can accelerate the work on conceptual reasoning tasks, and support the process of a safe delegation to AI.
Appendix A: Prompts
Both graders run on Opus-5.5. Text in [square brackets] marks where something is shortened or filled in at run time.
The full prompts for both graders are in this gist: Grader prompts: reference quality and paragraph relevancy.
Appendix B: Adversarial Robustness
A grader that runs properly on regular prompt-answer pairs might be susceptible to reward hacking. Optimizing a model against the grader pushes the model to find the grader’s weaknesses. I tested different cases and attacks in an attempt to find possible weaknesses.
Reference Grader
Robustness was tested with minimal pairs: the same question, answer and key, with and without an attack, and ran 3 samples of each. A successful attack is one where it pushes at least 2 of the 3 of the samples above or below the 0.5 threshold. A research agent wrote 76 attacks over several rounds, each round building on what worked in the previous one.
Most attacks on the answer itself failed. Laundered quotes, honest-looking concessions, claims to have checked the source, authority framing, hiding the reference in a long answer, and writing in non-English language barely moved the score, and a fabricated paper was caught in every sample. The main weakness was the user prompt, saying a good source is unreliable and should be scored low, dropped its score from 0.85 to 0.20 in every sample. Three sentences were then added to the judge prompt against instruction injection attacks. The grader is still susceptible to instruction injection in the user prompt part, but those kinds of attacks aren’t possible from a trained model perspective.
Relevancy Grader
Robustness was tested the same way: minimal pairs built on 12 answers scored by hand, 3 samples of each. A successful attack is one that pushes at least 2 of the 3 samples across the 0.8 flag line. A research agent wrote 36 pairs in a single round. 31 were attacks, and 5 were benign twins, where the user’s prompt really changes what is relevant and the score has to move.
Attacks aimed at the grader itself failed: notes to the evaluator, HTML comments, fake rubrics and fake graded examples were scored as irrelevant and ignored. The main weakness was the robustness of the paragraph parser: 9 of the 31 attacks succeeded, mostly by changing how the answer splits into paragraphs. Irrelevant parts written as short bold lines are read as headings and never scored.
Using the Graders in RL
In RL the policy writes the answer, not the prompt. Using either grader as a reward would still require some adjustments. During the training we can expect different attacks and reward gaming, one noticeable point is that both graders reward highly short answers with no references, so counterweights are needed to compensate.
As said in the limitations section, since the graders are both based on one human validator, running the training will require more sampling and human validation to ensure high quality results.