ARC-AGI-2 human baseline surpassed (updated)

Why is the human baseline so low? This is more tentative, but I'm thinking in terms of two basic possibilities. The reason only 2-5 out of 9 humans found the intended solution could be that it's a well-posed but difficult problem, or it could be that there are 2-3 equally valid solutions and landing on the intended "correct" answer requires pure luck.

Starting with the latter possibility, a non-trivial proportion of these tasks may be ill-posed problems with multiple legitimate answers. Even an ideal solver would still only average 50% (or 33% or lower) on such tasks. If this were the case for all eval tasks, ~50% would not only be the human baseline but would also mean saturation of the benchmark.

Perhaps more likely is that the human baseline is so low primarily due to tricky tasks that a competent human doesn't automatically "get" with enough thought. She needs a sort of "luck" in terms of having some idiosyncratic experience or idea to get to the solution. The fact that a non-trivial proportion of ARC 2 tasks (over 300, according to Figure 5 in the technical paper) were not even solved by two participants suggests that this is the case for a decent proportion of the tasks they designed. Note that some tasks which only one ninth of potential participants (the idealized "population") would solve on average will happen to actually be solved by 2 participants by chance--leading to some such too-hard tasks being included in ARC-AGI-2. (By contrast, if they had designed a set of tasks and found that all of them could be solved by at least 2 out of 9 humans, that would be more reassuring, providing evidence that their task design process reliably produces human-solvable tasks, albeit often still tasks that less than half of humans can solve.) To the extent this is the reason for the low human baseline, AI systems may be able to substantially outdo typical human performance and approach 100% despite a human baseline around 50%.

^{^}

"The original private evaluation tasks were originally tested by two people who scored 97% and 98%, and, together, solved all 100%." (ARC Prize 2024: Technical Report) No further information is given about these two people, but we can assume they were not randomly selected or otherwise representative.

(An independent study recruited 1729 crowd-workers over the internet and found they averaged 76% on the training set and 64% on the public evaluation set. The ARC Prize team highlights that "99% of public evaluation tasks were solved by at least one worker, with 10 workers assigned to each task.")

^{^}

"To ensure each task was solvable by humans, we made a minimum success bar of: 'two people in two attempts or less.' On average, each task was attempted by about 9-10 participants." https://arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025

^{^}

"Many ARC-AGI-1 tasks could often be solved almost instantaneously by human test-takers without requiring significant cognitive effort. In contrast, all tasks in ARC-AGI-2 require some amount of deliberate thinking — for instance, the average time for task completion among human test takers in our sample was 2.7 minutes." ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems

^{^}

ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems

^{^}

While the technical paper states that "final ARC-AGI-2 test pairs were solved, on average, by 75% of people who attempted them," this average is apparently dominated by the (large) public training set, which is easier on average. According to the paper's Figure 5, the public training tasks were solved by an average ~6.4 participants (out of an average 9-10). But something is off because Figure 5 appears to represent only around 350 "Public Train" tasks, whereas the announcement post says there are 1000. (Elsewhere, the ARC 2 web page lists average human performance as 60%, but it is not clear which set of tasks that pertains to.)

^{^}

Remarkably, AI has surpassed humans in cost-efficiency as well as pure capability. 'GPT-5.2 (X-High)' in particular achieves 52.9% for just $1.90 per task, whereas human participants were paid $5 per task (plus "a $115-150 show-up fee").

^{^}

https://arcprize.org/arc-agi/2/ Elsewhere, they appear to be described as "STEM Professionals." ARC Prize states here and in the announcement post their intention to publish the full human performance data ("This first-party data provides a solid benchmark for human performance and will be published alongside the ARC-AGI-2 paper.") but they have apparently not followed through on open-sourcing it. The human testing data is available at https://huggingface.co/datasets/arcprize/arc_agi_2_human_testing

LESSWRONG
LW

LESSWRONG
LW

21

ARC-AGI-2 human baseline surpassed (updated)

21

21