ARC-AGI-2 human baseline surpassed

Why is the human baseline so low? This is more tentative, but I'm thinking in terms of two basic possibilities.

One is that a non-trivial proportion of these tasks may be ill-posed problems with multiple legitimate answers. The reason only 2 out of 9 humans found the intended solution could be that it's a difficult well-posed problem, or it could be that there are 2-3 equally valid solutions and landing on the intended "correct" answer requires luck. A perfectly capable solver would still only average 50% or even 33% on such tasks. In this case, around 50% would not only be the human baseline but would also mean saturation of the benchmark.

Or maybe it is more likely that the human baseline is so low due to tricky tasks that a competent human doesn't automatically "get" with enough thought and needs to get lucky in terms of having some idiosyncratic experience or random idea for getting the solution. The fact that a non-trivial proportion of their tasks (over 300, according to Figure 5 in the technical paper) were not even solved by two participants suggests that this is the case for a decent proportion of the tasks they designed. Note that some tasks which only one ninth of potential participants would solve on average will happen to actually be solved by 2 participants by chance--leading to some such too-hard tasks being included in ARC-AGI-2. (By contrast, if they had designed a set of tasks and found that all of them could be solved by at least 2 out of 9 humans, that would be more reassuring that their task design process reliably produces human-solveable tasks, albeit still tasks that less than half of humans can solve.) To the extent this is the reason for the low human baseline, AI systems may be able to substantially outdo typical human performance and approach 100% despite a human baseline around 50%.

^{^}

"The original private evaluation tasks were originally tested by two people who scored 97% and 98%, and, together, solved all 100%." (ARC Prize 2024: Technical Report) No further information is given about these two people, but we can assume they were not randomly selected or otherwise representative.

(An independent study recruited 1729 crowd-workers over the internet and found they averaged 76% on the training set and 64% on the public evaluation set. The ARC Prize team highlights that "99% of public evaluation tasks were solved by at least one worker, with 10 workers assigned to each task.")

^{^}

"To ensure each task was solvable by humans, we made a minimum success bar of: 'two people in two attempts or less.' On average, each task was attempted by about 9-10 participants." https://arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025

^{^}

"Many ARC-AGI-1 tasks could often be solved almost instantaneously by human test-takers without requiring significant cognitive effort. In contrast, all tasks in ARC-AGI-2 require some amount of deliberate thinking — for instance, the average time for task completion among human test takers in our sample was 2.7 minutes." ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems

^{^}

ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems

^{^}

While the technical paper states that "final ARC-AGI-2 test pairs were solved, on average, by 75% of people who attempted them," this average is apparently dominated by the (large) public training set, which is easier on average. According to the paper's Figure 5, the public training tasks were solved by an average ~6.4 participants (out of an average 9-10). But something is off because Figure 5 appears to represent only around 350 "Public Train" tasks, whereas the announcement post says there are 1000.

LESSWRONG
LW

LESSWRONG
LW

13

ARC-AGI-2 human baseline surpassed

13

13