The Metaculus/Manifold questions for AGI aren't very helpful! The most popular questions focus on passing a Turing test, but this is likely to be a lagging indicator. As long as AI capabilities remain "spiky", then an experienced interlocutor can focus in on whatever areas AI is known to be weak at. Passing an adversarial Turing test with an expert judge thus requires an AI which is at human level even in its weakest domains, at which point its other domains will probably be wildly superhuman and it will be closer to ASI than AGI.

The current focus of the frontier labs is on automating coding and R&D, and so the most important questions in my mind are "When can AI R&D be automated?" or "What effect will AI automation of R&D have on capabilities progress?" There are a couple of questions that are close to this in spirit, but the details are very odd and not really very connected to important capabilities, which means that the prediction is not very informative.

Is there a better question, or better resolution criteria that would actually elicit interesting answers?

Making an account because I had also been wondering this! The impression I got is that they seem quite loosely committed to that prediction; in Machines of Loving Grace, Dario says that powerful AI "could arrive as early as 2026" but could also take much longer, which seems a much more defensible position. Later, this seemed to firm up to "most likely around late 2026 or early 2027", but that claim stopped appearing in their posts for a while, and the post from Jack Clark that you linked is the only recent reference to it that I'm aware of.

Given how surprising the claim is, how no-one else can figure out their reasoning despite looking at the same data sources that they reference, and how sporadically they talk about it despite its apparent importance, my guess is that it's just not really a strongly held prediction. Instead, it's more something like a slightly stale "company line" that hasn't been officially rescinded yet; something which is not openly contradicted, but neither is it mentioned unless someone asks about it directly. I expect them to either update the prediction or quietly drop it and stop talking about it, and would be surprised if they strongly recommit to that exact timeline or share a more comprehensive justification for it.