Kaj's shortform feed

Kaj_Sotala

This is a special post for quick takes by Kaj_Sotala. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Similar to other people's shortform feeds, short stuff that people on LW might be interested in, but which doesn't feel like it's worth a separate post. (Will probably be mostly cross-posted from my Facebook wall.)

This is a special post for quick takes by Kaj_Sotala. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I feel a bit crazy in that the recent METR report is explicitly saying that the measured progress is on well-specified tasks and that they cannot tell how much of this transfers to ill-specified tasks; and now half the blogs in AI safety sphere seem to be taking this as "ASI is imminent", dropping the caveat. In METR's words:

Acceleration might be limited to well-specified tasks. Much of the evidence we looked at measures performance on well-specified tasks that are easy to grade automatically or with a language model. If recent models have been growing more quickly on “clean” tasks than on “messy” tasks (relative to historical relative growth rates), then we may be overstating the acceleration in real-world impact. For instance, recent progress may be overfitting to the types of tasks which are included in AECI.

If we assume that humans are capable of solving ill-specified tasks, then at least by a "better than humans at any cognitive task" definition of ASI, ASI would need to be capable of solving ill-specified tasks. And "progress on well-specified tasks is accelerating" does not automatically translate to "soon we will understand how to solve ill-specified tasks".

I do think that it makes sense to assume that progress on well-specified tasks increases the odds of making progress on the rest of them as well and that we can't necessarily assume that the acceleration stays restricted to well-specified tasks, but that isn't the same as "we can put this trend on a graph which tells us that ASI will be here in a year".

Earlier related comment.

I enjoy your analyses, if it helps. What we want is to be right, and act on it. The Litany of Tarski still applies.

23paulfchristiano1h

A few months ago I gave a ~10% probability of TEDAI within 12 months; my intuition is now closer to ~30% if we go full speed ahead. Some of that was from the passage of time, but it's also about a ~2x update.[1] For me the math and CS research progress is a bigger update than the METR/Anthropic reporting. I think we saw maybe a decade of progress on the hardest open problems in math and theoretical computer science with no sign of relevant capabilities slowing down. The ability to outperform large communities of human researchers at the task of designing and improving algorithms is a core part of an intelligence explosion, so I think that's a really important indicator.[2] I also think it's useful to track accelerated progress on well-defined metrics (like pretraining or RL performance). Regardless of whether that progress transfers, it indicates that underlying technical developments are accelerating. And beyond that, I think "we haven't seen the pace of progress double yet" is a good argument that the singularity is likely at least 6-12 months away (at least for a trend-extrapolator like me). So once it's happened it's a lot harder to be confident that you won't see something crazy happen in the near term, which naturally lead to a vibes shift even if it still leaves significant probability on things take a while. So maybe my main disagreement with you is that it looks fairly likely that a significant period of self-sustaining acceleration in AI R&D (which is suggested both by the massive math progress and early signs of rapid adoption+acceleration from AI labs) would likely lead to ASI. This is a messy discussion I can't do justice to, but here are some things I believe: * If you max out performance on the well-defined tasks then you will also have superhuman learning algorithms (which are necessary to perform well on broad distributions of virtual tasks). Those algorithms will quickly become superhuman at real-world tasks from exposure to real-world data. E

-1Afterimage3h

I don't think this "better than humans at any cognitive task" definition of ASI is useful at all. I think most people now use the definition that is - in practice - somewhere between "could kill a big chunk of humans if improperly aligned" and "could kill all humans if improperly aligned"

Kaj's shortform feed

6

Ω 2

Kaj's shortform feed

6

Ω 2