Leon Lang's Shortform

Leon Lang

This is a special post for quick takes by Leon Lang. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I remember that at the end of 2024 there were many reports of strongly diminishing returns in the development (and specifically pretraining) of foundation models, right around the time when reasoning models were starting to emerge. I also remember that many people on Lesswrong thought AI is developing more slowly than they had previously expected.

How are people feeling about this now? My impression is there was no overall slowdown, but I am curious in other people's takes.

Something going slower than I expected is voice, and multimodality in general, though it's hard to say whether this is due to a research roadblock or simply due to the companies' focus on reasoning, coding, and agentic text-based workflows.

3cubefox8mo

In pretraining. I think it's pretty clear that the change from GPT-4o to GPT-5 Instant was rather incremental. Like the change from GPT-4 to GPT-4o.

6Mateusz Bagiński8mo

That sounds very plausible to me, but how would you evaluate it without access to the base model or the instance of the model before it was trained to reason?

GPT-5 Instant doesn't do dedicated reasoning. It is probably still able to reason sometimes in the actual reply block (it did so in the past in the seahorse emoji case), so there seems to be some degree of RLVR involved, but even with that advantage, GPT-5 Instant was not a big improvement over GPT-4o.