A very sober trip report.

Thanks!

A talented kid from a math school, I’d say (about a typical virtual character I was interacting with during those conversations). Not bad for the time being…

I don't know. I started (my experience talking with GPT-4 and such) with asking it to analyze a 200 lines of non-standard code with comments stripped out. It correctly figured out that I was using nested dictionaries to represent vector-like objects, and that that was an implementation of a non-standard unusually flexible (but slow) neural machine.

This was obviously the case of "true understanding" (and it was quite difficult to reproduce, as the models evolved the ability to analyze this code well was lost, then eventually regained in better models; those better models eventually figured even more non-trivial things about that non-standard implementation, e.g. at some point newer models started to notice on their own that that particular neural machine was inherently self-modifying; anyway, very obvious evolution from inept pattern matching to good understanding, with some setbacks during the evolution of models, but eventually with good progress towards better and better performance).

Then I asked it to creatively modify and creatively remix some Shadertoy shaders, and it did a very good job (even more so if one considers that that model was visually blind and was unable to see the animations produced by its shaders). Nothing too difficult, but things like taking a function from one of the shaders and adding a call to this function from another shader with impressive visual effects... Again, with all the simplicity, it was more than would have occurred to me, if I were trying to do this manually...

But when I tried to manually iterate these steps to obtain "evolution of interesting shaders", I got a rapid saturation, not an unlimited interesting evolution...

So not bad at all (I occasionally do rather creative things, but it is always an effort, so on the occasions when I am unable to successfully do this kind of effort, I start to feel that the model might be more creative than "me in my usual mode" (although, I don't know if these models are already competitive with "me in my peak mode")).

When they first introduced Code Interpreter, I asked it to solve a math competition problem, and it did a nice job, and then I asked it what would I do, if I want to do it only using a pen and a paper with limited precision (it was a problem with a huge answer), and it told me to take logarithms, and demonstrated how to do that with logarithms.

That immutable tree processing I mentioned was good in this sense, very elegant, taught me some Python tricks I have not known.

Then when reasoning models were first introduced I asked for a linear algebra problem (which I could not solve myself, but people doing math competitions could), and weaker models could not do it, but o1-preview could one-shot it.

(All these are conversations I am publishing on github, so if one wants to take a closer look, one can.)

Anyway, my impression is that it's not just training data, it's more than that. This is not doable without reasonable understanding, without good intuition.

At the same time, they can be lazy, can be sloppy and make embarrassing mistakes (which sometimes don't prevent them from proceeding to the right result). But it's the reliability which is a problem, not creative capability which seems to be quite robust (at least, on the "medium creativity" setting).

We're supposed to be talking about novel ideas about minds / intelligence, and then implementing them in a way that really tests the hypothesis.

No, just about how to actually make non-saturating recursive self-improvement ("intelligence explosion").

Well, with the added constraint of not killing everyone or almost everyone, and not making almost everyone miserable either...

(Speaking of which, I noticed recently that some of people's attempts at recursive self-improvement now take longer to saturate than before. And, in fact, they are taking long enough that people are sometimes publishing before pushing them to saturation, so we don't even know what would happen if they were to simply continue pushing a bit harder.)

Now, implementing those things to test those ideas can actually be quite unsafe (that's basically "mini-foom" experiments, and people are not talking enough about safety of those). So before pushing harder in this direction, it would be better to do some preliminary work to reduce risks of such experiments...

Go ahead and make an argument then.

Yes, LLMs mostly have reliability/steerability problem. I am seeing plenty of "strokes of genius" in LLMs, so the potential is there. They are not "dumb", they have good creativity (in my experience).

We just can't get them to reliably compose, verify, backtrack, and so on to produce the overall quality work. Their "fuzzy error correction" still works less well than human "fuzzy error correction", at least on some of the relevant scales. So they eventually accumulate too many errors on the long horizon tasks and they don't self-correct enough on the long horizon tasks.

This sounds to me more like a "character upbringing problem" than a purely technical problem...

That's especially obvious when one reads reasoning traces of reasoning models...

What I see there sounds to me as if their "orientation" is still wrong (those models seem to be thinking about how to satisfy their user or their maker, and not about how to "do the right thing", whereas a good human solving a math problem ignores the aspect of satisfying their teacher or their parent, and just tries to do "an objectively good job", and that's where LLMs are still falling short)...

If they are novel ideas, it's much much harder for current LLMs to rightly implement them.

That's not my experience.

Even with the original GPT-4, asking for some rather non-standard things which I was unable to find online (e.g. immutable transformations of nested dictionaries in Python, so that I could compute JAX gradients with respect to variables located in the nodes of the trees in question) resulted in nice implementations (I had no idea how to even start doing that in Python, I only knew that trying to literally port my Clojure code doing the same thing would be a really bad idea).

Not one-shot, of course, but conversations with several turns have been quite productive in this sense.

And that's where we have decent progress lately and expect further rapid progress soon (this year, I think, so I would expect a much better one-shot experience in this sense soon).

No I think it's the first one.

It depends on whether one expects "the solution" to be purist or hybrid.

If the expectation is "purist", then the barrier is high (not insurmountable, I think, but I am a bit reluctant to discuss the technical details in public).

If the expectation is that the solution will be "hybrid" (e.g. powerful LLMs can be used as components, and one can have read-write access to the internals of those LLMs), then it should be very doable (we have not even started to really "unhobble" those LLMs, reasoning models non-withstanding). Quite a bit of non-triviality still remains even under a "hybrid" approach, but the remaining gap is much lower than for "purist" attempts. It's obvious that LLMs are more than "half-way there", even if it might be true that "something fundamental is still missing".

I like talking to LLMs and discussing things (including various ideas some of which might be related to advanced AI or to AI existential safety). And that helps somewhat.

But if I can ask them to implement my ideas end-to-end in prototype code and to conduct computational experiments for me end-to-end, my personal velocity would grow an order of magnitude, if not more...

And if I can ask them to explore variations and recombinations of my ideas, then...

It's not that people don't have good AGI ideas, it's more that the road from a promising idea to a viable implementation is difficult, and a lot of good ideas remain "hanging in the air" (that is, insufficiently tested for us to know if those promising ideas are actually good).

I think OpenAI, DeepMind and other “big labs” work on many things. They release some of them.

But also many other people work on many non-standard things, e.g. Sakana AI, Liquid AI and others, and with non-standard approaches compute might be less crucial, and open reasoning models like the new R1 are only a few months behind the leading released closed source models, so those smaller orgs can have their research accelerated as well even if they decide they don’t want to use OpenAI/DeepMind/Anthropic models.

It’s not AGI (even in a narrow sense of a human-equivalent software engineer, mathematician, and AI researcher).

However, the question is: how long till it is accelerating AI research more and more noticeably and more and more autonomously?

This mode might start way before AGI (even in the narrow sense of the word I mention above) is achieved…

Thanks for writing this!

Of course, we are still missing METR-style evaluations measuring the ability to complete long tasks for all recent systems (Claude 4, newer versions of Gemini-2.5, and, importantly, for agentic frameworks such as OpenAI Codex, Claude Code and similar systems).

When we obtain those evaluations, we’ll have a better understand of the shape of the curve, whether the doubling period keeps shrinking, and if so, how rapidly it shrinks…

Nice!

An explanation of the APD acronym would be helpful to the readers. In this sense, including this link should be sufficient: https://www.lesswrong.com/posts/EPefYWjuHNcNH4C7E/attribution-based-parameter-decomposition