Me explaining to someone why they are wrong to expect that the singularity will be based on analytic thought as opposed to synthetic thought, namely because as counterintuitive as it is to us analytic thought is actually a subset of/implemented on top of synthetic thought, which should be centrally thought of as something like clustering+ analogical reasoning cohering into a latent space.

Explanation:

https://x.com/QiaochuYuan/status/2080485791952928868

FWIW, I agree with this in the sense that like, no it doesn't really change anything that LLMs can disprove semi famous conjectures. My point is more just that...okay as a very basic observation, if by "synthetic thought" you centrally mean clustering the entire literal problem with symbolic reasoning is that you cannot define a concept like "two" without clustering. You have to be able to do object recognition and say "that is an object, which we will say is object one, and that is an object, which we will say is object two, and we will say that when objects cluster together in this way we have two of them" This is just so atomic and unconscious to us that we do a map territory confusion and start thinking the symbols for "two" start meaning something outside of a latent space which can host the concept of two. This is not to say that symbolic reasoning isn't powerful, it is arguably the most powerful subset of the human latent space. As Alan Kay points out, the alternative to it is doing a guess and check binary search over whole designs, and keeping these giant empirical libraries of things that did and didn't work. You know, the ancient Sumerians didn't have math but they still built complex structures, they did it by basically hill climbing and basing new structures on a learned distribution of old structures to kind of intuit which structures would be likely to stand or not. You can make things that way, but it's not nearly as efficient as math. But math coheres out of that, it's the regular patterns/structure you get from this if you insist on searching for the invariants, it's not like, some separate thing. If this is confusing, consider that like, the fundamental epistemic foundations of mathematics eventually bottom out in clustering operations we can do extremely reliably. Bertrand Russel famously spends 379 pages to prove that 1 + 1 = 2, and the proof is famously wrong. The way you get math is by finding a subset of clustering operations that are so reliable they can be used to define invariants and then repurposing the human in-context reward machinery to assign value to the outcome of algorithms defined in terms of these foundational clustering operations.

Which is also, basically, the way that LLMs are doing math. They use external proofs and solvers to sharpen their neural circuits until they're reliable enough to do increasingly complex mathematical operations mentally without external checking. In humans this is accomplished by pretraining a world model and a little bit of inductive bias (I believe 1, 2, and 3 actually have dedicated neural hardware to make sure a human learns them, or something?) where you can find a reliable enough implementation of the arithmetic equality operator that you can say "1 + 1 = 2", and then you invent algorithms for arithmetic which you use as tools to sculpt your neural circuits with in-context reward until more operations become reliable enough to increase your inferential reach and therefore the library of potential algorithms that can be executed using concepts in your latent space.

You can make things that way, but it's not nearly as efficient as math.

Fundamentally what I'm saying is something like the natural selection argument. It's not that natural selection is an efficient optimizer, or a particularly good way to make things, but it's the first thing that makes things which make things, it is the mechanism by which matter becomes self organizing into life. In the same way it's not that doing gradient search to find a giant matrix which extrapolates new thoughts based on a learned distribution of existing thoughts and observations is the most efficient way to think, but it is the way that the things which make other forms of thought possible get built. You can't have useful symbolic reasoning without it.

It's a bit like the argument that say, surely capitalism is inefficient since an intelligent optimizer can just directly specify the things that should be built. Why do we need to have wasteful distributed intelligence when we can have efficient centralized intelligence? And most of contemporary economics is just various ways to try and explain to people that as intuitive as this idea is it does not work and much of the complexity that a central designer has to deal with is actually irreducible and most efficiently managed by distributed design. Neural nets are a similar thing. The things you need to build to do the feature extraction that makes the application of algorithms possible have irreducible complexity that requires you to do something like gradient search in a high k-complexity program space. There is no simple trick to overcome this, the closer your method gets to the convergent shape of gradient search over generating programs of the surprising amount of detail in reality the more neural-net like it becomes in the same way that the more closely your economic system gets to the convergent shape of distributed intelligence doing price discovery the more it begins to resemble Neoliberalism, which is why the Soviet Union was ideologically doomed in a way that the modern Chinese government has been able to escape by simply establishing markets and regulating them.