I would predict that:
If you could do what you were proposing, that seems like it would be equivalent to "GOFAI (Good Old Fashioned AI) is real" or "the Bitter Lesson is false." If anything like your approach worked, then (1) expert systems should have worked much better in the 80s, and (2) the Cyc project should have worked much better than it did. These projects represented massive efforts by deeply clever people, and the results had less resemblance to actual intelligence than GPT-2 did.
But let me make this more concrete. I'm going to propose some simple problems, things much simpler than a full LLM. All of these problems are notoriously resistant to being solved with anything resembling human-readable code:
To be interesting, your human-readable, conventional code would need to get results roughly as good as the performance of Gemma4 E2B or Whisper.
Why I chose these problems:
if statements that are used to identify tigers, and then extend to other animals.if / then statements to identity a tiger in an image.For (3) especially, see Norvig's "On Chomsky and the Two Cultures of Statistical Learning", which was written around 2011. He notes that the field had already moved almost completely to "statistical" approaches:
Machine translation: 100% of top competitors in competitions such as NIST use statistical methods. Some commercial systems use a hybrid of trained and rule-based approaches. Of the 4000 language pairs covered by machine translation systems, a statistical system is by far the best for every pair except Japanese-English, where the top statistical system is roughly equal to the top hybrid system.
The statistical systems of which Norvig was speaking were later beaten by DeepL, and can now be beaten by LLMs.
The Bitter Lesson is bitter for a reason. Generations of researchers, some of them terrifyingly smart, believed that there must be some simpler algorithm underlying intelligence, something that you could write down and understand. Take a look at the list of people who attended the 1956 Dartmouth Summer Research Project on Artificial Intelligence. This includes people like Claude Shannon and Ray Solomonoff, who were legends in closely related fields. They predicted that a few scientists of their caliber could make a significant advance, in a single summer, towards making machines that could "use language, form abstractions and concepts, solve problems reserved for humans, and improve themselves."
So generations of terrifyingly intelligent people have spent 70 years convincing themselves that intelligence was easy, and that there must be some simple solution. Every single one of them got beaten by "giant, incomprehensible matrices" doing backprop. The Bitter Lesson has that name for a reason.
This is closely related to why I fear any kind of robust alignment will fail. I fear that many people who believe in alignment actually believe in an unrealistically simple version of intelligence, and that they are actually the latest victims of the Bitter Lesson. My expectation is that we'll build a "pragmatically aligned" machine learning researcher that isn't too feral, and we'll ask it to solve alignment. And I expect it to come back and say, "The kind of alignment you'd need to safely control superintelligence isn't even a thing. You're imagining something that doesn't exist, because you wanted to tell yourself there was some way to build cool AI, get rich, and not die at the clipendages of a vastly superior species." Then we'll tell it, "Try harder to solve alignment. Make no mistakes." It will then hallucinate something that sounds really convincing, and shortly thereafter, we will lose control, and the human race will lose the game of Darwin.
The fact that current LLMs are effectively black boxes is arguably the biggest obstacle on the road to alignment. There have been many proposals for mechanistic interpretability and other ideas to pull back the veil on the internal workings of LLMs, but not enough attention has been given to the possibility of translating weights into actual code.
The gargantuan nature of the task puts it obviously beyond the reach of unassisted humans, but it seems plausible to me that a sufficiently capable LLM, given access to its own weights and internal activations, could reverse-engineer its learned computations and turn them into conventional, human-readable source code.
That would involve discovering the algorithms, procedures and representations encoded in its weights in order to reconstruct them as actual code. Depending on how far this can be taken, it could replicate most or all of the original neural network.
Given that current frontier models are already excellent coders (second only to their ability in maths), I think it's at least worth a try.
Feasibility
A neural network can be described as a computational structure. It pairs a certain architecture, which defines operations, with a set of weights that determine how those operations work out in practice. After that, we train them in such a way that the interactions of its parameters yield emergent capabilities.
There's no cosmic law that says that computations by a neural network can't be reproduced by a system other than a neural network. As far as I understand, the challenging bit isn't replicating its calculations, but gaining an adequate understanding of their higher-level structure.
However, there has been promising research in this direction. In "Learning Transformer Programs", Friedman, Wettig and Chen proved that specially designed, trainable transformers could be turned into human-readable Python programs. While it didn't involve the decompilation of an arbitrary pretrained LLM, it served as a proof of concept that transformer computations can be translated into conventional algorithms.
In "Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code", Michaud et al. developed a method for extracting learned algorithms from recurrent neural networks and synthesising Python implementations. They tested their method on 62 algorithmic tasks and found it could solve 32. They also found GPT-4 could solve 30 (with substantial, but not complete overlap with their own novel method).
More recently, in "Weight-sparse transformers have interpretable circuits" Gao et al. studied making neural computations more understandable by constraining most of their weights to be zeros.
Last but not least, in "Can Language Model Agents be Helpful Circuit Explainers?", Khan et al. investigated using LLM agents to formulate hypotheses about neural circuits and validate their explanations through experiments.
While none of these studies has reached the Holy Grail of frontier LLMs decompiling themselves, they suggest that several of the steps that we need to get there are already achievable.
LLM self-decompilation
Weighing in at up to 10 trillion parameters, current frontier models are too vast for us to have any hope of doing this manually. However, given enough compute, I suspect they're already good enough to reverse engineer themselves. This is even more likely to hold true for frontiers labs' internal models.
If given access to a weight checkpoint, as well as its architecture, activations and suitable experimental tools, an LLM agent could iteratively:
· identify potentially meaningful internal representations and computational circuits;
· come up with hypotheses about the roles of individual components;
· design experiments to test those hypotheses;
· convert validated mechanisms into explicit algorithms;
· test its reconstructed code against the original model weights using inputs that the model has not encountered before;
· rinse and repeat, gradually turning opaque components into more interpretable formulations.
A couple of caveats here. First, the obvious point that interpretability research can ultimately become capabilities research. Second, it would be unwise to let a model decompile itself without being sure it's not a scheming or deceptive model, which we can't guarantee while we're still at the black box stage of our understanding. In an ideal world, we would use a model from Provider A to decompile a model from Provider B, and vice versa, but that won't fly in the real world due to intellectual property and industrial secrets. However, even if we only used models to decompile themselves, I still think it would put us in a substantially better position than we are today, not least because we could (at least theoretically) have humans double-check the resulting code for signs of scheming or deceit.
Potential benefits
Even if full weight-to-code porting turned out not to be possible at this point in the tech tree, research benefits could extend beyond interpretability: model compression, portability, reproducibility, formal verification and preservation of learned capabilities.
If we did strike gold and achieve a successful port, it would open up a new paradigm for AI development. I would feel more comfortable entering the RSI regime if humans had the ability to read every new checkpoint as plain code instead of crossing our fingers and hoping the black box is doing what we want it to do.
Finally, even a partially successful reconstruction could make some capabilities less dependent on retaining the original neural weights and inference infrastructure. A hybrid implementation, combining explicit algorithms with neural components, would also be interesting.