A Gödelian Way of thinking about limits to AI and potential economic outcomes arising from these shortcomings
Motivations:
As an incoming college freshman about to place a fairly important bet on what might be worth studying so as to remain relevant at the end of four years, I have been increasingly thinking about how the world might look in the next decade, a decade that in all probability is going to be characterised by super-advanced artificial intelligence (while one may call it AGI, but I'm slightly apprehensive of using AGI without a very clear definition or benchmark.)[1] While I understand that there is no definite result to this question, I still think that at the very least it’s an interesting thought exercise precisely because of the speed of technological advancement in AI for which we don’t have any historical parallels. This lack of historical parallels makes this a particularly difficult task, as previous revolutions like the Bronze Age or the Industrial Revolution primarily threatened human physical strength and efficiency, which, while being considered an important ability, has many challengers in the natural world; cognition and intellectual abilities have thus far been the sole invincible arguments to ‘human exceptionalism,' which are now coming under strain. The thoughts I present are not arguments to validate human exceptionalism but more a scenario of how I think the AI revolution might transpire.
Introduction:
This scenario draws inspiration heavily from the generalized, albeit non-rigourous, idea of Gödel’s Incompleteness Theorem that Douglas Hoffsteder presents in GEB: An Eternal Golden Braid. Gödel's Incompleteness The theorem famously shows that it’s not possible for a formal system to reason about itself. In the context of number theory, he presents all proofs, statements, and other textual references as numbers to show a contradiction that occurs when formal systems can refer to themselves within that system and how, in this system, certain truths are unprovable. Self-reference by itself is fairly common: the idea of metacognition, for instance, implies that minds are thinking about minds, which means that one’s thoughts are insufficient to draw out a complete logical structure for their brain’s inner workings. That being said, in the context of mathematics, the formal system incompleteness was first thought of for it; it’s not a major worry for mathematicians. It’s widely believed that solutions to famous open problems like the Riemann Hypothesis or the Collatz conjecture are unlikely to run into limits imposed by the theorem. That being said, there are some more abstract problems like the continuum hypothesis, which involves the proof that the infinite set of real numbers is strictly the second largest set of numbers after the set of natural numbers, which are known to be unprovable because of incompleteness. The continuum hypothesis also has certain implications in the context of quantum field theory, most notably in the proof that was used to debunk the simulation hypothesis—the idea that our entire world is a computer simulation of a higher entity. The implications of this mean that there is no computationally feasible ‘theory of everything,' as various non-algorithmic processes also comprise this world.
In the absence of computational models, science tends to rely on computational models to approximate various interactions. Such models are commonplace in quantum dynamics, as many problems are seen as intractable within the formal system used to reason about them. Beyond quantum field theory, problems like predicting human responses, or corollaries of the same problem, like predicting market reactions or even determining exact medical outcomes on a patient-by-patient basis, are also likely to be non-algorithmic.[2] Further, if the Lucas-Penrose argument is to be believed, the very nature of human consciousness is not computable and is a quantum system, therefore being beyond the scope of a Turing machine like LLMs to emulate completely, bringing the entire premise of AGI into question.[3] Thus, human consciousness is another example of a non-algorithmic field where models dominate over completely correct explanations.
In using a Gödelian method to reason about AI, we make the assumption that AI is in and of itself an internally consistent formal system. While literature does not clearly draw this parallel, I believe it’s a reasonable assumption to make, as all LLM outputs are at their very basic levels outputs of probabilistic distributions and the results of matrix multiplication, which are arithmetic systems, for which Gödelian incompleteness applies. So, I believe it’s reasonable to call the underlying working of LLMs and their output part of a formal system.
This article aims to generalize the ideas about incompleteness presented thus far to the context of artificial intelligence and problem-solving with the tool. In particular, I attempt to present my version of why I don’t think AI is a solution to everything and how the gaps left by this might continue to remain a domain for human intellectual output and some implications of this in a concrete sense.
What Incompleteness in AI may look like?
Incompleteness in AI may become most clear if strange loops in the context of AI are formulated. Hoffsteeder uses the term "strange loop" to describe a hierarchical self-referential system. An example of a strange loop in a more conventional context would be, for instance, in time travel, where actions of the current entity in the past have a cyclical effect on that present entity for all sequences of the passing of time. This system naturally breaks down as we don't know at which instant in time the event actually occurs and to what extent events of the present are a result of the present entity’s actions in the past. In the context of AI, the extent to which the system breaks down is not that clear but existent, nevertheless.
For example, let’s assume a dystopian world where all decisions are a product of AI’s actions; this creates a strange loop where AI ends up reasoning and responding to decisions taken by itself. One may argue that the exact effect occurs in humans, where it’s not seen as a strange loop. In this case, I believe the system doesn’t break down (too much) in the human context due to the sheer diversity within human civilization. The decision-making process of one human differs significantly from that of the other, which leads to heterogeneous decision-making, even if at times it's inefficient. Large language models, on the other hand, are known to show great convergence in their output, be it in linguistic style or, more importantly, in the context of what decision it thinks is most optimal. I think the lack of diversity in creative responses leads to a self-referential loop where optimal decisions are no longer optimal because the best decisions are based on some delta in knowledge or decision-making ability.[4] Further, Shumailov et al. show that model failure occurs when they are recursively trained on data that is generated by the model. This represents a statistically rigorous bound on LLM ability as well as a role for humans (see the section on The Human Role in Preventing Model Failure). The authors attribute this model collapse to homogeneity in model-generated textual quality, but I believe this broad pattern also applies in the context of decision-making.
Admittedly, the previous example is highly abstract, and I’m not sure whether this is likely to have a major impact, primarily because if one continues this line of argumentation, one might also arrive at a highly desirable and equal world, which is not necessarily a bad outcome. That caveat aside, I believe there is another more concrete self-referential loop in the context of solving AI alignment. If we leave a future super-AI to become better, fix its biases, and perform exceedingly well in safety benchmarks, it’s analogous to asking a person to become better in isolation. I think this is a fairly pointless activity, because it creates a loop with an indefinite end point.
To make the point introduced previously more rigorous, one can invoke the idea of Rice’s theorem, especially in the context of the alignment problem. Rice’s theorem suggests that all non-trivial semantic properties of programs are undecidable. An example of a non-trivial semantic property might include questions as to whether the program might accept an empty string or whether it will generate a particular output; these examples are classified as such properties as they vary from computer to computer and are comments on the program itself as opposed to the language of the program. The question on alignment and AI safety is a difficult, non-trivial semantic property; thus, it can be conjectured that it’s not possible for a program to determine whether another AI is safe or not, making the recursive improvement of AI highly impossible.
I believe these gaps arising from AI referential loops, which are also related to a baseline level of stochasticity in this world, will allow for some areas for more constructive human-AI collaboration.[5] In the two examples taken (decision-making and alignment research) and other such areas of constructive collaboration, I believe the common thread is subjects that involve modeling to find proximate solutions as opposed to finding a definite solution. While I don’t think this is going to fundamentally shift the areas of intellectual exploration, I am confident it’s going to change the ways in which that exploration is carried out. More on what I think this pipeline might look like along with examples in the next section.
Previously cited research indicates that there is unlikely to be a universal algorithm to explain every phenomenon, and I think that this also leads to the potential to ask infinite questions as mundane or abstract as they may be. I think the initiative to ask these questions and direct AI towards these various directions will more naturally expose the incompleteness in AI’s ability to create a global logic. While I'm not sure how these gaps might arise, I’m sure that they will arise even for a super-intelligent machine, where a human’s role to ask questions and focus the efforts of the AI will be useful, especially if that model, like those of today, is better at short-term, focused tasks. I will not be delving too much into this argument, as it’s one that has been made many times in a far more rigorous manner than how I have introduced it.
Implications of the Incompleteness:
Thus far, the focus of this essay has been on the idea that the fundamental incompleteness in the world will lead to impetus on creating better models for our world as opposed to definitive solutions. This has been presented as an area for constructive human-AI collaboration, but there is an argument to be made that the models generated by AI will be superior in performance to those generated by a human, leaving the human without any major contribution in the model-building task. I think this is certainly a highly plausible scenario, especially for a future AI model that is equivalent to or better than experts in a wide number of fields. So, then are we back to the same problem?
I’d like to believe that in this scenario there is a crucial difference from the typical case where there is only one correct answer. I think with the infinitude of models the AI will be able to generate, taking into account different assumptions, there is no scarcity in computation or idea generation; instead, with the sheer number of models and ways of conceiving knowledge, the focus is on aligning models with one’s beliefs and values. This is in some ways close to the idea of the much-touted idea of 'taste,' where, with so many options of investigation or for building, there is going to be a lot of impetus given to what we do with that power. While I think this will be an important dimension, I would think that a super-AI that's asked to be curious will generate original, worthy hypotheses and pursue them, completing the entire loop of inquiry.[6] However, in model selection and hypothesis generation (and execution), I believe there are also elements of personal belief and values, which serve as a valid ground to push back against an AI’s output. The point being that in the absence of there being an objectively best model, there are arguments for other equally plausible theories, which are irrefutable.
To give a more concrete example of where this might work, I first think of my interactions with AI in a chemometric project that I have most recently been working on. An example of incompleteness in AI’s output would possibly be best seen by the fact that while it did suggest a number of methods of analyzing the spectra, based on my tinkering with the spectra, I noticed another metric that would help us in our core hypothesis, not because I think AI was incapable of it because it mentioned the top-k approaches, as it understood, but because in reality there are certainly more possibilities that both I and the model haven’t thought of. On the question of taste and ideas adjacent to it, called Taste+, hereinafter, present AI models in their bid to be most comprehensive tend to look at analysis metrics that produce very interesting charts but are not necessarily informative (In my opinion.) Taste+ can be defined as exercising non-computable, non-arbitrary decision-making, which draws on lived experiences, situated values, or epistemic commitments. The choice of adding a ‘+’ at the end of 'Taste' is to refer not only to the fact that it’s an extension to the accepted definition of 'Taste,' but also to refer to related ideas like ‘selection’ or 'initiative,' which go hand in hand with Taste.
The point I’m getting at is that incompleteness in knowledge and AI’s generation capabilities mean that there will always be valid reasons to push back against AI-generated output, and personal beliefs come into play in this gap, preserving a role for another diverse perspective, which can be brought by the humans in the loop. The premise here lies in the fact that humans are not going to be valued for their technical brilliance or sheer cognitive abilities to generate ideas but rather for the diversity in thought they bring and their ability to use that thought to critically oppose the status quo and make a decision accordingly.[7]
To generalize beyond personal examples, Taste+ might show itself in financial markets where analysts don’t have an edge because of knowledge about how to execute traditional strategies or pricing metrics but rather asset-specific metrics, which, while having been developed with AI and a little human thought for diversity, still make the decision to specifically invest in recent stocks or brute force maximize profits a personal decision. The human’s primary focus (also aided by AI) could, in this situation, be shifted towards the higher abstraction of finding new patterns in multiple asset-specific metrics. And, when a decent level of progress (also a human decision based on beliefs to break the loop) is made in a faster time than ever before in history, they simply move to the next idea or questions, which may or may not be AI-generated.[8] This example also offers a primer for what I think may be a future impetus on generalization as opposed to specialization, which I discuss in the next section.
In particular, I believe that the role of Taste+ becomes most profound in the bigger questions, like “What’s consciousness?” or “How do we solve global hunger? "I think this is true because given the uncertain nature of the answer to these solutions, models become more important, and where there are models, there are a greater number of non-deterministic ethical and performance tradeoffs among infinite possible tradeoffs to be made, which I hypothesize becomes the human task.
Possible Economic Opportunities Arising From It:
Following the widespread availability of LLMs, common career advice has been to go into fields like agriculture or construction, where AI penetration thus far has not been very significant. I believe that such advice is very reactionary and short-term, because AI in robotics is only further away on the AI development ladder compared to better AI for cognitive tasks, but it is something that’s surely going to catch up, so it’s unlikely that this advice will be of great importance while deciding on the average career spanning 40 years, because this technology is very likely to arrive by such a time. This leads me to the belief that while there may be an initial temporary influx towards blue-collar jobs, in the long term the focus will shift back to white-collar tasks that require the active use of Taste+. Beyond just Taste+, I also agree that tasks where human ability or interaction is important should continue to exist to serve the human population: jobs like athletes, counselors, conflict resolvers, or theater actors. But beyond this, I believe Taste+Initiative in particular might have greater importance in the economic context as more routine tasks get automated away; the impetus in this case lies in knowledge or product creation where the idea of Taste+ plays an important role as both pathways are unconstrained and have infinite possibilities, which need selection.
This leads me to believe that with greater impetus on initiative, research (be it in academia or industry) and entrepreneurship are likely to see a renaissance.[9] But even here, I think the fundamental shift is going to be that with specialized tasks being automated away, we are increasingly going to have labs and companies chasing for answers to more fundamental questions and solutions to far bigger problems across different levels. One way this might manifest is far more interdisciplinary work wherein pharmaceutical chemists working on nutritional supplements also look at developmental economics to understand the feasibility of their supplements to reduce malnutrition in the Global South. Or, in another example, perhaps theoretical computer scientists, often working on fairly esoteric limits to computation, may extend their work to the larger philosophical implications of it and work towards educating others on the ideas. This line of work is unlikely to be completely challenged by AI given the previously stated role of Taste+ in deciding between attitudinal possibilities and in adding diversity to thinking.
In some ways there is an odd cyclicality to this phenomenon, which is worth noting but is not an argument in and of itself. Back in Ancient Greece, by extension (as far as intellectual inspiration goes), the Renaissance had polymathic thinkers like Aristotle, who was a philosopher and the father of biology, and Leonardo da Vinci, who was a painter, engineer, and anatomist. So, with specialization being tasked away to AI, broader, more interdisciplinary fields like Naturalis Philosophiae or Quantitative Policymaking can come to the fore.
That being said, it’s worth also saying that the labs, individual enthusiasts, and private enterprises working on these tasks are likely to be smaller in their headcount due to the efficiency gains that AI will bring with it. To a certain extent, it can be a good thing, as it will mean a reduction in the over-allocation of talent towards the same select few industries and will push more people towards experimenting and innovation.[10] The reallocation in this context will involve more than just a drive towards entrepreneurship but will also involve diversification in the sectors where talent is concentrated beyond technology to also include sectors like education or transportation or welfare.
But this diversification is likely to only account for some job creation, which is likely to be insignificant in front of the greater unemployment that efficiency gains will create. So, I don’t think that everybody is going to come to the same level in exercising Taste+, and not enough people will be willing to take the plunge, leading to significant unemployment. This risks exacerbating further inequality, which is not a very favorable outcome for society as a whole. I think this is where ideas such as Universal Basic Income (UBI) and improving welfare support in countries are likely to come into play.[11]
In saying this, the important thing to note is that while inequality can rise greatly with artificial intelligence, I don’t believe the fundamental nature of this inequality is going to be different from the inequality we experience today. That is to say, I don't believe inequality is going to be a product of the absence of economic opportunities. It’s rather going to be a product of economic opportunities becoming more constrained and possibly more stochastic in who is able to gain the greatest advantage due to the uncertain nature of entrepreneurial and research breakthroughs. Welfare mechanisms need to be put in place to cushion everyone from its effects. But even in this, baseline inequalities in global GDPs are likely to significantly affect the extent to which this welfare cushion is available.
The positive belief I have with regard to global disparities and AI is that AI penetration is likely to correlate strongly with country GDP. Richer countries, with typically smaller populations, are likely to invest more in making AI mainstream; at the same time, their governments have the financial resources to provide support to their citizens through UBI or other welfare policies, who are also fewer in number. The effects of AI are likely to be far more pronounced in the Global South, but structural inefficiencies may mean that AI penetration occurs at a much slower rate, especially in informal economies, which make up a large proportion of these economies. This may mean that these countries have a longer time period enabling better planning from the learnings of the Global North and possibly time for some transfer of financial resources to these countries so that we come closer to the economic aspects of ensuring everyone has a somewhat equal right to life at least as far as economic aspects are concerned. Also, as time progresses with falling global fertility rates, we might find ourselves in a position where we need AI to sustain economic output; that said, this is a possible implication in the very long term.
Some Related but Adjacent Thoughts:
An Extention to the Idea of Initiative:
Two subjects that are increasingly being used as standard benchmarks for the performance of AI are math and programming. In math, especially with an OpenAI model disproving Erdős's Unit Distance Conjecture, there are a growing number of skeptics with regard to the future role of mathematicians. In programming tasks, as well, LLM-produced code has reached a very high degree of accuracy.[12] Some mathematicians continue to believe that it will only serve as an aid or as a tool for quick checking, allowing for greater collaboration, and I think this is certainly possible; a lot of, if not all, low-hanging mathematical fruits are likely to be solved autonomously as well as certain non-trivial mathematics, which by themselves are going to be a dent in the actual quantity of output of mathematicians. So, I think here as well initiative is going to fill the gaps of greater saturation of existing math, which is going to, in my opinion, lead to the creation of new mathematics. Mathematics has always been built on looking at new structures or investigating the possibilities of different assumptions, so, for example, we may investigate a new mathematics with an additional logical axiom. In this situation we are looking for new models of mathematics built of new assumptions, where Taste+ once again comes into play. I think the idea generalizes easily to the “hard sciences": computer science and theoretical physics, where perhaps one could conjecture a physical realm where pre-Galilean motion exists and whether this might have any actual applications. For programming-related tasks, I don’t directly see any parallel that might fit the pattern where models may be created, but I may be wrong.
Maybe a Precursor to Homo Deus:
Yuval Noah Hariri, in his book “Homo Deus,”talks about a future union between man and computer as man strives to achieve goals of superintelligence, artificial modifications, and longevity. Especially with the developments in synthetic biology and Brain-Computer Interfaces (BCI), that’s increasingly becoming a plausible future reality. But, with the AI revolution and as a method to preserve economic meaning, a possible method may leverage the ability of AI to become user-specific. The broad idea underlying this is that when a human is tasked with a job, it’s a given that the human and the AI that’s fine-tuned to their way of thinking, their workflows, etc. are deputized to the task. I think that these personalized AI bots, as opposed to centralized AI, are likely to come about as a result of user preferences, but I think it has some important positive implications for AI safety.
With social media, one of the factors that allowed companies to get away with highly addictive and polarizing features and content was the monopoly power of the companies. For effective regulation over AI, I believe we need a greater number of frontier labs that are actively competing. This I believe will push companies to distinguish themselves with more fine-tunable and possibly open-source models, which can greatly accelerate AI safety research. Further, I think this might also reduce inequality, as we won't be concentrating the power that the technology gives and the economic benefits to a select group. On other issues, like AI for malicious use, I believe, admittedly idealistically, that user beliefs can steer more companies to enact stronger safeguards to prevent malicious use, which may not be possible when users have very few options in the first place.
The Human Role in Preventing Model Failure:
A major reason behind the success of LLMs has been their ability to scale with greater data and improve their output substantially with greater data. That said, Shumailov et al. conclude that AI models that are trained on their output become increasingly uniform and lack the intrinsic diversity of human interaction. The authors in the paper emphasize that future model training must give greater importance to human prompting and human-bot interactions to enhance model performance as opposed to relying completely on data scraped from the internet, which might have a greater amount of AI-generated content in the future. Therefore, a possible way for humans to contribute economically is by interacting with bots for a diverse range of tasks and manually making photos and videos to ensure that LLMs preserve their quality for the majority of the users. This argument has not been presented as a major argument in justifying the economic utility of human workers, primarily because this argument only holds if the future of AI is only going to be unlocked with LLMs. While I believe LLMs will still continue to be an important tool in language tasks involving direct interaction with humans for the foreseeable future, I do believe better architectures will emerge for specific tasks like AI4Science or an architecture that has enhanced memory recall. I once again add the disclaimer that I don’t think this possible pathway of human economic contribution is likely to be sufficient to provide employment to the maximum human population of 10.3 billion people that are likely to inhabit the planet by the mid-2080s.
Another Argument Against a Gödel Machine and Taste+:
A Gödel machine is a problem-solving tool that makes use of self-referentiality to iteratively improve itself and get better at a task. These models consist of a layer that interacts with the user and a meta layer that finds a mathematical proof that a specific modification in the model architecture will improve performance. A modern variant of the model makes use of empirical observations as opposed to mathematical proofs.[13] For these models, the previous argument using Rice’s theorem applies, as the theorem implies that there will exist a class of improvements that are mathematically unprovable and therefore can’t be detected by the meta-layer of the machine. Even if empirical benchmarks are used as opposed to mathematical proofs, there are risks that machines will optimize for hacking the benchmark as opposed to genuine improvement. But a bigger concern is the computational complexity of the problem in the first place. Searching for proofs or empirical improvements of a random number of improvements is computationally challenging due to the sheer size of the subspace, and even if results are obtained, there are no guarantees that those improvements will be substantial enough to search the entire space. I believe the question of Taste+ comes here again not because I think human intuition is in anyway superior or more likely to excel at the task, but because some basic set of beliefs and assumptions needs to be made to guide the exploration. By the heterogeneity in human thought, these assumptions are likely to be different enough for a large number of people, thus increasing the likelihood of global progress towards the task even if there is no guarantee for individual success.
Conclusion:
Thinking about various issues related to AI has been an interesting intellectual exercise, but that said, I don't necessarily expect all or any of my predictions to be accurate. As is true for the dawn of any technology, the future is highly uncertain, and I think it may be an interesting idea to compare current uncertainty as reflected in media with that of previous revolutions, like with the discovery of the internet in the 1990s or the Industrial Revolution. My arguments, right from using Gödel's Incompleteness for AI, which may not qualify as a formal system, and the adjacent thoughts I mentioned are closer to a primer for an argument as opposed to a complete argument with a proper analysis of previous literature. That said, I do believe in the economic promise of incompleteness in AI. But this doesn't take away from the risk of high unemployment or AI wars or extreme inequality; the version of events I present are some possibilities and solutions I think are worth considering. Homo sapiens are a resilient species, and for the sake of our continued prosperity, I hope for this essay to serve as one of many models (as described in the essay) for the future, which is a worthy hypothesis to debate more than a completely accurate representation, which I don’t believe exists.
While the ARC-AGI does exist as a benchmark for AGI, I’m not sure whether adaptable learning is the only or best benchmark for AGI. Also, there are also valid arguments to be made that models that don’t perform very well on ARC AGI might be fully capable of replacing many human tasks, which is frequently seen as the primary result of AGI.
I use ‘likely’ in the absence of formal proofs that suggest their non-algorithmic nature as they exist in QFT. I think that this is reasonable, as even if some magical theory exists, it’s likely to be very complex and difficult, making models easier and more efficient options.
While I believe that there are certain processes like human consciousness that are non-computable, I won't go so far as to say that’s because of quantum processes in our brain. I believe certain parts of the argument do silently promulgate human exceptionalism, which I don't think is a universal truth.
This statement may seem contradictory, but it represents the fact that while AI is often able to suggest high-quality responses, the nature of those responses is remarkably similar across models, with methods by which it gets to those creative decisions often non-existent or partially explained.
While I’m not sure whether the first AGI model will completely finish this loop, it’s certainly possible for models that are more intrinsically curious by design, perhaps JEPA-type models which actively explore our world, like us, in their learning process.
Status Quo may not be the best word, but it’s used as a synonym for beliefs, opinions, and ideas purported by AI. Also, it’s worth clarifying that I don’t advocate disagreeing with the bot for disagreement’s sake but rather for bringing in diversity of thought.
In this specific instance, the idea of finding generalization is an example of Taste+Initiaitve and the deliberate choice of maximizing one principle over the other can be considered an example of Taste+Selection. Henceforth, Taste+ will be used as an overarching term to refer to these corollaries as well.
There are already some indications of this trend that’s coming to the fore with greater venture capital investment in AI, as opposed to continuing with more conventional wealth redistributive roles in finance, law, or consulting.
This article doesn't attempt to examine the validity or complete feasibility of UBI. There are far more detailed studies on the subject by people who have thought about it far more than me. I introduce UBI as something to talk about intercountry differences in economic capital to deal with disruption due to AI.
The only caveat to this point is that some people may feel that AI proofs are not as clear in motivating the solution to the problem or that AI code is not very readable. While these are certainly true in the current context, I’m not sure whether these are problems that can’t be solved with fine-tuning.
Gödel Darwin Machines, by looking at empirical improvement as the only benchmark, there are grave safety concerns, as it risks further reducing model interpretability.
A Gödelian Way of thinking about limits to AI and potential economic outcomes arising from these shortcomings
Motivations:
As an incoming college freshman about to place a fairly important bet on what might be worth studying so as to remain relevant at the end of four years, I have been increasingly thinking about how the world might look in the next decade, a decade that in all probability is going to be characterised by super-advanced artificial intelligence (while one may call it AGI, but I'm slightly apprehensive of using AGI without a very clear definition or benchmark.)[1] While I understand that there is no definite result to this question, I still think that at the very least it’s an interesting thought exercise precisely because of the speed of technological advancement in AI for which we don’t have any historical parallels. This lack of historical parallels makes this a particularly difficult task, as previous revolutions like the Bronze Age or the Industrial Revolution primarily threatened human physical strength and efficiency, which, while being considered an important ability, has many challengers in the natural world; cognition and intellectual abilities have thus far been the sole invincible arguments to ‘human exceptionalism,' which are now coming under strain. The thoughts I present are not arguments to validate human exceptionalism but more a scenario of how I think the AI revolution might transpire.
Introduction:
This scenario draws inspiration heavily from the generalized, albeit non-rigourous, idea of Gödel’s Incompleteness Theorem that Douglas Hoffsteder presents in GEB: An Eternal Golden Braid. Gödel's Incompleteness The theorem famously shows that it’s not possible for a formal system to reason about itself. In the context of number theory, he presents all proofs, statements, and other textual references as numbers to show a contradiction that occurs when formal systems can refer to themselves within that system and how, in this system, certain truths are unprovable. Self-reference by itself is fairly common: the idea of metacognition, for instance, implies that minds are thinking about minds, which means that one’s thoughts are insufficient to draw out a complete logical structure for their brain’s inner workings. That being said, in the context of mathematics, the formal system incompleteness was first thought of for it; it’s not a major worry for mathematicians. It’s widely believed that solutions to famous open problems like the Riemann Hypothesis or the Collatz conjecture are unlikely to run into limits imposed by the theorem. That being said, there are some more abstract problems like the continuum hypothesis, which involves the proof that the infinite set of real numbers is strictly the second largest set of numbers after the set of natural numbers, which are known to be unprovable because of incompleteness. The continuum hypothesis also has certain implications in the context of quantum field theory, most notably in the proof that was used to debunk the simulation hypothesis—the idea that our entire world is a computer simulation of a higher entity. The implications of this mean that there is no computationally feasible ‘theory of everything,' as various non-algorithmic processes also comprise this world.
In the absence of computational models, science tends to rely on computational models to approximate various interactions. Such models are commonplace in quantum dynamics, as many problems are seen as intractable within the formal system used to reason about them. Beyond quantum field theory, problems like predicting human responses, or corollaries of the same problem, like predicting market reactions or even determining exact medical outcomes on a patient-by-patient basis, are also likely to be non-algorithmic.[2] Further, if the Lucas-Penrose argument is to be believed, the very nature of human consciousness is not computable and is a quantum system, therefore being beyond the scope of a Turing machine like LLMs to emulate completely, bringing the entire premise of AGI into question.[3] Thus, human consciousness is another example of a non-algorithmic field where models dominate over completely correct explanations.
In using a Gödelian method to reason about AI, we make the assumption that AI is in and of itself an internally consistent formal system. While literature does not clearly draw this parallel, I believe it’s a reasonable assumption to make, as all LLM outputs are at their very basic levels outputs of probabilistic distributions and the results of matrix multiplication, which are arithmetic systems, for which Gödelian incompleteness applies. So, I believe it’s reasonable to call the underlying working of LLMs and their output part of a formal system.
This article aims to generalize the ideas about incompleteness presented thus far to the context of artificial intelligence and problem-solving with the tool. In particular, I attempt to present my version of why I don’t think AI is a solution to everything and how the gaps left by this might continue to remain a domain for human intellectual output and some implications of this in a concrete sense.
What Incompleteness in AI may look like?
Incompleteness in AI may become most clear if strange loops in the context of AI are formulated. Hoffsteeder uses the term "strange loop" to describe a hierarchical self-referential system. An example of a strange loop in a more conventional context would be, for instance, in time travel, where actions of the current entity in the past have a cyclical effect on that present entity for all sequences of the passing of time. This system naturally breaks down as we don't know at which instant in time the event actually occurs and to what extent events of the present are a result of the present entity’s actions in the past. In the context of AI, the extent to which the system breaks down is not that clear but existent, nevertheless.
For example, let’s assume a dystopian world where all decisions are a product of AI’s actions; this creates a strange loop where AI ends up reasoning and responding to decisions taken by itself. One may argue that the exact effect occurs in humans, where it’s not seen as a strange loop. In this case, I believe the system doesn’t break down (too much) in the human context due to the sheer diversity within human civilization. The decision-making process of one human differs significantly from that of the other, which leads to heterogeneous decision-making, even if at times it's inefficient. Large language models, on the other hand, are known to show great convergence in their output, be it in linguistic style or, more importantly, in the context of what decision it thinks is most optimal. I think the lack of diversity in creative responses leads to a self-referential loop where optimal decisions are no longer optimal because the best decisions are based on some delta in knowledge or decision-making ability.[4] Further, Shumailov et al. show that model failure occurs when they are recursively trained on data that is generated by the model. This represents a statistically rigorous bound on LLM ability as well as a role for humans (see the section on The Human Role in Preventing Model Failure). The authors attribute this model collapse to homogeneity in model-generated textual quality, but I believe this broad pattern also applies in the context of decision-making.
Admittedly, the previous example is highly abstract, and I’m not sure whether this is likely to have a major impact, primarily because if one continues this line of argumentation, one might also arrive at a highly desirable and equal world, which is not necessarily a bad outcome. That caveat aside, I believe there is another more concrete self-referential loop in the context of solving AI alignment. If we leave a future super-AI to become better, fix its biases, and perform exceedingly well in safety benchmarks, it’s analogous to asking a person to become better in isolation. I think this is a fairly pointless activity, because it creates a loop with an indefinite end point.
To make the point introduced previously more rigorous, one can invoke the idea of Rice’s theorem, especially in the context of the alignment problem. Rice’s theorem suggests that all non-trivial semantic properties of programs are undecidable. An example of a non-trivial semantic property might include questions as to whether the program might accept an empty string or whether it will generate a particular output; these examples are classified as such properties as they vary from computer to computer and are comments on the program itself as opposed to the language of the program. The question on alignment and AI safety is a difficult, non-trivial semantic property; thus, it can be conjectured that it’s not possible for a program to determine whether another AI is safe or not, making the recursive improvement of AI highly impossible.
I believe these gaps arising from AI referential loops, which are also related to a baseline level of stochasticity in this world, will allow for some areas for more constructive human-AI collaboration.[5] In the two examples taken (decision-making and alignment research) and other such areas of constructive collaboration, I believe the common thread is subjects that involve modeling to find proximate solutions as opposed to finding a definite solution. While I don’t think this is going to fundamentally shift the areas of intellectual exploration, I am confident it’s going to change the ways in which that exploration is carried out. More on what I think this pipeline might look like along with examples in the next section.
Previously cited research indicates that there is unlikely to be a universal algorithm to explain every phenomenon, and I think that this also leads to the potential to ask infinite questions as mundane or abstract as they may be. I think the initiative to ask these questions and direct AI towards these various directions will more naturally expose the incompleteness in AI’s ability to create a global logic. While I'm not sure how these gaps might arise, I’m sure that they will arise even for a super-intelligent machine, where a human’s role to ask questions and focus the efforts of the AI will be useful, especially if that model, like those of today, is better at short-term, focused tasks. I will not be delving too much into this argument, as it’s one that has been made many times in a far more rigorous manner than how I have introduced it.
Implications of the Incompleteness:
Thus far, the focus of this essay has been on the idea that the fundamental incompleteness in the world will lead to impetus on creating better models for our world as opposed to definitive solutions. This has been presented as an area for constructive human-AI collaboration, but there is an argument to be made that the models generated by AI will be superior in performance to those generated by a human, leaving the human without any major contribution in the model-building task. I think this is certainly a highly plausible scenario, especially for a future AI model that is equivalent to or better than experts in a wide number of fields. So, then are we back to the same problem?
I’d like to believe that in this scenario there is a crucial difference from the typical case where there is only one correct answer. I think with the infinitude of models the AI will be able to generate, taking into account different assumptions, there is no scarcity in computation or idea generation; instead, with the sheer number of models and ways of conceiving knowledge, the focus is on aligning models with one’s beliefs and values. This is in some ways close to the idea of the much-touted idea of 'taste,' where, with so many options of investigation or for building, there is going to be a lot of impetus given to what we do with that power. While I think this will be an important dimension, I would think that a super-AI that's asked to be curious will generate original, worthy hypotheses and pursue them, completing the entire loop of inquiry.[6] However, in model selection and hypothesis generation (and execution), I believe there are also elements of personal belief and values, which serve as a valid ground to push back against an AI’s output. The point being that in the absence of there being an objectively best model, there are arguments for other equally plausible theories, which are irrefutable.
To give a more concrete example of where this might work, I first think of my interactions with AI in a chemometric project that I have most recently been working on. An example of incompleteness in AI’s output would possibly be best seen by the fact that while it did suggest a number of methods of analyzing the spectra, based on my tinkering with the spectra, I noticed another metric that would help us in our core hypothesis, not because I think AI was incapable of it because it mentioned the top-k approaches, as it understood, but because in reality there are certainly more possibilities that both I and the model haven’t thought of. On the question of taste and ideas adjacent to it, called Taste+, hereinafter, present AI models in their bid to be most comprehensive tend to look at analysis metrics that produce very interesting charts but are not necessarily informative (In my opinion.) Taste+ can be defined as exercising non-computable, non-arbitrary decision-making, which draws on lived experiences, situated values, or epistemic commitments. The choice of adding a ‘+’ at the end of 'Taste' is to refer not only to the fact that it’s an extension to the accepted definition of 'Taste,' but also to refer to related ideas like ‘selection’ or 'initiative,' which go hand in hand with Taste.
The point I’m getting at is that incompleteness in knowledge and AI’s generation capabilities mean that there will always be valid reasons to push back against AI-generated output, and personal beliefs come into play in this gap, preserving a role for another diverse perspective, which can be brought by the humans in the loop. The premise here lies in the fact that humans are not going to be valued for their technical brilliance or sheer cognitive abilities to generate ideas but rather for the diversity in thought they bring and their ability to use that thought to critically oppose the status quo and make a decision accordingly.[7]
To generalize beyond personal examples, Taste+ might show itself in financial markets where analysts don’t have an edge because of knowledge about how to execute traditional strategies or pricing metrics but rather asset-specific metrics, which, while having been developed with AI and a little human thought for diversity, still make the decision to specifically invest in recent stocks or brute force maximize profits a personal decision. The human’s primary focus (also aided by AI) could, in this situation, be shifted towards the higher abstraction of finding new patterns in multiple asset-specific metrics. And, when a decent level of progress (also a human decision based on beliefs to break the loop) is made in a faster time than ever before in history, they simply move to the next idea or questions, which may or may not be AI-generated.[8] This example also offers a primer for what I think may be a future impetus on generalization as opposed to specialization, which I discuss in the next section.
In particular, I believe that the role of Taste+ becomes most profound in the bigger questions, like “What’s consciousness?” or “How do we solve global hunger? "I think this is true because given the uncertain nature of the answer to these solutions, models become more important, and where there are models, there are a greater number of non-deterministic ethical and performance tradeoffs among infinite possible tradeoffs to be made, which I hypothesize becomes the human task.
Possible Economic Opportunities Arising From It:
Following the widespread availability of LLMs, common career advice has been to go into fields like agriculture or construction, where AI penetration thus far has not been very significant. I believe that such advice is very reactionary and short-term, because AI in robotics is only further away on the AI development ladder compared to better AI for cognitive tasks, but it is something that’s surely going to catch up, so it’s unlikely that this advice will be of great importance while deciding on the average career spanning 40 years, because this technology is very likely to arrive by such a time. This leads me to the belief that while there may be an initial temporary influx towards blue-collar jobs, in the long term the focus will shift back to white-collar tasks that require the active use of Taste+. Beyond just Taste+, I also agree that tasks where human ability or interaction is important should continue to exist to serve the human population: jobs like athletes, counselors, conflict resolvers, or theater actors. But beyond this, I believe Taste+Initiative in particular might have greater importance in the economic context as more routine tasks get automated away; the impetus in this case lies in knowledge or product creation where the idea of Taste+ plays an important role as both pathways are unconstrained and have infinite possibilities, which need selection.
This leads me to believe that with greater impetus on initiative, research (be it in academia or industry) and entrepreneurship are likely to see a renaissance.[9] But even here, I think the fundamental shift is going to be that with specialized tasks being automated away, we are increasingly going to have labs and companies chasing for answers to more fundamental questions and solutions to far bigger problems across different levels. One way this might manifest is far more interdisciplinary work wherein pharmaceutical chemists working on nutritional supplements also look at developmental economics to understand the feasibility of their supplements to reduce malnutrition in the Global South. Or, in another example, perhaps theoretical computer scientists, often working on fairly esoteric limits to computation, may extend their work to the larger philosophical implications of it and work towards educating others on the ideas. This line of work is unlikely to be completely challenged by AI given the previously stated role of Taste+ in deciding between attitudinal possibilities and in adding diversity to thinking.
In some ways there is an odd cyclicality to this phenomenon, which is worth noting but is not an argument in and of itself. Back in Ancient Greece, by extension (as far as intellectual inspiration goes), the Renaissance had polymathic thinkers like Aristotle, who was a philosopher and the father of biology, and Leonardo da Vinci, who was a painter, engineer, and anatomist. So, with specialization being tasked away to AI, broader, more interdisciplinary fields like Naturalis Philosophiae or Quantitative Policymaking can come to the fore.
That being said, it’s worth also saying that the labs, individual enthusiasts, and private enterprises working on these tasks are likely to be smaller in their headcount due to the efficiency gains that AI will bring with it. To a certain extent, it can be a good thing, as it will mean a reduction in the over-allocation of talent towards the same select few industries and will push more people towards experimenting and innovation.[10] The reallocation in this context will involve more than just a drive towards entrepreneurship but will also involve diversification in the sectors where talent is concentrated beyond technology to also include sectors like education or transportation or welfare.
But this diversification is likely to only account for some job creation, which is likely to be insignificant in front of the greater unemployment that efficiency gains will create. So, I don’t think that everybody is going to come to the same level in exercising Taste+, and not enough people will be willing to take the plunge, leading to significant unemployment. This risks exacerbating further inequality, which is not a very favorable outcome for society as a whole. I think this is where ideas such as Universal Basic Income (UBI) and improving welfare support in countries are likely to come into play.[11]
In saying this, the important thing to note is that while inequality can rise greatly with artificial intelligence, I don’t believe the fundamental nature of this inequality is going to be different from the inequality we experience today. That is to say, I don't believe inequality is going to be a product of the absence of economic opportunities. It’s rather going to be a product of economic opportunities becoming more constrained and possibly more stochastic in who is able to gain the greatest advantage due to the uncertain nature of entrepreneurial and research breakthroughs. Welfare mechanisms need to be put in place to cushion everyone from its effects. But even in this, baseline inequalities in global GDPs are likely to significantly affect the extent to which this welfare cushion is available.
The positive belief I have with regard to global disparities and AI is that AI penetration is likely to correlate strongly with country GDP. Richer countries, with typically smaller populations, are likely to invest more in making AI mainstream; at the same time, their governments have the financial resources to provide support to their citizens through UBI or other welfare policies, who are also fewer in number. The effects of AI are likely to be far more pronounced in the Global South, but structural inefficiencies may mean that AI penetration occurs at a much slower rate, especially in informal economies, which make up a large proportion of these economies. This may mean that these countries have a longer time period enabling better planning from the learnings of the Global North and possibly time for some transfer of financial resources to these countries so that we come closer to the economic aspects of ensuring everyone has a somewhat equal right to life at least as far as economic aspects are concerned. Also, as time progresses with falling global fertility rates, we might find ourselves in a position where we need AI to sustain economic output; that said, this is a possible implication in the very long term.
Some Related but Adjacent Thoughts:
An Extention to the Idea of Initiative:
Two subjects that are increasingly being used as standard benchmarks for the performance of AI are math and programming. In math, especially with an OpenAI model disproving Erdős's Unit Distance Conjecture, there are a growing number of skeptics with regard to the future role of mathematicians. In programming tasks, as well, LLM-produced code has reached a very high degree of accuracy.[12] Some mathematicians continue to believe that it will only serve as an aid or as a tool for quick checking, allowing for greater collaboration, and I think this is certainly possible; a lot of, if not all, low-hanging mathematical fruits are likely to be solved autonomously as well as certain non-trivial mathematics, which by themselves are going to be a dent in the actual quantity of output of mathematicians. So, I think here as well initiative is going to fill the gaps of greater saturation of existing math, which is going to, in my opinion, lead to the creation of new mathematics. Mathematics has always been built on looking at new structures or investigating the possibilities of different assumptions, so, for example, we may investigate a new mathematics with an additional logical axiom. In this situation we are looking for new models of mathematics built of new assumptions, where Taste+ once again comes into play. I think the idea generalizes easily to the “hard sciences": computer science and theoretical physics, where perhaps one could conjecture a physical realm where pre-Galilean motion exists and whether this might have any actual applications. For programming-related tasks, I don’t directly see any parallel that might fit the pattern where models may be created, but I may be wrong.
Maybe a Precursor to Homo Deus:
Yuval Noah Hariri, in his book “Homo Deus,” talks about a future union between man and computer as man strives to achieve goals of superintelligence, artificial modifications, and longevity. Especially with the developments in synthetic biology and Brain-Computer Interfaces (BCI), that’s increasingly becoming a plausible future reality. But, with the AI revolution and as a method to preserve economic meaning, a possible method may leverage the ability of AI to become user-specific. The broad idea underlying this is that when a human is tasked with a job, it’s a given that the human and the AI that’s fine-tuned to their way of thinking, their workflows, etc. are deputized to the task. I think that these personalized AI bots, as opposed to centralized AI, are likely to come about as a result of user preferences, but I think it has some important positive implications for AI safety.
With social media, one of the factors that allowed companies to get away with highly addictive and polarizing features and content was the monopoly power of the companies. For effective regulation over AI, I believe we need a greater number of frontier labs that are actively competing. This I believe will push companies to distinguish themselves with more fine-tunable and possibly open-source models, which can greatly accelerate AI safety research. Further, I think this might also reduce inequality, as we won't be concentrating the power that the technology gives and the economic benefits to a select group. On other issues, like AI for malicious use, I believe, admittedly idealistically, that user beliefs can steer more companies to enact stronger safeguards to prevent malicious use, which may not be possible when users have very few options in the first place.
The Human Role in Preventing Model Failure:
A major reason behind the success of LLMs has been their ability to scale with greater data and improve their output substantially with greater data. That said, Shumailov et al. conclude that AI models that are trained on their output become increasingly uniform and lack the intrinsic diversity of human interaction. The authors in the paper emphasize that future model training must give greater importance to human prompting and human-bot interactions to enhance model performance as opposed to relying completely on data scraped from the internet, which might have a greater amount of AI-generated content in the future. Therefore, a possible way for humans to contribute economically is by interacting with bots for a diverse range of tasks and manually making photos and videos to ensure that LLMs preserve their quality for the majority of the users. This argument has not been presented as a major argument in justifying the economic utility of human workers, primarily because this argument only holds if the future of AI is only going to be unlocked with LLMs. While I believe LLMs will still continue to be an important tool in language tasks involving direct interaction with humans for the foreseeable future, I do believe better architectures will emerge for specific tasks like AI4Science or an architecture that has enhanced memory recall. I once again add the disclaimer that I don’t think this possible pathway of human economic contribution is likely to be sufficient to provide employment to the maximum human population of 10.3 billion people that are likely to inhabit the planet by the mid-2080s.
Another Argument Against a Gödel Machine and Taste+:
A Gödel machine is a problem-solving tool that makes use of self-referentiality to iteratively improve itself and get better at a task. These models consist of a layer that interacts with the user and a meta layer that finds a mathematical proof that a specific modification in the model architecture will improve performance. A modern variant of the model makes use of empirical observations as opposed to mathematical proofs.[13] For these models, the previous argument using Rice’s theorem applies, as the theorem implies that there will exist a class of improvements that are mathematically unprovable and therefore can’t be detected by the meta-layer of the machine. Even if empirical benchmarks are used as opposed to mathematical proofs, there are risks that machines will optimize for hacking the benchmark as opposed to genuine improvement. But a bigger concern is the computational complexity of the problem in the first place. Searching for proofs or empirical improvements of a random number of improvements is computationally challenging due to the sheer size of the subspace, and even if results are obtained, there are no guarantees that those improvements will be substantial enough to search the entire space. I believe the question of Taste+ comes here again not because I think human intuition is in anyway superior or more likely to excel at the task, but because some basic set of beliefs and assumptions needs to be made to guide the exploration. By the heterogeneity in human thought, these assumptions are likely to be different enough for a large number of people, thus increasing the likelihood of global progress towards the task even if there is no guarantee for individual success.
Conclusion:
Thinking about various issues related to AI has been an interesting intellectual exercise, but that said, I don't necessarily expect all or any of my predictions to be accurate. As is true for the dawn of any technology, the future is highly uncertain, and I think it may be an interesting idea to compare current uncertainty as reflected in media with that of previous revolutions, like with the discovery of the internet in the 1990s or the Industrial Revolution. My arguments, right from using Gödel's Incompleteness for AI, which may not qualify as a formal system, and the adjacent thoughts I mentioned are closer to a primer for an argument as opposed to a complete argument with a proper analysis of previous literature. That said, I do believe in the economic promise of incompleteness in AI. But this doesn't take away from the risk of high unemployment or AI wars or extreme inequality; the version of events I present are some possibilities and solutions I think are worth considering. Homo sapiens are a resilient species, and for the sake of our continued prosperity, I hope for this essay to serve as one of many models (as described in the essay) for the future, which is a worthy hypothesis to debate more than a completely accurate representation, which I don’t believe exists.
While the ARC-AGI does exist as a benchmark for AGI, I’m not sure whether adaptable learning is the only or best benchmark for AGI. Also, there are also valid arguments to be made that models that don’t perform very well on ARC AGI might be fully capable of replacing many human tasks, which is frequently seen as the primary result of AGI.
I use ‘likely’ in the absence of formal proofs that suggest their non-algorithmic nature as they exist in QFT. I think that this is reasonable, as even if some magical theory exists, it’s likely to be very complex and difficult, making models easier and more efficient options.
While I believe that there are certain processes like human consciousness that are non-computable, I won't go so far as to say that’s because of quantum processes in our brain. I believe certain parts of the argument do silently promulgate human exceptionalism, which I don't think is a universal truth.
This statement may seem contradictory, but it represents the fact that while AI is often able to suggest high-quality responses, the nature of those responses is remarkably similar across models, with methods by which it gets to those creative decisions often non-existent or partially explained.
What I mean by 'constructive’ is an area where a super AI’s abilities are unlikely to be significantly more advanced than those of a human.
While I’m not sure whether the first AGI model will completely finish this loop, it’s certainly possible for models that are more intrinsically curious by design, perhaps JEPA-type models which actively explore our world, like us, in their learning process.
Status Quo may not be the best word, but it’s used as a synonym for beliefs, opinions, and ideas purported by AI. Also, it’s worth clarifying that I don’t advocate disagreeing with the bot for disagreement’s sake but rather for bringing in diversity of thought.
In this specific instance, the idea of finding generalization is an example of Taste+Initiaitve and the deliberate choice of maximizing one principle over the other can be considered an example of Taste+Selection. Henceforth, Taste+ will be used as an overarching term to refer to these corollaries as well.
To openly disclose my own conflict of interest, I hope to pursue a career in research, so I’m naturally biased towards believing that’s the future.
There are already some indications of this trend that’s coming to the fore with greater venture capital investment in AI, as opposed to continuing with more conventional wealth redistributive roles in finance, law, or consulting.
This article doesn't attempt to examine the validity or complete feasibility of UBI. There are far more detailed studies on the subject by people who have thought about it far more than me. I introduce UBI as something to talk about intercountry differences in economic capital to deal with disruption due to AI.
The only caveat to this point is that some people may feel that AI proofs are not as clear in motivating the solution to the problem or that AI code is not very readable. While these are certainly true in the current context, I’m not sure whether these are problems that can’t be solved with fine-tuning.
Gödel Darwin Machines, by looking at empirical improvement as the only benchmark, there are grave safety concerns, as it risks further reducing model interpretability.