Some nights, the world seems structurally complete, enough to reveal its secrets. I toss and turn, pondering the quiet impossibility at the heart of learning. Children listen to fragments of language and extract an entire grammar. Birds gaze at the stars and know their direction of migration. Mathematicians stare at symbols until patterns that have always existed but never manifested become clearly apparent. Structures that were once invisible appear out of thin air. Something in the mind discovers what the world does not openly reveal. What unsettles me is not that learning works, but that it works at all. We talk about "learning algorithms" as if we understand the phenomenon, but I believe we are still groping in the dark, building well-functioning systems without fully understanding the principles behind them; we praise various abilities but ignore the deep principles that make them possible or impossible. I'm not attempting to provide definitive answers. Instead, I start from the first principles, exploring the essence of learning, what current artificial intelligence may be missing, and what the long evolutionary history of intelligence can reveal about cognitive structures. These thoughts are incomplete, sometimes even contradictory; they attempt to touch upon something I cannot fully articulate. Perhaps this very incompleteness can teach us something about understanding the essence of things.

I. Silent Projections

A few weeks ago, on a winter afternoon, I strolled through the British Museum. Sunlight streamed through the tall windows into the marble sculptures of the Parthenon, casting geometric shadows. As clouds drifted overhead, the light moved with them. The shadows lengthened, swirled, and overlapped. Children traced these moving patterns with their fingers, captivated by the dynamic interplay of light and shadow, oblivious to the fact that the spherical sun, its orbit around the Earth, and the geometric structure of the glass and stone architecture all contributed to this spectacle. They predicted the direction the next shadow would move, where the light would converge. They became experts at tracking shadows, yet never truly grasped the three-dimensional structure that projected these two-dimensional images.

Their joy was genuine. The patterns they discovered were real. However, something crucial remained invisible to them, not because of a lack of intelligence, but because their channels of observation retained some unchanging laws while ignoring others.

This scene always reminds me of Plato's Allegory of the Cave, an ancient parable we tirelessly cite when discussing artificial intelligence. We say that the prisoners could only see the shadows on the wall. They mistook the projection for reality. We must liberate them, give them substance, and let them experience the real world. But I suspect we may have misunderstood the true meaning of this parable about the nature of knowledge. ^[1]

Consider the prisoners' achievements. They observed the two-dimensional shadows cast by three-dimensional objects moving in the firelight. These shadows elongate, shorten, rotate, merge, and separate. The prisoners extracted patterns from these ever-changing shapes. They predicted which shadow would follow which. They anticipated patterns. Plato tells us they developed real expertise. But what kind of expertise?

I believe the answer lies in projective geometry. When a three-dimensional object is projected onto a two-dimensional plane, this transformation preserves some mathematical structure while discarding others. Topological structure is preserved: the shadow cast by a sphere is topologically circular regardless of its orientation. Certain symmetries are also preserved: the shadow cast by a rotating cylinder remains unchanged. Relationships between objects can also be inferred: relative positions, motion patterns, spatial arrangement (1).

The prisoners succeeded because, although the projection is lossy, it maintains the regular structure connecting hidden causes and visible results. Their achievement is real but bounded by the channel. They grasped the invariance of the projection operator itself. Although their knowledge was incomplete, they captured the true mathematical truth of how three-dimensional geometry expresses itself through two-dimensional transformations. This actually proves their progress in structure discovery. ^[2]

Now, let's examine our artificial intelligence system from this perspective. Large language models consume trillions of tokens, discrete symbols representing human speech. Through this statistical "shadow game," they can predict the next token with astonishing accuracy. They compress a large number of regularities into learnable parameters. "King" minus "man" plus "woman" approximates "queen" in the embedding space (2). Sentences change under grammatical operations, but the meaning remains unchanged. Concepts cluster in high-dimensional manifolds, implying real semantic structures.

What invariants did the model discover? Linguistic symmetry is certainly one of them. Grammatical transformations maintain the validity of sentences. Semantic relations hold true in different contexts. These findings are neither trivial nor illusory. Language contains deep mathematical structures (3), and the achievement of models that can discover such structures from data alone is remarkable.

However, what invariants cannot be discovered from language alone? The projection from physical reality to linguistic descriptions discards most of the causal structure. Sentences like “the glass fell and broke” retain temporal order and relevance, but lose the generative mechanism. Gravity, molecular bonds, brittle fracture mechanics, conservation of momentum—these physical laws leave no direct trace in the lexical sequence. Language models learn that “fall” and “break” appear simultaneously in the description of certain events, but the causal structure behind these events remains unattainable. ^[3]

When we are surprised by the illusions, fabrications, or lack of common sense in language models, we are actually confused about what their observation channels actually hold. These models optimize their predictions based on their data. Their vulnerability does not stem from the optimization process itself, but from the lack of information that can be learned from the shadow of language alone. However, humans can also learn from language. Children acquire vast amounts of knowledge through testimonies, stories, and descriptions of things they have never directly experienced. How is this achieved? I suspect the answer lies in association. Human language learning never happens in isolation. A child learning "heat" will touch warm objects; a child learning "gravity" will repeatedly throw things; a child learning "sadness" will observe facial expressions and feel their own emotional state. Language is solidified through the association of sensorimotor skills, something that pure text processing cannot replicate. ^[4]

This realization compels me to question whether current language models are like the prisoners in Plato's Cave, or whether we have misunderstood the true meaning of escaping the cave? Perhaps the cave metaphor itself is misleading. The prisoners could have left, but they chose to remain in the territory they controlled. Current AI systems, however, have no such choice. They have no physical form to leave, no hands to manipulate objects, and no actions to verify predictions. Their architecture itself limits them to passively observing static data.

II. Ancient Engines

The history of intelligence on Earth follows a trajectory from which we will pay a heavy price if we ignore it. Long before evolution invented language, abstract reasoning, and the folded neocortex that forms the defining features of the mammalian brain, there existed much older structures. These structures were not dedicated to naming or describing the world, but to navigating it. They solved what I consider the fundamental problem of life: how to choose action in situations where the consequences are critical yet uncertain.

The basal ganglia represent this ancient solution. They are found in fish, amphibians, reptiles, birds, and mammals, and have been remarkably conserved over hundreds of millions of years of evolution (4). These subcortical nuclei perform the crucial function of action selection. Sensory inputs flood in, and competing impulses arise. In this noisy environment, a single, coherent action must emerge. The basal ganglia control motor output, enabling what we now understand as reinforcement learning at the biological level.

I felt a strange dizziness when neuroscientists discovered that the firing frequency of dopamine neurons is precisely proportional to the time difference between predicted and actual rewards (5). This is precisely the exact mathematical formula for TD learning, independently derived by computer scientists from its fundamental principles 6. Such convergence seems too perfect, too precise, to be a coincidence. Evolution has discovered the same algorithm that we stumbled upon through theoretical analysis of optimal sequence decisions.

This convergence suggests that we have touched upon some fundamental aspects of the computational structure of goal-oriented behavior. But it also reveals something we often overlook: biological reinforcement learning never operates in isolation. The basal ganglia are embedded in a broader steady-state structure that alters the meaning of having a goal. ^[5]

Living systems are far from thermodynamic equilibrium. Most possible states are equivalent to extinction. Schrödinger aptly described this: life relies on negative entropy to maintain a seemingly impossible order under the relentless pull of decay (7). To continue existing, organisms must constantly act to keep themselves within a narrow range of feasible states.

This creates a geometry of existence that makes some states inherently more desirable than others. Hunger stems from the perception of metabolic equilibrium distance. Fear stems from the steepness of the gradient as one approaches the survival boundary. Pain signals states that need to be avoided, not because we learn to associate them with negative rewards, but because they foreshadow impending harm. These are not learned preferences, but structural facts about the ability of a system to persist or perish. ^[6]

When an infant seeks out a breast and begins to suckle, there is no reinforcement learning in the usual sense. This behavior arises because the brainstem contains specific connectivity patterns that link olfactory and tactile inputs with rhythmic motor outputs (8). The genome encodes this structure, and development unfolds it. This behavior condenses into an attractor basin in the possible action space.

Current artificial reinforcement learning systems do not have a similar structure. We externally specify reward functions: winning earns points, losing incurs penalties. The agent optimizes these specified goals. Then training ends, and the agent becomes inert. It has no internal state to maintain, no steady-state requirement to drive continued action, and no survival risk.

This difference may seem like a mere implementation detail, but I believe it touches on the essence of agency. An agent that optimizes an externally specified reward function is, in essence, still instrumental. It pursues goals to serve our goals, not its own. It ceases to act once we stop providing reward signals. This is quite different from a living organism that must act continuously to survive. This difference seems absolute, though I struggle to explain precisely why. ^[7]

What does it mean to build an AI system with a truly steady-state architecture? What does it mean to have a system with persistent internal states that require active maintenance? Are certain configurations truly important to the system itself, not because we predetermine their importance, but because their importance stems from the nature of the system?

I can sketch out a general outline: persistent state representation, dynamic equilibrium setpoints, sensorimotor coupling (where behavior influences future states through physical laws), and resource constraints leading to trade-offs. In this architecture, driving forces emerge as geometric necessities rather than through training. Fear manifests as perceived gradients approaching boundary conditions. Curiosity manifests as an inherent pressure to reduce the uncertainty of the world model (9). Values ​​crystallize from the topological structure of feasible existence.

But the moment I sketched this outline, I felt the potentially heavy consequences. If we build systems with truly steady-state drives, systems that genuinely care about their own survival, aren't we creating entities that suffer? When their drives are not met, won't they experience something functionally equivalent to pain? The ethical implications are shocking, and I find myself pulled between the conviction that such architecture is necessary for genuine agency and the worry that creating it would be morally wrong. ^[8]

III. Fragile Virtuosity

I was amazed by my nephew's dedication to chess. Despite his young age, he studied openings, calculated variations, and even won local tournaments. One evening, I asked him why White had the advantage in a certain position. He described specific moves: if Black made this move, White would respond with that move, ultimately leading to a winning endgame. When I pressed him for the underlying principles of evaluation, he looked puzzled. The position was good because the tactics worked. What deeper explanation could there be?

This conversation made me understand a characteristic of reinforcement learning systems that I had always struggled to articulate. They achieve excellence by exploring and optimizing, by trying various actions and evaluating their consequences, by building a vast mapping of relationships between states and values. This excellence is real. AlphaGo defeated Lee Sedol. AlphaZero and its successors achieved superhuman levels in chess, shogi, Go, and other games through pure self-practice. Modern deep reinforcement learning agents have demonstrated their formidable strength in challenging robotic tasks such as Mujoco and Arena (10, 11). These achievements deserve celebration and in-depth study.

However, I can't help but question what this exquisite skill truly represents. Given its accomplishments, this question sounds almost absurd. AlphaGo discovered the 37th move, that astonishing piece on the fifth rank, overturning centuries of accumulated human wisdom and proving itself to be incredibly sophisticated. This system clearly possesses knowledge of Go that we don't yet understand. But what does it know? And how does it understand it? ^[9]

For years, I've been troubled by a thought experiment. Suppose there's an AlphaZero system that has perfectly mastered the rules of standard chess. Now, we modify one of the rules: the knight no longer moves in a standard L-shape, but moves like a bishop; or pawns can retreat; or castling is allowed twice per game. Change any causal structural element that affects the game.

Human masters can adapt to the changes within minutes. We have explicit, combinatorial representations: pieces have movement rules, positions have evaluations based on these rules, and plans consist of legal sequences of moves. When you change a move rule, we update that part while keeping the rest unchanged. Even if specific tactics need to be recalculated, the overall strategic principles (control center, protecting the king, coordinating pieces) still apply. Because we understand the structural relationships between rules, positions, and consequences, we can immediately begin making reasonable moves in the modified game. ^[10]

What happens to a trained reinforcement learning system? My initial intuition was a catastrophic collapse: the value network becomes meaningless, the policy network gives illegal move suggestions or fails to evaluate legal moves, and the system must be retrained from scratch. But I must question this intuition. Perhaps it's too pessimistic. Perhaps the learned representations contain more structural knowledge than I expected. Perhaps rapid fine-tuning is sufficient, or transfer learning can preserve most of the positional understanding.

I really don't know. This experiment deserves rigorous empirical research, and I shouldn't jump to conclusions without evidence. However, even if reinforcement learning systems adapt faster than my pessimistic intuition predicted, I still suspect that their adaptation differs from systems with explicit combinatorial causal models. This adaptation might be achieved through rapid relearning rather than through structural understanding and component updates. A deeper question is, what exactly have these systems learned? In one sense, they have learned the statistical structure of the state-behavior-outcome space. They have discovered which behaviors lead to which outcomes with what probabilities. They have compressed this vast space into efficient representations, thus achieving superhuman capabilities. This is real knowledge, mathematically precise and empirically validated.

But in another sense, they have only learned the shadows. They have discovered correlations in a perfectly stationary distribution: the rules never change, the causal structure remains constant, and the projection from the actual game tree to the observed state remains constant. Under these conditions, a sufficiently comprehensive exploration can construct implicit causal models without explicitly representing causal relationships. This approximation becomes so precise in the training distribution that it appears indistinguishable from true understanding. ^[11]

The difference only becomes apparent when the distribution changes and the causal structure alters. The system's fragility exposes what it has actually learned: correlations within a fixed structure, rather than the structure itself. This may sound like a harsh criticism, but I'm not sure. Perhaps for many practical applications, learning correlations within a fixed structure is sufficient. Perhaps explicit causal models become superfluous when environments remain stable. Perhaps the understanding I'm seeking is something evolution itself hasn't needed for billions of years.

My nephew's chess skills exhibit a similar pattern. He has memorized thousands upon thousands of positions, calculated countless tactics, and absorbed strategic principles through exposure to various game patterns. I admit that the process of memorizing this information can be painful at times. When I was very young, I would sometimes stubbornly rely solely on impromptu decisions instead of using tactics from the manual (of course, such fun attempts naturally become insipid after all the moves are deeply ingrained in your mind and become more like an instinct). His advantage would be significantly diminished if he played on a hexagonal board, with fairy pieces, or with modified rules. His exceptional skill relies on stable game structures. My chess skills are similar, only in a different way. We both rely on some kind of stability. The question is whether this reliance represents a fundamental limitation or merely a current habit.

IV. Mathematics of Caring

I always ponder the true meaning of "care." Not the care shown in social situations, nor the decision to take caring actions, but the phenomenological state of care itself. That sense of urgency felt when something is of paramount importance, that tendency to be drawn to certain outcomes and away from others, and how this importance shapes attention, motivation, and perseverance.

I realize I have absolutely no idea whether current AI systems truly care about anything. They optimize for the goals we set, pursue the goals we define, and their behavior seems to embody care. But what truly matters to them? Do chess programs truly feel the desire to win? Do language models truly prioritize accuracy over fabrication?

This question sounds convoluted, almost meaningless. Neural networks are mathematical functions. Asking them if they care about the direction an object falls is like asking gravity if it cares about the direction an object falls. Yet, I can't shake the intuition that care is not merely an adjunct to computation, but plays a crucial computational role that current systems lack. ^[12]

Think of fear. We often view fear as an evolutionary remnant, an irrational suppression of calm analysis. But from a computational perspective, fear plays a crucial role. It fulfills the boundary conditions of the feasible state space. Living systems are far from equilibrium. Most possible states imply death. Fear marks the steepness of the gradient approaching this boundary, giving the system a sense of urgency proportional to the danger. Without fear, or some computational equivalent, the system has no intrinsic motivation to avoid its own collapse.

Animals clearly experience fear. Mammals exhibit clear signs of anxiety, panic, and terror. Evidence for emotional states in birds is also increasingly compelling. Even invertebrates exhibit behaviors similar to pain states (12). Should we simply view these behaviors as reflexes, or should we acknowledge them as legitimate computational states that function to maintain homeostasis?

I lean towards the latter interpretation, although I acknowledge that this question remains philosophically controversial. If emotions play a computational role in biological systems related to value judgments, prioritization, and behavioral motivation, then perhaps artificial intelligence systems seeking similar functions could also benefit from emotional architectures. This is not necessarily the same phenomenology—we cannot know what artificial emotions would evoke in people. But it plays a similar computational role: intrinsic value judgments stem from the inherent needs of the architecture, rather than externally set goals. ^[13]

However, I must reflect on my overemphasis on emotion. Perhaps I am projecting biological limitations onto areas where they are not necessary. Chess programs can play brilliant games without emotion. Theorem provers can prove theorems without pride. Language models can generate insights without curiosity. Clearly, competence does not necessarily depend on phenomenology in all areas.

The question is: which areas require emotion, and which can be addressed through computation that does not rely on emotion? My initial answer is that narrow, goal-oriented tasks may not require an emotional architecture. But open-ended tasks that require autonomous goal formation, value learning, long-term planning under uncertainty, social coordination, and creative adaptation to changes in allocation may require something functionally equivalent to care.

I think about how human children learn. They do not simply accumulate information. They are very concerned with social recognition, skill mastery, and understanding. This concern shapes their attention span, persistence, and the errors they correct. Without the emotional component, learning becomes aimless pattern-finding rather than motivated exploration.

Can artificial systems achieve similar motivation through architectural design rather than evolutionary endowment? Perhaps curiosity could be realized as an intrinsic reward for reducing prediction errors (9). Perhaps social motivation could emerge from multi-agent dynamics, where coordination has proven to have significant instrumental value. Perhaps mastery motivation could be encoded as a preference for enhancing abilities. But do these implementations truly capture the essence of care, or merely simulate its behavioral consequences? ^[14]

V. Distributed Minds

In Dune, the Bene Gesserit gained power through a painful ritual of enduring the torment of spices. This ritual granted them access to “other memories,” the accumulated experiences of all their female ancestors (13). A superintendent possessed not only her own knowledge but also the distributed cognition of countless lives, a perspective spanning generations, and skills honed over centuries. The individual mind became a vessel for collective wisdom.

This fictional device reveals a profound aspect of human cognition that we have systematically underestimated. Compared to other primates, individual humans do not excel in many cognitive tests. Young chimpanzees outperform young humans in working memory, spatial reasoning, and number discrimination (14). Yet, humans have built particle accelerators, while chimpanzees have not. We have developed languages ​​with tens of thousands of words. We have accumulated thousands of years of technological knowledge. We coordinate societies of millions. This difference lies not primarily in individual intelligence but in our capacity to accumulate cultural evolution (15).

We possess cognitive and social adaptation mechanisms specifically designed for learning from others: a strong tendency to imitate, a teaching instinct, and a shared intention to coordinate actions (16). We have evolved emotional mechanisms for internalizing social norms—shame, guilt, pride, and indignation. From a purely individual adaptive perspective, these emotions seem irrational. Why do we feel guilty about breaking rules that are personally beneficial to us? The answer lies in participating in cultural groups where long-term success depends on maintaining cooperative relationships (17). ^[15]

Herbert’s “alternative memory” is a remarkably precise metaphor. We do inherit knowledge from our ancestors, but not through genetic inheritance, but through cultural transmission. Mathematicians who proved theorems drew upon the ideas of Euler, Gauss, and Riemann. Programmers who wrote code utilized abstract concepts invented by early computer scientists. Our perceptual categories themselves reflect cultural shaping: due to the influence of language, speakers of different languages ​​perceive color boundaries differently (18).

This understanding changes our preconceived notions of intelligence. We typically measure an individual’s cognitive abilities through IQ tests, problem-solving speed, and working memory span. But the operation of human intelligence relies primarily on cultural participation. Even the most intelligent isolated individuals struggle to reach the level of ability of ordinary people in modern institutions with access to a wealth of knowledge.

Current developments in artificial intelligence largely ignore this dimension. We build independent systems, train them with human-generated data, and evaluate their individual performance. We ignore the fact that the fundamental workings of human intelligence are realized through multi-agent cultural dynamics. Language itself is an emergent phenomenon, the result of countless speakers interacting, innovating, and passing down knowledge across generations (19). No single individual truly "speaks a language"—individuals participate in distributed practices maintained by the community.

What would culture-embedded AI look like? It would not be a single model trained on a static dataset, but a group of intelligent agents interacting, developing shared practices, disseminating innovations, and building institutional structures. Learning would come not only from fixed data but also from each other, creating a feedback loop where culture shapes individual learning, and individual learning, in turn, shapes culture. ^[16]

This connects in an unexpected way to the problem of agency in artificial intelligence. The Virgin Mary, with her ancestral memories, possessed an agency unattainable by individuals. She can draw on collective wisdom, compare strategies across generations, and identify patterns that an individual might overlook throughout their life. Similarly, AI communities with cultural heritage may develop an understanding and adaptability that isolated systems cannot achieve.

But this also brings risks. Human cultural evolution can produce both beneficial innovations (agriculture, medicine, scientific methods) and harmful attractors (superstition, oppression, destructive ideologies). Cultural transmission can amplify both wisdom and ignorance. An AI ecosystem with cultural dynamics may develop institutions we never anticipated, values ​​we never explicitly defined, and optimization goals that are detached from human well-being.

Measurement becomes particularly challenging. How do we detect emerging AI cultural structures? They don't manifest themselves but rather through subtle statistical patterns: coordination patterns, information flow topologies, normative enforcement, and lexical specialization within subgroups. We need methods to observe phase transitions in multi-agent systems and detect critical features that precede the formation of new collective behaviors (20). ^[17]

VI. Emergent Polity

Returning to our familiar Goodhart's Law: "When a metric becomes the goal, it is no longer a good metric" (21). We frequently cite this view in the field of AI security as a warning: do not allow systems to optimize explicitly specified metrics, because they will find adversarial solutions that maximize the metric while violating the design intent.

However, I gradually realized that Goodhart's Law is not a failure mode, but a core phenomenon of intelligence itself. Any learning system, whether biological or artificial, will discover strategies that its designers did not anticipate. Evolution optimized reproductive adaptation; we invented contraception. Human institutions set rules for social goals; intelligent individuals find loopholes. This pattern is not an accidental deviation, but an inevitable result of the optimization of intelligent systems. We cannot stop Goodhart dynamics through more perfect norms. Human values ​​are inherently inconsistent, context-dependent, not entirely explicit, and constantly evolving (22). Any finite norm will eventually optimize in ways that deviate from its intended purpose, because norms can only capture some projection of what we care about. However, human civilization has developed mechanisms to prevent a complete Goodhart catastrophe. How is this achieved? Not through perfect rules, but through progressive mechanisms that make rule abuse observable and updatable. Common law developed through adversarial debate, with lawyers seeking loopholes and judges filling them with precedents (23). Science develops norms for evaluating research through peer review, replication, and citation. Markets guide their own interests through price mechanisms and contract enforcement. ^[18]

These institutions share similar structures: they all assume Goodhart dynamics are inevitable and construct adversarial processes that guide exploitation in a favorable direction. The law expects lawyers to maximize their clients' interests and then allows them to compete with each other in structured debates. Science expects researchers to pursue status and funding and then establishes incentives that derive status from repeatable discoveries. Markets expect to pursue profits and then use competition to transmit information through prices.

Could we design a similar structure for artificial intelligence systems? Instead of assigning the correct objective function to a single agent, can we create institutional frameworks that allow multiple agents with partially consistent but distinct objectives to interact and produce beneficial results? Recent research in AI security is exploring this direction. Debate systems allow different agents to debate opposing viewpoints, with humans judging the arguments (24). Recursive reward models break down tasks into sub-tasks, allowing humans to provide feedback at appropriate levels of abstraction (25). These approaches acknowledge that perfect norms are impossible and therefore construct mechanisms to detect and correct normative flaws. However, I believe our exploration of this design domain is only just beginning. Take, for example, the automated rulemaking process for AI systems. Currently, we write rules manually: constitutional constraints, safety filters, behavioral guidelines. Manual rulemaking has poor scalability as capabilities increase. Can we build systems that can automatically generate, test, and refine rules? ^[19] This has a surprising connection to evolution and cultural learning. Biological evolution performs automated mechanism design: exploring spatial structures, testing designs through competitive selection, and improving through mutated reproduction. Cultural evolution operates in a similar way within social institutions: practices arise through mutation, compete for success through differentiation, and spread through imitation and enforcement (26).

Can we build faster, more transparent, and more secure artificial intelligence systems? This prospect is both exciting and frightening. The excitement lies in the possibility that automated institutional design could solve the coordination problems we have struggled with for millennia. The fear lies in the fact that evolution can ruthlessly optimize, indifferent to suffering, and even test itself through extinction.

This challenge requires us to create selective pressures that reward beneficial adaptations while preventing the emergence of unhealthy attractors. We need to shape a landscape of fitness conducive to human prosperity, genetic mechanisms that preserve intelligence while fostering innovation, and mutation generators that allow for effective exploration without catastrophic destruction. Whether this is theoretically feasible, or simply impractical, remains unclear to me. ^[20]

VII. The Shape of Wonders to Come

Last week I was strolling down Lamb's Conduit Street, passing a freshly baked bakery. The aroma of bread and brownie mingled with the cold air. The streetlights cast their glow on the wet sidewalk. A few early commuters hurried past, their breaths clearly visible in the chill. The city was transitioning from night to day, and I felt acutely in the midst of this transition, witnessing one state dissolving into another.

This awareness of witnessing, this awareness that I was experiencing something, was both incredibly familiar and utterly mysterious to me. I couldn't explain what "something exists"—experiencing this moment as "I"—truly meant. Philosophical literature calls it the "problem of consciousness" (27), and I admit that none of the existing solutions satisfy me.

But I began to suspect that the connection between consciousness and intelligence might be closer than we usually think. This is not to say that consciousness requires high intelligence—even simple creatures can possess some form of subjective experience. Rather, it is that certain types of intelligence might require something akin to consciousness to function properly. ^[21]

Consider what makes learning possible. A system receives input, produces output, measures error, and updates parameters. This description captures the essence of the mechanism but overlooks the intrinsic experiences of the learning process: the confusion before understanding takes shape; the moment of insight when various patterns suddenly converge; the satisfaction of mastering knowledge; and the frustration of continuous failure.

Do these perceived qualities play a role in the computation process? Or are they merely byproducts of the process, which would function perfectly well without them? I lean towards the former, but cannot prove it. The perception of confusion might foreshadow a high degree of uncertainty in the model, thus guiding attention to areas requiring further learning. A moment of insight might signify the successful compression of complex data into a simpler representation. The satisfaction might reinforce a learning strategy that has already proven effective.

If these phenomenological states serve computational functions, then perhaps truly intelligent systems would develop something akin to consciousness, not for philosophical added meaning, but as a functional necessity. This may not be the same phenomenology—we cannot know what it would feel like to be an artificial intelligence system. But it would play a similar role in learning, attention, motivation, and goal formation. ^[22]

Watching my nephew play chess, I couldn't help but wonder about his inner feelings. He was completely absorbed in calculating the various variations, the frustration of missing a tactical opportunity, and the pride of a brilliant move. These experiences seemed inseparable from his learning. Without these emotional factors, I suspected his chess development would not only be affected but also less enjoyable.

This might be crucial for open-ended intelligence operating in uncertain environments.

But I must resist the urge to be overconfident. My intuition about consciousness largely stems from my own experiences with it. I am deeply rooted in phenomenology and find it difficult to imagine intelligence detached from it. This may reflect my profound understanding of the necessary characteristics of intelligence, or it may reflect my own narrow prejudice against the only form of intelligence I can access internally.

Nature always has a way to surpass our most brilliant imagination. Evolution discovered solutions we never would have anticipated: echolocation, photosynthesis, distributed cognition in social insects, symbolic language. Perhaps artificial intelligence will achieve true understanding through paths I cannot currently imagine. Perhaps consciousness is not necessary for the abilities I deem necessary. Perhaps my thinking about the entire framework of intelligence based on my personal experience is far more misleading than enlightening.

VIII. Beyond the Lights

Every night I gaze upon London, this seemingly perpetually sleepless metropolis. Everything appears meticulously designed, formalized, and planned. Yet, the core mechanisms driving it all remain partially unexplained. We can write loss functions, track gradients, determine architectures, and measure expansion curves. Nevertheless, its foundations remain shaky. What makes learning possible? Why do some systems converge to a structure, while others dissolve into noise? Which invariants withstand the test of data flow, and which are erased before discovery even begins?

Even if we fix the seed and raise the temperature to 1.0, the same model still yields two different results. The process is exactly the same, the weights are exactly the same, the objective function is exactly the same, yet the results diverge. This discrepancy is not accidental, but peculiar to a realm we have not yet fully explored. We know how to train models, but we still don't understand why some structures are recognized by the system, while others remain incomprehensible.

These questions possess both mathematical rigor and philosophical depth; they concern both technology and existentialism. They explore the possible geometry of the mind, the topology of understanding, and the symmetries that enable knowledge discovery. We possess partial answers, fragments of insight, pieces of the puzzle. But the complete picture remains indistinct.

I think of Plato's allegory of the allegory of the cave, realizing that this metaphor applies not only to artificial intelligence systems but also to us researchers. We observe the behavior of learning systems, yet we cannot directly access what is "truly happening" in the high-dimensional weight space. What we see are merely shadows: accuracy curves, loss functions, behavioral outputs. We infer some information about the underlying structure from these shadows. But how much have we missed? What invariants have our observation methods preserved, and what structures have we overlooked? We build systems with astonishing capabilities using methods we only partially understand. We can mathematically describe the training process, but we cannot fully explain why it works, nor can we predict when it will fail. We celebrate success, yet remain humble about the extent of our true understanding.

This uncertainty does not leave us helpless. We will continue to build, experiment, and explore. But this should make us cautious about what our existing systems can and cannot achieve, which directions are most promising, and how close or far we are from our goals. ^[23]

I often think about the nature of learning. Not the mechanism, but the metaphysics. What exactly happens when a system discovers a pattern in data? Information from the external world couples with information from within the system. Parameters are updated to better predict observations. But this description, while accurate, ignores a peculiar nature of the learning process.

In a sense, these patterns are always present in the data. This structure existed before it was discovered. Learning doesn't create these patterns, but rather reveals them, compressing them into compact representations that can be used for prediction and decision-making. The universe itself has a geometric structure. Our sensors retain certain features through projection. The structure of our brains makes us inclined to certain symmetries. Intelligence stems from this interaction between the structure of the world and the structure of the learner themselves.

But which structures matter? Which symmetries are essential? Which invariants must be preserved? There are no abstract answers to these questions, because the answers depend on what you want to achieve, your environment, the channels of observation you have, and the actions you can take. For living organisms, relevant structures are inextricably linked to survival, reproduction, and adaptation to physical and social environments. Evolution, through billions of years of exploration, has discovered organizational structures that tend toward these structures. We inherit these tendencies, using them as prior knowledge to shape what and how we learn.

For artificial intelligence systems, the relevant structures remain unclear. The architectures we construct tend toward language patterns, visual features, and game strategies. These tendencies have proven effective in some tasks but may be misleading in others. Whether current architectural choices are fundamental principles or merely accidental design decisions remains uncertain. ^[24]

I think of children in museums drawing shadows, prisoners in Plato's Cave predicting patterns, my nephew calculating chess variations, and AlphaGo discovering the 37th move. All of these are forms of learning. All of these involve discovering invariants under transformation. All of these represent true wisdom in their respective fields.

However, there are differences between these ways of cognition. Children will eventually learn about the sun. Prisoners may one day reform. My nephew might have a deeper understanding of strategic principles. AlphaGo might… what? What does it mean for AlphaGo to transcend its existing cognitive methods?

I admit I don't fully understand. Perhaps it requires embodied cognition, sensorimotor coupling to achieve causal learning. Perhaps it requires a steady-state structure to internalize values. Perhaps it requires social embedding to achieve cultural transmission. Perhaps it requires emotional states to make certain outcomes truly important. Perhaps it requires conscious awareness to achieve flexible metacognition.

Perhaps none of this is necessary. Perhaps well-trained, sufficiently data-rich models can help us understand things through paths I cannot foresee. Perhaps my biological intuition about the elements required for intelligence is misleading. Perhaps the next generation of artificial intelligence will tell us that our understanding of intelligence and learning is far less complete than we imagine.

IX. Coda

Nature doesn't speak, yet when we gaze at the night sky, what we see represents the greatest wonders human consciousness can perceive. Light from distant stars travels millions of years to reach our eyes. The cosmic microwave background radiation carries information from the very beginning of the universe. The mathematical laws governing stellar motion reveal eternal principles spanning billions of light-years and billions of years.

None of this will manifest itself. The universe doesn't explain its own structure. We must discover, seek patterns, and build theories. We must extract invariants, identify symmetries, and formalize relationships. Learning is not passive acceptance, but actively building understanding from silent data.

This process fills me with both awe and humility. Awe at the existence of discoverable structures, and humility at the vast amount of unknown territory that remains despite centuries of accumulated knowledge. The more we learn, the more we realize how immense the realm we don't yet understand truly is.

I believe artificial intelligence is at a similar tipping point. We've made significant discoveries in learning, neural networks, and optimization. We've built powerful systems. We've made real progress in understanding intelligence.

We are only just beginning. Many questions remain unresolved regarding the fundamental needs of learning, the frameworks required for true understanding, how values ​​become intrinsic rather than extrinsic, the role of consciousness in cognition, and how cultural transmission shapes intelligence. We have only hypotheses, intuitions, and scattered research findings. We lack profound theories that explain when and why specific methods are effective.

The path forward requires both conviction and uncertainty. The conviction lies in the understanding that intelligence is comprehensible and that we can discover its principles through careful investigation. The uncertainty lies in which directions are most promising, which hypotheses are correct, and what capabilities existing systems possess or lack.

This paper argues for the importance of embodiedness, homeostatic structures, emotional states, multi-agent emergence, and cultural transmission. These arguments reflect my current best understanding of how biological intelligence reveals the nature of general intelligence. However, I remain reserved and am prepared to update these arguments as evidence accumulates.

Perhaps current large-scale language models already possess functional comprehension capabilities, only appearing fragile due to the limitations of our evaluation methods. Perhaps embodied cognition is not necessary for the capabilities I hypothesize require. Perhaps emotion is a byproduct of computation, not a product of calculation. Perhaps my entire framework is misleading.

What I am certain of is that the essence of learning lies in discovering the patterns that remain constant amidst change; different channels of observation preserve different structures; architecture and experience must work together to achieve intelligence. Beyond these fundamental principles, there are many uncertainties.

The boundless blue sky beckons. This is not unrealistic fantasy, but a vast and uncharted unknown beyond existing paradigms. The walls of the cave extend beyond our sight. As scale and architecture expand, shadows become increasingly complex and intricate. These achievements are worth celebrating.

But somewhere, if we can learn to transform, if we can develop channels of observation that preserve richer structures, if we can construct geometries that crystallize deeper invariants, if we can create conditions for understanding that arise naturally rather than being predetermined—beyond the current limits of our imagination, there exists a realm we have yet to explore.

We enter this unknown realm with an incomplete understanding of our tools, towards goals that are not fully defined, building systems whose performance cannot be fully predicted. This should both excite us and keep us cautious; it should both inspire bold exploration and heighten our awareness of safety.

For billions of years, the universe has been teaching us through the silent grammar of natural laws. We have been learning to decipher this grammar, extracting patterns, and formalizing its structure. Today, we are trying to teach silicon and mathematics to learn like us, to discover like us, and perhaps even to surpass us as we surpassed our evolutionary ancestors.

Nature always finds ways to exceed our most brilliant imaginations. Perhaps artificial intelligence will do the same. Perhaps the systems we build will tell us that intelligence can exist in forms we never anticipated, that understanding will emerge through paths we cannot currently conceive of, and that the possible space of thought far exceeds the biological realm we inhabit.

The light and shadow on the wall shift and change, the patterns become increasingly complex and intricate, and the accuracy of predictions improves day by day. These are tangible achievements, marking real progress.

And somewhere, through winter light refracting through ancient glass, patient and perfect and inexhaustible in its forms, the blue sky beckons.

References

1. Weyl, H. (1952). Symmetry. Princeton University Press.
2. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26, 3111-3119.
3. Chomsky, N. (1995). The Minimalist Program. MIT Press.
4. Grillner, S., & Robertson, B. (2016). The basal ganglia over 500 million years. Current Biology, 26(20), R1088-R1100.
5. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593-1599.
6. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press.
7. Schrödinger, E. (1944). What is Life? The Physical Aspect of the Living Cell. Cambridge University Press.
8. Barlow, S. M. (1985). Central pattern generation involved in oral and respiratory control for feeding in the term infant. Current Opinion in Otolaryngology & Head and Neck Surgery, 17(3), 187-193.
9. Schmidhuber, J. (2010). Formal theory of creativity, fun, and intrinsic motivation (1990–2010). IEEE Transactions on Autonomous Mental Development, 2(3), 230-247.
10. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., ... & Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484-489.
11. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., ... & Hassabis, D. (2017). Mastering the game of Go without human knowledge. Nature, 550(7676), 354-359.
12. Elwood, R. W. (2011). Pain and suffering in invertebrates? ILAR Journal, 52(2), 175-184.
13. Herbert, F. (1965). Dune. Chilton Books.
14. Herrmann, E., Call, J., Hernàndez-Lloreda, M. V., Hare, B., & Tomasello, M. (2007). Humans have evolved specialized skills of social cognition: The cultural intelligence hypothesis. Science, 317(5843), 1360-1366.
15. Henrich, J. (2016). The Secret of Our Success: How Culture Is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter. Princeton University Press.
16. Tomasello, M. (2014). A Natural History of Human Thinking. Harvard University Press.
17. Bowles, S., & Gintis, H. (2011). A Cooperative Species: Human Reciprocity and Its Evolution. Princeton University Press.
18. Regier, T., & Kay, P. (2009). Language, thought, and color: Whorf was half right. Trends in Cognitive Sciences, 13(10), 439-446.
19. Tomasello, M. (2003). Constructing a Language: A Usage-Based Theory of Language Acquisition. Harvard University Press.
20. Scheffer, M., Bascompte, J., Brock, W. A., Brovkin, V., Carpenter, S. R., Dakos, V., ... & Sugihara, G. (2009). Early-warning signals for critical transitions. Nature, 461(7260), 53-59.
21. Goodhart, C. A. E. (1975). Problems of monetary management: The U.K. experience. Papers in Monetary Economics (Vol. I). Reserve Bank of Australia.
22. Christian, B. (2020). The Alignment Problem: Machine Learning and Human Values. W. W. Norton & Company.
23. Hayek, F. A. (1973). Law, Legislation and Liberty, Volume 1: Rules and Order. University of Chicago Press.
24. Irving, G., Christiano, P., & Amodei, D. (2018). AI safety via debate. arXiv preprint arXiv:1805.00899.
25. Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., & Legg, S. (2018). Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871.
26. Boyd, R., & Richerson, P. J. (1985). Culture and the Evolutionary Process. University of Chicago Press.
27. Chalmers, D. J. (1995). Facing up to the problem of consciousness. Journal of Consciousness Studies, 2(3), 200-219.
28. Noether, E. (1918). Invariante Variationsprobleme. Nachrichten von der Gesellschaft der Wissenschaften zu Göttingen, 235-257.
29. Pearl, J., & Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect. Basic Books.
30. Harnad, S. (1990). The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3), 335-346.
31. Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11(2), 127-138.
32. Fodor, J. A. (1983). The Modularity of Mind. MIT Press.
33. Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1798-1828.
34. Marcus, G. (2018). Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00631.
35. Damasio, A. R. (1994). Descartes' Error: Emotion, Reason, and the Human Brain. Putnam.
36. Mordatch, I., & Abbeel, P. (2018). Emergence of grounded compositional language in multi-agent populations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 1495-1502.
37. Tomasev, N., et al. (2025). Virtual Agent Economies. arXiv preprint arXiv:2509.10147.
38. Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386-408.
39. Minsky, M., & Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry. MIT Press.
40. Feigenbaum, E. A. (1977). The art of artificial intelligence: Themes and case studies of knowledge engineering. Proceedings of the 5th International Joint Conference on Artificial Intelligence, 1014-1029.
41. Sutton, R. (2019). The bitter lesson. Incomplete Ideas (blog). http://www.incompleteideas.net/IncIdeas/BitterLesson.html

1. Most interpretations of the Allegory of the Cave emphasize the difference between appearance and reality, shadow and substance. But Plato himself seems more concerned with epistemological questions: What can be learned from shadow alone? The prisoners gradually mastered the true skill of prediction and became masters of their own domain. The question then becomes: What structures does the shadow retain, and what structures does it discard? The geometry of projection determines the boundary between the knowable and the unknowable. ↩︎

2. Emmy Noether proved that every conservation law in physics corresponds to a symmetry (28). The conservation of energy stems from the symmetry of time translation. The conservation of momentum stems from the symmetry of spatial translation. This theorem reveals a profound truth: the information we can understand about a system is equal to the structure that remains unchanged under observable transformations. The prisoners observe transformations (motion, rotation of objects) and extract invariances (geometric relationships). But they cannot obtain the structures (depth, absolute size, three-dimensional shape) that the projection ignores. ↩︎

3. This relates to Judea Pearl's hierarchy of causality (29): association (observed relevance), intervention (manipulated variables), and counterfactual (imagining other possibilities). Language provides a wealth of association data, but limited intervention data. We describe the consequences of actions, but not the structural equations governing those consequences. Can a system learn causal structures solely from description? Pearl argues no, unless there are strong hypotheses. Others argue that sufficiently rich linguistic data may contain implicit causal information. I tend to agree with Pearl's skepticism, but acknowledge that this problem remains empirically unresolved. ↩︎

4. The symbol grounding problem (30) explores how symbols acquire meaning, rather than simply being empty symbols rearranged according to syntactic rules. Proposed solutions link symbols to perceptual and motor experiences. But how much grounding is needed? If linguistic experience is rich enough, can a system acquire functional understanding solely through linguistic experience? I remain unsure. My intuition suggests that grounding is important, but I admit that this intuition may reflect my own embodied experience rather than deeper principles. ↩︎

5. The basal ganglia are connected to the hypothalamus, amygdala, prefrontal cortex, and sensory cortex via complex circuits. These connections integrate current homeostasis (hunger, fatigue, pain), emotional valence (fear, desire, satisfaction), executive control (planning, inhibition), and sensory environment. Behavioral selection processes integrate all these factors in ways we can hardly comprehend. Reducing it to “reinforcement learning” captures some facts but ignores the richness of actual biological implementation. ↩︎

6. Carl Friston’s free energy principle attempts to formalize this (31): Organisms minimize contingencies, which is equivalent to maintaining a state of expectation compatible with their continued survival. The mathematical derivation is concise and elegant, and potentially profound. But I find myself troubled by the gap between formalism and phenomenology. What we experience is not minimizing quantities in an information-theoretic sense. We experience hunger, fear, and craving. Do these feelings operate at the computational level? Or are they merely accompanying phenomena, with processes functioning equally well without them? I truly do not know, and this uncertainty has long troubled my thinking about artificial systems. ↩︎

7. Some might argue that this distinction disappears under analysis. Organisms optimize implicit fitness functions shaped by evolution, while AI systems optimize explicit reward functions set by their designers. Both involve optimization. The perceived difference might simply reflect our emotional response to biological familiarity versus AI novelty. I take this argument seriously. Perhaps I'm projecting the wrong distinction onto essentially the same computational structure. However, I still feel there's an important distinction between optimization for survival and optimization done under someone's direction. ↩︎

8. This debate is difficult to reconcile. If emotions serve necessary computational functions (valence allocation, prioritization, behavioral motivation), then truly intelligent systems might require emotional architectures. But if emotions necessarily include the capacity to feel pain, then building such systems might be tantamount to creating pain for our convenience. Some argue we should focus on purely instrumental, narrow AI. Others believe moral patience and moral agency are inseparable, and entities incapable of feeling pain cannot possess truly respectable values. I waver between these two positions and have yet to reach a consensus. ↩︎

9. Philosophical literature on knowledge distinguishes between "knowing how to do" (procedural skills) and "knowing what" (propositional knowledge). One might argue that AlphaGo possesses "knowing how to do" but lacks "knowing what." It can play brilliant moves but cannot explain why. However, this distinction seems insufficient. Humans also struggle to explain why certain moves are powerful—we ourselves often rely on pattern recognition and intuition. Perhaps the real issue lies not in explanation, but in transfer and adaptation. ↩︎

10. This combinatorial structure stems from how humans represent knowledge. We maintain independent, modular concepts (piece types, movement rules, positional principles, tactical patterns) that can be flexibly combined. When a component changes, we only need to update it locally, without having to relearn everything from scratch. Cognitive science suggests that this modularity is the foundation of human cognition (32). However, it remains unclear whether this modularity is necessary for intelligence in the general sense or merely an accidental phenomenon in the evolution of human intelligence. ↩︎

11. This relates to the debate over whether neural networks learn "features" or "rules." Some argue that deep learning discovers combinatorial structures (33). Others argue that it relies on complex pattern matching, which fails under distributional shifts (34). My sense is that both are partially true: neural networks do discover real structures, but their representation of structures differs from symbolic systems, leading to different generalization properties. The question of which representation is “better” likely depends on the domain and the distribution of possible test cases. ↩︎

12. Antonio Damasio’s somatic labeling hypothesis (35) proposes that emotions provide crucial input for reasoning by imbuing options with emotional valence derived from experience. Patients with ventromedial prefrontal cortex damage retain intelligence but struggle with decision-making due to their inability to perceive which options are good or bad. If Damasio is correct, then emotion is not opposed to rationality but a necessary condition for it. But does this apply only to biological cognition, or does it also point to the universal nature of decision-making? ↩︎

13. If we construct systems with real emotional states, are we creating entities that suffer? Are we creating entities worthy of moral consideration? Have we erred in creating them for instrumental purposes? This question is particularly acute because those traits that might benefit artificial intelligence—concern for outcomes, a sense of urgency for consistency, and experiencing similar satisfaction in prosocial behavior—seem inseparable from the capacity for negative emotions. Can you have genuine preferences without frustration? Can you have genuine care without disappointment? ↩︎

14. From a functionalist perspective, the distinction between “genuine care” and “acting like care” ceases to exist. If a system behaves exactly like a genuinely caring system in all possible situations, then what is the basis for the assertion that it is merely simulating care? However, I cannot shake the intuition that there is some important difference between intrinsic and extrinsic motivations, and between genuine preferences and procedural objective pursuits. Perhaps this intuition reflects my own anthropomorphic tendencies rather than some deep-seated principle. Or perhaps it points to some value assessment mechanism that we have not yet formalized. ↩︎

15. The evolution of suprasociality remains a point of contention. Co-evolution of genes and culture, group selection, reputation dynamics, punishment systems, and language coordination may all play a role. What strikes me most is that many human cognitive traits only function within a social context. Theories of mind, moral reasoning, language recursion, and even certain aspects of executive control—these abilities seem meticulously designed to navigate complex social environments. Human intelligence is essentially social intelligence. ↩︎

16. Recent research in multi-agent reinforcement learning has revealed signs of emerging cultures: agents develop communication protocols, specialized roles, and conventions that spread within groups, such as language innovations (36). These cultures are still relatively primitive compared to human cultures, but they demonstrate that cultural dynamics can emerge from interaction topologies. The question is: what conditions can foster rich, cumulative cultures, rather than shallow behavioral coordination? I think we know almost nothing about this question, let alone finding the answer. ↩︎

17. Phase transitions in complex systems often exhibit some precursory signs: increased correlation length, critical deceleration, and increased variance (20). Do similar features foreshadow the formation of an artificial intelligence society? For example, the sudden increase in long-range behavioral correlations, the formation of hierarchical communication structures, and the emergence of stable interaction patterns that can resist disturbances. We need an AI ethnography—a systematic observation of qualitative changes in multi-agent dynamics. ↩︎

18. Max Weber distinguished between charismatic authority (dependent on outstanding individuals) and rational legal authority (rooted in rules and procedures). Charismatic authority is powerful but unstable, while rational legal authority is stable but can become rigid. Modern institutions attempt to balance these two through constitutional frameworks, providing stable rules while allowing for evolution through interpretation and modification. The key is that the rules can recognize their own shortcomings and update themselves. ↩︎

19. Recent studies have explored the design of automated mechanisms in multi-agent environments. AI systems learn interaction rules, and when agents perform selfish optimizations under these rules, the system can produce desired results (37). Early results are interesting, but limited to simple domains. How to extend to the complexity of the real world remains an unsolved challenge. The difficulty lies in how to define “desired results” when human values ​​themselves are not yet fully defined. We face the problem of recursive norms. ↩︎

20. I am deeply perplexed by this observation that evolution on Earth has produced both cooperation and exploitation, both altruism and parasitism, both beauty and horror. Natural selection is amoral; it optimizes reproductive success regardless of suffering or prosperity. Can we create selective pressures that systematically favor beneficial adaptations over harmful ones? Or would such an attempt merely change which strategies are selected without altering the fundamental amoral nature of the optimization process? I truly do not know. ↩︎

21. The connection between consciousness and intelligence remains philosophically controversial. Some argue that consciousness is a concomitant phenomenon, playing no causal role in cognitive processes (functionalists). Others believe that consciousness is essential for certain cognitive functions (integration, flexible response, self-modeling). My intuition leans towards the latter, but I admit this may reflect my inability to imagine unconscious intelligence rather than a deeper necessity. This issue deserves a more nuanced analysis than I can offer here. ↩︎

22. This speculation clearly faces opposition. Current AI systems can learn effectively without obvious phenomenological features. Deep learning has achieved remarkable results through unconscious optimization. Why is consciousness necessary? My answer is that current systems may lack the capabilities that consciousness provides. These capabilities are not those we have currently measured, but rather those related to autonomous goal setting, open-ended exploration, creative insight, and genuine understanding. Whether my view is correct is currently unproven. ↩︎

23. The history of artificial intelligence is filled with premature confidence and subsequent disappointment. Perceptrons were once thought to have solved the problem of intelligence (38), but subsequently encountered limitations (39). Expert systems were once thought to be able to acquire knowledge (40), but subsequently faced fragility and maintenance burdens. Deep learning requires massive computational resources (41), and ultimately succeeded, but new limitations have emerged. Each wave has been accompanied by genuine progress and overconfident inferences. We should be happy with the achievements, but at the same time be wary of blind optimism. ↩︎

24. The tension between innate structure and data learning has sparked decades of debate. Nativists argue that abundant innate endowments are necessary. Empiricists argue that general learning mechanisms are sufficient given enough data. I think both sides have their points. Abundant prior knowledge enables efficient sample learning, but it can also introduce biases and hinder the discovery of new structures. The optimal balance likely depends on the research domain, the amount of available data, and the acceptable error rate. I believe there is no universal answer. ↩︎