The progress and success of AI research in the past decade seem all point back to some very basic principles about the nature of intelligence discovered last century. These include 1) Feedback mechanism by Robert Wiener, 2) Information theory by Claude Shannon, 3) Universal computation by Alan Turing:

1) Feedback in a very general sense is an abstraction of information close-loop to ensure continual improvement/adaptation of an intelligent sub-system.

Humans and animals need feedback to improve skill-level of doing something. A product/technique needs real-world feedback signal to iteratively improve to be usable. Any useful autonomous system is closed-loop feedback system. From this perspective, it is unsurprising that open-loop self-supervised/imitation learning alone does not produce satisfactory results, as there is no feedback about the agent's performance in the real world while this information is necessary for improving/alignment. LLM trained via open loop self-supervised learning, despite unlocking amazing capabilities like ICL and CoT, does not know how to properly respond to humans. Similarly, lack of information feedback is one of the main reasons that advances in autonomous driving and robot learning are slow. Driving policy learned in a purely open-loop manner by imitating human driver demonstration has a much higher collision rate than that of the demonstration. It can only be improved by incorporating RL loss or any mechanism equivalent to DAgger, because without that it lacks the information of "an unsafe/collision state".

From this perspective, an AGI must be an agent that can autonomously interact with the environment to collect feedback on its behavior through observation, and continuously process and incorporate this information through some mechanism (e.g., RL as for AIXI, neural plasticity/hebbian learning/self-critique as for humans and animals, rewriting its source code as for Godel machine).

2) Information entropy is closely related to thermodynamic entropy via Landauer's principle, and together obeys the second law of thermal dynamics. As such, a closed system (e.g. a machine learning algorithm) without any information feedback from the external environment (adding more data, human's knowledge) typically cannot improve its intelligence level (entropy reduction).

The most used approach to bridging the information...

Posts

Wikitag Contributions

Comments