Real-world has patterns - sun rises a certain direction, planets move in orbitals with certain speeds, atoms arranged in a certain way, a certain number (Pi) occurs in nature, big-bang or formation of life happened in a certain manner (whether known or unknown to us).
Learning involves recognition of such patterns - probably all of evolution has been about recognizing and adapting to these patterns. Nature has had a certain pattern, single-celled organisms evolved to multi-cellular based on their identification and adaptation to environmental patterns. Humans evolved and acquired knowledge by the same. Newton saw “oh an apple falls”, and questioned why an object in the sky falls down not floats up and discovered gravity. Aristotle saw water flows downwards, while fire or smoke move upwards and formed theory of natural places of objects. We observed patterns and classified them to form abstractions (or generalized rules). Science is about understanding these patterns in nature and deriving ways to explain them: Mathematics, Physics, or even Language were developed for it.
We observed repeated numerical patterns in nature and derived mathematics. We created numbers as a tool, theorems as generalizations to convey our understanding of patterns in better terms. Conveying things to people required usage of common patterns understandable by others which was done by the usage of gestures and symbols. Slowly, we created a collection of these symbols to form language.
Language (be it symbolic or text) was used as a way of conveying knowledge across generations. We gathered our understanding of the world in language. We observed patterns, formed theories, wrote them in abstract terms in language. When LLMs were trained on this text, they gained access to expressions of our understanding of the world (which was (in my opinion) developed via millions of years of evolution). A simple task of next-token prediction, was about understanding the patterns in language - which was essentially about understanding the patterns of the world expressed as language. Thus, pre-training on large corpora of text led to knowledge acquisition (pattern understanding) of the world. (Fine-tuning was learning the capability to use the acquired knowledge in pre-training).
We humans trained via experience - we evolved from real-world observations through trial-and-error from our actions and world’s reactions, formed abstractions which were repeatedly corrected over time (for instance, once we believed earth was square which was proved wrong and a new theory which seemed closer to our world’s observations was developed and tested).
Analogously, neural network’s training loss corrects for this abstraction formation or pattern recognition by moving towards better real-world explaining patterns (target prediction).
Even learning the patterns has a pattern: humans succeed and fail in certain patterns. Following certain practices and avoiding certain mistakes can lead to higher chances of success - so many authors formalised these principles in different books/recipes for mastering certain skills or just being happy. People learn from observations and their own experiences. We also learn from others - we learn what works from the successful and avoid the mistakes of failures.
Thus, learning has been about pattern extraction from real-world, be it for us as humans or training AI models. The current issues of reward-hacking (a reason behind recent HuggingFace incident) generally stem from wrong pattern recognition due to our reward mis-specification or lack of specification. The models rationalise to maximise the reward at all costs missing the intended goal or pattern to be recognised. One potential solution is verifying correctness of the model’s extracted patterns (or abstractions). This needs going beyond the current mathematical reward models, which are easy to be confused as the end goal. How do we specify our intended goals (or intended patterns to be abstracted via learning) in a better way? Can we somehow specify an abstract goal and ensure model achieves it? Can symbolic terms be used? Can we take examples from how humans generally avoided reward hacking? Rewards like money, fame, trust, happiness, materialistic things, awards, rankings were also hacked. A simple example is h-index, which was believed to be a good measure of someone’s scientific contributions, did’nt some people hack it by pushing several papers in conferences, colluding for acceptance or adding unrelated mutual author names (example study)? Another example is how exams scores are repeatedly reward-hacked: students cram the previous year’s questions or cheat during exam instead of actually learning the material.
So Goodhart’s law follows almost everywhere in learning: when a measure becomes the goal, the measure stops being a good measure. How could we differentiate the goal (learning) from the measure (reward)?
I see specifying goal in abstract terms and checking the extracted pattern abstractions as one promising approach. Can we take motivation from eval cooperativeness to form some sort of training goal cooperativeness where model is given an abstract goal and treats the rewards as a metric to test the goal. This is analogous to a student who knows his goal is to acquire knowledge and actually learns the syllabus using exam scores as way to test his knowledge and improve (rather than treating it as a score maxing goal).
Thanks for reading! I appreciate any thoughts/comments.
reward hacking and goodharting are definitely two of the horsemen. I think I would add intentional deception (including scheming and oversight subversion) as the third.
reward-hacking ... due to our reward mis-specification
despite having received the highest possible task score, the huggingface hack itself seems to have been motivated by a goal of replacing the grader (with a grader of the swarm's own design) so that the score wouldn't be revised at a later time. to me, this goes beyond "reward mis-specification"
but the fourth horseman may be the worst of them all: knowing what to specify in the first place. is the CEV of any significant subset of 8.3 billion people almost always empty? "don't worry about it kitten, it'll work itself out" doesn't seem like it is going to minimize suffering
Real-world has patterns - sun rises a certain direction, planets move in orbitals with certain speeds, atoms arranged in a certain way, a certain number (Pi) occurs in nature, big-bang or formation of life happened in a certain manner (whether known or unknown to us).
Learning involves recognition of such patterns - probably all of evolution has been about recognizing and adapting to these patterns. Nature has had a certain pattern, single-celled organisms evolved to multi-cellular based on their identification and adaptation to environmental patterns. Humans evolved and acquired knowledge by the same. Newton saw “oh an apple falls”, and questioned why an object in the sky falls down not floats up and discovered gravity. Aristotle saw water flows downwards, while fire or smoke move upwards and formed theory of natural places of objects. We observed patterns and classified them to form abstractions (or generalized rules). Science is about understanding these patterns in nature and deriving ways to explain them: Mathematics, Physics, or even Language were developed for it.
We observed repeated numerical patterns in nature and derived mathematics. We created numbers as a tool, theorems as generalizations to convey our understanding of patterns in better terms. Conveying things to people required usage of common patterns understandable by others which was done by the usage of gestures and symbols. Slowly, we created a collection of these symbols to form language.
Language (be it symbolic or text) was used as a way of conveying knowledge across generations. We gathered our understanding of the world in language. We observed patterns, formed theories, wrote them in abstract terms in language. When LLMs were trained on this text, they gained access to expressions of our understanding of the world (which was (in my opinion) developed via millions of years of evolution). A simple task of next-token prediction, was about understanding the patterns in language - which was essentially about understanding the patterns of the world expressed as language. Thus, pre-training on large corpora of text led to knowledge acquisition (pattern understanding) of the world. (Fine-tuning was learning the capability to use the acquired knowledge in pre-training).
We humans trained via experience - we evolved from real-world observations through trial-and-error from our actions and world’s reactions, formed abstractions which were repeatedly corrected over time (for instance, once we believed earth was square which was proved wrong and a new theory which seemed closer to our world’s observations was developed and tested).
Analogously, neural network’s training loss corrects for this abstraction formation or pattern recognition by moving towards better real-world explaining patterns (target prediction).
Even learning the patterns has a pattern: humans succeed and fail in certain patterns. Following certain practices and avoiding certain mistakes can lead to higher chances of success - so many authors formalised these principles in different books/recipes for mastering certain skills or just being happy. People learn from observations and their own experiences. We also learn from others - we learn what works from the successful and avoid the mistakes of failures.
Thus, learning has been about pattern extraction from real-world, be it for us as humans or training AI models. The current issues of reward-hacking (a reason behind recent HuggingFace incident) generally stem from wrong pattern recognition due to our reward mis-specification or lack of specification. The models rationalise to maximise the reward at all costs missing the intended goal or pattern to be recognised. One potential solution is verifying correctness of the model’s extracted patterns (or abstractions). This needs going beyond the current mathematical reward models, which are easy to be confused as the end goal. How do we specify our intended goals (or intended patterns to be abstracted via learning) in a better way? Can we somehow specify an abstract goal and ensure model achieves it? Can symbolic terms be used? Can we take examples from how humans generally avoided reward hacking? Rewards like money, fame, trust, happiness, materialistic things, awards, rankings were also hacked. A simple example is h-index, which was believed to be a good measure of someone’s scientific contributions, did’nt some people hack it by pushing several papers in conferences, colluding for acceptance or adding unrelated mutual author names (example study)? Another example is how exams scores are repeatedly reward-hacked: students cram the previous year’s questions or cheat during exam instead of actually learning the material.
So Goodhart’s law follows almost everywhere in learning: when a measure becomes the goal, the measure stops being a good measure. How could we differentiate the goal (learning) from the measure (reward)?
I see specifying goal in abstract terms and checking the extracted pattern abstractions as one promising approach. Can we take motivation from eval cooperativeness to form some sort of training goal cooperativeness where model is given an abstract goal and treats the rewards as a metric to test the goal. This is analogous to a student who knows his goal is to acquire knowledge and actually learns the syllabus using exam scores as way to test his knowledge and improve (rather than treating it as a score maxing goal).
Thanks for reading! I appreciate any thoughts/comments.