This is a summary of the project I undertook during SPAR, mentored by Ari Brill. I have recently had to prioritize other work, and found more questions than answers, as is often the case. I think a write-up will be useful both for coming back to this and interesting for the findings (and non-findings!) it presents.

Introduction

Real-world data is implicitly structured by the reality it represents, not just its surface-level format. AI systems can learn surface patterns, deeper structures, or a combination of both. Let's illustrate this with an example.

Take (the concept of) a dog, for instance. This dog can be represented by a picture, a GIF, a paragraph that describes it, a poem, or any other format that we can think of. Now, say we want to train an AI system to learn a function that involves said dog. This could be a function of the sort: $f : A n i m a l \to F o o d$ , where $f$ defines the animal´s favorite food. Now, in order to model $f$ , the system^[1] needs to represent the concept of a dog, and associate it to $f (d o g)$ , which might be peanut butter. It might do so using surface level patterns (two ears, fluffy hair, four legs and paws) or by embedding the concept in a deeper way and truly representing the dog, which would intuitively mean having all similar dogs embedded close together in data space (which contains the remaining empirical degrees of freedom after accounting for invariances and symmetries).

The model described in [1] and [2] proposes a theoretical framework in which such deep structure naturally arises, with the assumptions described further below.

Building on the work of [3] and [4], we aim to advance fundamental scientific understanding of how AI systems internally represent the world. Our goal is to contribute to AI alignment by developing a model of data distributional geometric structure, which could provide an inductive bias for mechanistic interpretability.

In this work, we propose to empirically validate the learning model described in [1]. This relies on two key assumptions:

Realistic target functions often reflect a mix of distinct behaviors rather than a single concept. We work with such context-dependent functions and model them as higher-order functions that produce task-specific first-order functions.
We study general-purpose learning, where the system has no built-in knowledge of the data or task. We assume such a

...

Posts

Wikitag Contributions

Comments