This is a special post for quick takes by TurnTrout. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
A semi-formalization of shard theory. I think that there is a surprisingly deep link between "the AIs which can be manipulated using steering vectors" and "policies which are made of shards."[1] In particular, here is a candidate definition of a shard theoretic policy:
A policy has shards if it implements at least two "motivational circuits" (shards) which can independently activate (more precisely, the shard activation contexts are compositionally represented).
By this definition, humans have shards because they can want food at the same time as wanting to see their parents again, and both factors can affect their planning at the same time! The maze-solving policy is made of shards because we found activation directions for two motivational circuits (the cheese direction, and the top-right direction):
On the other hand, AIXI is not a shard theoretic agent because it does not have two motivational circuits which can be activated independently of each other. It's just maximizing one utility function. A mesa optimizer with a single goal also does not have two motivational circuits which can go on and off in an independent fashion.
- This definition also makes obvious the fact that "shards" are a matter of implementation, not of behavior.
- It also captures the fact that "shard" definitions are somewhat subjective. In one moment, I might model someone is having a separate "ice cream shard" and "cookie shard", but in another situation I might choose to model those two circuits as a larger "sweet food shard."
So I think this captures something important. However, it leaves a few things to be desired:
- What, exactly, is a "motivational circuit"? Obvious definitions seem to include every neural network with nonconstant outputs.
- Demanding a compositional representation is unrealistic since it ignores superposition. If dimensions are compositional, then they must be pairwise orthogonal. Then a transformer can only have shards, which seems obviously wrong and false.
That said, I still find this definition useful.
I came up with this last summer, but never got around to posting it. Hopefully this is better than nothing.
- ^
Shard theory reasoning led me to discover the steering vector technique extremely quickly. This link would explain why shard theory might help discover such a technique.
Showing 3 of 10 replies (Click to show all)
1
1
Sorry for replying to 6mo old post.
I think the statement is more about context or framing.
You can take a shard agent and produce a utility function to describe it's values, and you could have AIXI maximize that function. So in that sense "shard agents" are a subset of agents AIXI can implement, but AIXI by itself is not a shard agent.
Note that it is not enough to say the utility function values multiple different things, that's normal for a utility function, it has to specify that the agent acts to optimize different utility functions depending on the context which makes the utility function massively more complicated. This justifies having the distinction between an agent with a single utility function and an agent with context dependent utility.
And I would say, even if you have AIXI running with a sharded utility function, it is implementing a shard agent while itself being understood as a non-shard agent.
30
Funnily enough I was thinking about this yesterday and wondering if I'd be able to find it, so great timing! Thanks for the comment.
12
Here's some intuition of why I think this could be a pretty big deal, straight from Claude, when prompted with 'come up with an explanation/model, etc. that could unify the findings in these 2 papers' and fed Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition and Understanding and Controlling a Maze-Solving Policy Network:
[...]
When further prompted with 'what about superposition and task vectors?':
[...]
Then, when fed Alex's shortform comment and prompted with 'how well would this fit this 'semi-formalization of shard theory'?':
[...]
(I guess there might also be a meta-point here about augmented/automated safety research, though I was only using Claude for convenience. Notice that I never fed it my own comment and I only fed it Alex's at the end, after the 'unifying theory' had already been proposed. Also note that my speculation successfully predicted the task vector mechanism before the paper came out; and before the senior author's post/confirmation.)
Edit: and throwback to an even earlier speculation, with arguably at least some predictive power: https://www.lesswrong.com/posts/5spBue2z2tw4JuDCx/steering-gpt-2-xl-by-adding-an-activation-vector?commentId=wHeawXzPM3g9xSF8P.