Sometimes it's simpler (less edges) to use the attributes (Cute) or animals (Bunny) or both (eg a particularly cute bunny). Assumption 3 doesn't allow mixing different bases together.
So here we have 2 attributes (for−−→fatt) & 4 animals (for−−−−→fanimal).
If the downstream circuit (let's assume a linear + ReLU) reads from the "Cute" direction then: 1. If we are only using −−−−→fanimal : Bunny + Dolphin (interpretable, but add 100 more animals & it'll take a lot more work to interpret) 2. If we are only using −−→fatt: Cute (simple)
If a downstream circuit reads from the "bunny" direction, then the reverse: 1. Only −−−−→fanimal: Bunny (simple) 2. Only −−→fatt: Cute + Furry ( + 48 attributes makes it more complex)
However, what if there's a particularly cute rabbit?
Only −−−−→fanimal: Bunny + 0.2*Dolphin(?) (+ many more animals)
Only −−→fatt: 2*Cute + Furry (+ many more attributes)
Neither of the above work! BUT what if we mixed them:
3. Bunny + 0.2*Cute (simple)
I believe you're claiming that something like APD would, when given the very cute rabbit input, activate the Bunny & Cute components (or whatever directions the model is actually using), which can be in different bases, so can't form a dictionary/basis. [1]
Technically you didn't specify that c(x)can't be an arbitrary function, so you'd be able to reconstruct activations combining different bases, but it'd be horribly convoluted in practice.