Let be an input activation. Let contain feature vectors as rows. Write its -th row as , so is a column vector. Define

Weights are tied: the same detects features and reconstructs from them. Decoder bias is zero. No norm constraint yet.

The factor keeps derivatives clean. For the paper's convention , use and multiply the entire gradient below by two. This is an algebraic conversion; it does not assert identical optimizer dynamics under an arbitrary loss rescaling.

Write

Here is the reconstruction residual and records which ReLUs are active. Work away from ; at the kink, use the usual zero derivative convention.

The exact negative gradients are

Three pieces are visible in (3):

Tying the weights makes these contributions add. Updating one shared matrix updates the whole SAE. There is no need for a second independently trained weight matrix. But both uses of that matrix still contribute to its derivative.

To see why, perturb by . With and fixed,

The reconstruction loss changes by . The sparsity loss changes by , because ReLU activations are nonnegative. Collect the coefficients of to obtain (3). Appendix A in the paper gives the same calculation one weight at a time.

First take a linear encoder, , and set , . Equation (3) becomes

Expand just its decoder contribution:

This is the update expression in Oja's subspace algorithm, up to learning rate. The first term is Hebbian: output activity times input activity. The second subtracts the reconstruction contribution of the current features.

For feature , written as a column-vector update,

Now discard the cross-latent terms, equivalently replace by :

This is the single-neuron Oja rule, with .

Thus the general route from a tied linear AE gradient to independent Oja updates has two explicit omissions: the encoder contribution , and the off-diagonal terms in . Equation (6) is an exact identity; those omissions are approximations. Our paper’s Section 6 reverses the path attribution: it says the missing contribution is the decoder gradient. With the encoder/decoder conventions above, the subspace update retains the decoder-path contribution and omits the encoder-path contribution. This is a correction to the explanation, in addition to the duplicated term in Table 4.

Equation (6) also holds algebraically for ReLU outputs, but the full encoder term then has the gate . Calling an update "Oja's rule" additionally requires care about what its activity variable means.

A useful consequence: co-active latents interact in the reconstruction update. Independent Oja rules ignore those interactions. When at most one latent is active, the cross terms vanish exactly. Sparse activity reduces the number of possible interactions, but does not alone bound their total size.

We can make the connection stronger without dropping a gradient term.

Assume zero bias, unit-norm rows, and exactly one active ReLU latent . Let

All other preactivations are strictly negative. Locally, the reconstruction is . Therefore

The active neuron's encoder reconstruction contribution vanishes. Inactive neurons have both and . The full reconstruction update is consequently

That is Oja's rule exactly, at this point in parameter space.

Now enforce the unit-norm constraint. Its tangent-space projector is

Since , projected descent on the full reconstruction-plus- loss gives

Here means the Euclidean gradient projected onto the unit sphere. The reconstruction term contributes ; the activation penalty contributes .

The sign now has a simple meaning:

• : move toward the input along the sphere.
• : move away, reducing the activation.
• : the two contributions cancel.

This is a precise setting in which a tied SAE produces Hebbian and anti-Hebbian behavior from the same loss. The sign depends on activation magnitude relative to ; it is not the same rule as assigning a positive update to the winner and negative updates to every loser.

All three special-case assumptions need qualification in the experiment. The Hebbian setups averaged about 22 active latents per input; Appendix A.2 reports no explicit decoder-weight normalization; and Appendix A.2.1 sets bias = True. Exact unit norm and zero encoder bias were therefore not enforced. Equation (10) is an explanatory special case, not an exact description of those runs.

The unit sphere also makes the penalty connection easy to derive on its own. Take one weight vector , no bias, , and . Then (recall )

For any differentiable scalar penalty (just some function for now, do not startle), the chain rule gives

For positive activation :

The first is a constant-strength repulsive direction. The second is negative Oja, with repulsion weighted by the activation.

For an unconstrained weight vector, the respective negative gradients are just and on the positive branch. The correction comes from projection/normalization. It is not present in the ordinary unconstrained sparsity gradient.

Two distinctions matter. For signed linear activations, the coefficient is . For , both penalties have zero gradient on the inactive branch . Also, squared suppresses activation magnitude; it does not have the same zero-inducing behavior as .

The direction really does suppress a positive response. Under a continuous update

This concerns response to the current input. It does not prove mutual orthogonality of learned features or recovery of independent concepts.

The geometry requires unit norm, which is favorable in the SAE as well^[2]. In general,

Away from the sphere, the Oja direction includes a radial component. A finite Euler step also leaves the sphere by a second-order amount, so exact constrained optimization requires renormalization or another suitable constrained update. Unit-norm rows do not mean orthogonal rows; a 3,200-row dictionary in 64 dimensions cannot have all rows mutually orthogonal.

Separate three quantities that are easy to mix:

Here is the raw score (preactivation), the SAE code (activation), and a softmax competition weight (where the method name comes from). For this comparison, biases are omitted. Notation clarification: Table 4 writes inside its softmax and normalization term. That symbol must not silently inherit the post-ReLU meaning from the SAE definition. Equation (13) below uses the preactivation explicitly; the implementation qualification follows the equation. A bias-free competitive variant has the form

The checked upstream implementation computes competition from preactivations. HebbSoftKrotovLinear defaults to mode = 0, with positive winner and negative losers; it also has rank-selected modes. Bias enters the preactivation when enabled. Updates are batch-aggregated and rescaled, with additional bias and learning-rate machinery. The coefficient is exact for mode 0. The schematic simplifications in (13) are omitted bias, batch aggregation, update rescaling, and learning-rate machinery.

Appendix A.2.1 reports t_invert = 50. The upstream activation function uses , equivalent to . That does not establish near-hard competition without the score gaps. If is the largest score,

With 3,200 equal scores, at any positive temperature. With well-separated scores it can approach one. Winner-probability or entropy measurements would establish which regime the runs occupied.

On unit-norm rows (recall ), (13) is . Its geometry matches the attraction/repulsion directions above. Its coefficient generally differs:

• Oja uses .
• Projected descent on an active latent uses a constant negative coefficient.
• Projected squared- descent uses a coefficient proportional to .
• The competitive rule uses , which depends on every score and on winner selection.

For a loser with , can be read as projected descent with an input-dependent step size, or as projected squared- descent with step size . This is a pointwise direction match. It does not identify the rule with ordinary descent on one fixed weighted SAE loss. For an inactive ReLU latent, that direction match fails: its SAE penalty gradient is zero, whereas its competitive update can be nonzero.

One further subtlety: writing as a proposed loss does not solve the problem. Differentiating it also differentiates . Freezing those weights during differentiation defines a surrogate update, not the ordinary gradient of that expression.

The distinction between magnitude and rank (i.e. top-1, top-2 neuron, etc) is worth discussing. Equation (10) changes sign at an activation threshold. Equation (13) changes sign according to who wins. The rank-selected Krotov variant tabulated in Table 4 uses a constant negative coefficient on the selected neuron, matching projected descent on a positive activation up to that coefficient. This is a comparison to the tabulated rule, not a claim about which experimental mode ran.

The upstream rank modes differ. Mode 1 assigns the selected neuron coefficient ; mode 2 assigns . Other neurons retain in both modes. Only mode 2 has a constant selected-neuron coefficient. Even there, the full update norm need not be constant: it also depends on .

The useful connection is explicit: reconstruction supplies Hebbian terms; activation penalties supply suppressive (anti-Hebbian) terms; normalization supplies the projection correction. Competition, coefficient choices, and omitted interactions determine how far a particular algorithm departs from exact tied-SAE descent.

Use for an input coordinate, for a latent, and for a reconstructed coordinate. From (1),

The first term differentiates the decoder use of ; the second differentiates the encoder use. Hence

Because for ReLU,

Combining them reproduces (3). This also fixes the transpose/shape conventions: each term is .

For a general elementwise activation and a differentiable penalty , the same argument yields

For linear signed codes with , use and away from zero. For a batch-mean loss, average the per-input gradients. Additional coordinate-wise normalizations of MSE or sparsity must be included explicitly in the coefficients.

The exact per-row ReLU update is another useful way to inspect the omissions:

This is a direct algorithm for updating the shared SAE matrix. An autodiff engine is optional. The residual and cross-feature sums remain required for exact descent; removing autodiff does not make those dependencies disappear. "Local" therefore needs a specified meaning: layer-local computation is different from a rule using only one synapse's weight and its pre/post activities.

Even with winner identity fixed, the score-space coefficient field is generally not the Euclidean gradient of a scalar potential. For ,

These cross derivatives differ when . A smooth scalar potential would require them to agree. This rules out that particular ordinary score-space gradient interpretation. It does not rule out a metric-dependent optimization view, a surrogate construction, a Lyapunov function, or an interpretation with additional state.

Mean standard deviation over five runs, transcribed from Table 1 in the paper. Metrics use the training subset. Dictionary counts reflect the paper's reported activity metric; the precise evaluation window requires its implementation.

• Table 4 labels its fast-AI Krotov implementation with the 2018 citation; the competitive-hidden-units work is actually the 2019 paper named "Unsupervised learning by competing hidden units".
• The special-case (see A3. An exact special case: one active feature.) derivation here is an explanatory extension; it is not a new experiment or a claim of historical priority.
• Correction to the paper's Table 4:
  • its two bracketed terms for the linear tied AE both simplify to . The full negative gradient is , as in (5). The subspace term is the decoder-path contribution under the conventions used here.
  • Section 6 also names the omitted path incorrectly: it is the encoder contribution, not the decoder contribution.
  • Separately, Equation (13) (from A5. Where the competitive SoftHebb rule fits.) makes the preactivation/SAE-code notation explicit rather than carrying over Table 4’s ambiguous .
  • These corrections above concern the derivation and its interpretation; they do not by themselves establish an error in the experimental implementation.