Abstract

This paper presents $\text{ProLU}$, an alternative to $\text{ReLU}$ for the activation function in sparse autoencoders that produces a pareto improvement over both standard sparse autoencoders trained with an L1 penalty and sparse autoencoders trained with a Sqrt(L1) penalty.

The gradient wrt. $b$ is zero, so we generate two candidate classes of differentiable $\text{ProLU}$:

• ${\text{ProLU}}_{ReLU}$
  • $\frac{{\partial }^{\ast }{\text{ProLU}}_{ReLU}(m_i,b_i)}{\partial b_i}=\frac{\partial {\text{ProLU}}_{ReLU}(m_i,b_i)}{\partial m_i}=\{\begin{array}{cc}1& \text{if}{m}_i+b_i>0\text{and}{m}_i>0\\ 0& \text{otherwise}\end{array}$
• ${\text{ProLU}}_{STE}$
  • $\frac{{\partial }^{\ast }{\text{ProLU}}_{STE}(m_i,b_i)}{\partial m_i}=\{\begin{array}{cc}1+m_i& \text{if}{m}_i>0\text{and}{m}_i+b_i>0\\ 0& \text{otherwise}\end{array}$
  • $\frac{{\partial }^{\ast }{\text{ProLU}}_{STE}(m_i,b_i)}{\partial b_i}=\{\begin{array}{cc}{m}_i& \text{if}{m}_i>0\text{and}{m}_i+b_i>0\\ 0& \text{otherwise}\end{array}$

PyTorch Implementation

Introduction

SAE Context and Terminology

Learnable parameters of a sparse autoencoder:

• $W_{enc}$ : encoder weights
• $W_{dec}$ : decoder weights
• $b_{enc}$ : encoder bias
• $b_{dec}$ : decoder bias

The output of an SAE is given by

Training

Notation: Encoder/Decoder

Let 

$$\text{encode}\left(x\right)=\text{ReLU}\left(\right(x-b_{dec})W_{enc}+b_{enc})$$$$\text{decode}\left(a\right)=a{W}_{dec}+b_{dec}$$

 so that the full computation done by an SAE can be expressed as 

$$\text{SAE}\left(x\right)=\text{decode}\left(\text{encode}\right(x\left)\right)$$

An SAE is trained with gradient descent on 

where $\lambda$ is the sparsity penalty coefficient (often "L1 coefficient") and $P$ is the sparsity penalty function, used to encourage sparsity.

$P$ is commonly the L1 norm $||a|{|}_1$ but recently $P\left(a\right)=||a|{|}_{\frac{1}{2}}^{\frac{1}{2}}$ has been shown to produce a Pareto improvement on the L0 and CE metrics. We will use this as a further baseline to compare against when assessing our models in addition to the standard $\text{ReLU}$-based SAE with L1 penalty.

Motivation: Inconsistent Scaling in Sparse Autoencoders

Due to the affine translation, sparse autoencoder features with nonzero encoder biases only perfectly reconstruct feature magnitudes at a single point.

This poses difficulties if activation magnitudes for a fixed feature tend to vary over a wide range. This potential problem motivates the concept of scale consistency:

A scale consistent response curve

The bias maintains its role in noise suppression, but no longer translates activation magnitudes when the feature is active.

The lack of gradients for the encoder bias term poses a challenge for learning with gradient descent. This paper will formalize an activation function which gives SAEs this scale-consistent response curve, and motivate and propose two plausible synthetic gradients, and compare scale-consistent models trained with the two synthetic gradients to standard SAEs and SAEs trained with Sqrt(L1) penalty.

Scale Consistency Desiderata

Notation: Centered Submodule

The use of the decoder bias can be viewed as performing centering on the inputs to a centered SAE then reversing the centering on the outputs: 

$$SAE\left(x\right)=SA{E}_{cent}(x-b_{dec})+b_{dec}$$

 

$$SA{E}_{cent}\left(x\right)=\text{ReLU}(x{W}_{enc}+b_{enc})W_{dec}$$

Notation: Specified Feature

Let $W^i$ denote the weights and $b_{enc}^i$ the encoder bias for the $i$-th feature. Then, let 

$$SA{E}^i\left(x\right)=SA{E}_{cent}^i(x-b_{dec})+b_{dec}$$

 

$$\text{where}SA{E}_{cent}^i\left(x\right)=\text{ReLU}(x{W}_{enc}^i+b_{enc}^i)W_{dec}^i$$

Conditional Linearity

Noise Suppresion Threshold

Proportional ReLU $\left(\text{ProLU}\right)$

The concept of using a thresholding nonlinearity instead of a learned bias term for inhibition autoencoders was introduced a decade prior to this work in "Zero-Bias Autoencoders and the Benefits of Co-adapting Features." Their TRec activation function is very similar to ProLU, with the following key differences:

• ProLU has a different threshold parameter for each hidden unit
• Learning of ProLU's threshold parameter ($b_i$) is enabled via psuedoderivatives

We define the base (without psuedoderivatives not yet defined) Proportional ReLU $\left(\text{ProLU}\right)$ as:

Backprop with $\text{ProLU}$:

To use $\text{ProLU}$ in SGD-optimized models, we first address the lack of gradients wrt. the $b$ term.

$\text{ReLU}$ gradients:

For comparison and later use, we will first consider $\text{ReLU}$: partial derivatives are well defined for $\text{ReLU}$ at all points other than $x_i=0$:

Gradients of $\text{ProLU}$:

Partials of $\text{ProLU}$ wrt. $m$ are similarly well defined: 

However, they are not well defined wrt. $b$, so we must synthesize these.

Related Work

 [Zero-bias autoencoders and the benefits of co-adapting features] introduced zero-bias autoencoders which are very related to this work. The ProLU SAE can be viewed as a zero-bias autoencoder where the threshold parameter theta is learned using straight-through estimators and which uses an L1 penalty to encourage sparsity  

Methods

Notation: Synthetic Gradients

Let $\frac{{\partial }^{\ast }f}{\partial x}$ denote the synthetic partial derivative of $f$ wrt. $x$, and ${\nabla }^{\ast }f$ the synthetic gradient of f, used for backpropagation as a stand-in for the gradient.

Different synthetic gradient types

We train two classes of $\text{ProLU}$ with different synthetic gradients. These are distinguished by their subscript:

• ${\text{ProLU}}_{ReLU}$
• ${\text{ProLU}}_{STE}$

They are identical in output, but have different synthetic gradients. I.e. 

Defining ${\text{ProLU}}_{ReLU}$: $\text{ReLU}$-like gradients

The first synthetic gradient is very similar to the gradient for $\text{ReLU}$. We retain the gradient wrt. $m$, and define the synthetic gradient wrt. $b$ to be the same as the gradient wrt. $m$:

Defining ${\text{ProLU}}_{STE}$: Derivation from straight-through estimator

The second class of $\text{ProLU}$ uses synthetic gradients for both $b$ and $m$ and can be motivated by framing $\text{ProLU}$ and $\text{ReLU}$ in terms of the threshold function, and a common choice of straight-through estimator (STE) for the threshold function. This is a plausible explanation for the observed empirical performance but it should be noted that there are many degrees of freedom and possible alternative 

Setup

The threshold function $\text{Thresh}$ is defined as follows:

We will rephrase the partial derivative of $\text{ReLU}$ in terms of the threshold function for ease of later notation: 

It is common to use a straight-through estimator (STE) to approximate the gradient of the threshold function:

We can reframe $\text{ProLU}$ in terms of the threshold function: 

Synthetic Gradients wrt. $m$

Now, we take partial derivatives of $\text{ProLU}$ wrt. $m$ using the STE approximation for the threshold function:

Synthetic Gradients wrt. $b$

Choice of Straight-Through Estimator

There are many possible functions to use for $S{T}_{\text{Thresh}}\left(x\right)$. In our experiments, we take the derivative of $\text{ReLU}$ as the choice of straight-through estimator. This choice has been used in training quantized neural nets.^[1]

$S{T}_{\text{Thresh}}\left(x\right):=\text{Thresh}\left(x\right)$

then, synthetic gradients wrt. $m$ are given by,

and wrt. $b$ are given by,

$\text{ProLU}$ Sparse Autoencoder

We can express the encoder of a $\text{ProLU}$ SAE as

No change is needed to the decoder. Thus,

Experiment Setup

Shared among all sweeps:

• Adam optimizer, with:
  • ${\beta }_1=0.9,{\beta }_2=0.999$
• $\text{batch size}=4096$
• Data
  • Trained on gpt2 layer 6 pre-residual activations
  • Tokens: ~400m tokens from The Pile @ack(Alan Cooney's pre-tokenized pile)
  • -> ~100k gradient steps
• LR schedule
  • Warmup for $\frac{2}{1-{\beta }_2}=2,000$ steps in accordance with On the adequacy of untuned warmup for adaptive optimization
  • Linear warmup after each resample. Same value 2,000
  • Linear cooldown to 1/10 initial value over 20,000 steps starting at 75,000 steps
• Anthropic resampling
  • I used 3e-6 as the dead threshold rather than 0
  • Resample at 25,000 and 50,000 steps
  • The proportion of the average encoder norm resampled to varied between sweeps
• Normalization:
  • L2 normalization as proposed by Anthropic
• SAE details
  • Dictionary expansion factor of 16
  • Tied decoder bias, untied encoder/decoder weights

Varying between sweeps:

• Experiment 1:
  • $lr=0.001$
  • 30 total runs
  • Resampled to 0.02 of avg encoder norm
• Experiment 2:
  • $lr=0.0003$
  • 48 total runs
  • Resampled to 0.02 of avg encoder norm
• Experiment 3:
  • $lr=0.001$
  • 30 total runs
  • Resampled to 0.2 of avg encoder norm
  • adjusted L1-coefficient ranges for each model, to get more overlap in L0 ranges.
    • different architectures respond very differently to l1 coefficients

Varying within sweeps

• L1 coefficient
• Architecture choice of nonlinearity:
  • $\text{ReLU}$
  • ${\text{ProLU}}_{ReLU}$
  • ${\text{ProLU}}_{STE}$
• L1 Penalty type
  • L1: $P\left(a\right)=||a|{|}_1$
  • Sqrt(L1): $P\left(a\right)=||a|{|}_{1/2}^{1/2}$

Results

Let:

• $L_{model}$ be the CE loss of the model unperturbed on the data distribution
• $L_{reconstructed}$ be the CE loss of the model when activations are replaced with the reconstructed activations
• $L_{zero}$ be the CE loss of the model when activations are replaced with the zero vector

Degradation: or Information Lost. This measures how much information about the correct next token the model loses by having its activations $a$ replaced with the SAE's reconstruction $\text{SAE}\left(a\right)$.

• $\text{degradation}=L_{reconstructed}-L_{model}$

For $L0<25$:

• The pareto-best architecture uses the ${\text{ProLU}}_{STE}$ nonlinearity with an L1 sparsity penalty.

For $L0>25$:

• There are no occurrences of models using ${\text{ProLU}}_{STE}$ with L1 penalty in this L0 range
• Of the remaining models, ${\text{ProLU}}_{\text{ReLU}}$ with Sqrt(L1) penalty is pareto-best.

Further Investigation

MSE/L1 Pareto Frontier

The gradients of $\text{ProLU}$ are not the gradients of the loss landscape, so it would be a reasonable default to expect these models to perform worse than a vanilla SAE. Indeed I expect they may perform worse on the optimization target, and that the reason why this is able to work is there is slack in the problem introduced by us being unable to optimize for our actual target directly -- our current options are to optimize for L1 or Sqrt(L1) as sparsity proxies for what we actually want because L0 is not a differentiable metric.

Actual target: minimize L0 and bits lost
Optimization (proxy) target: minimize L1 (or $\sqrt{L1}$)) and MSE

Because we're not optimizing for the actual target, I am not so surprised that there may be weird tricks we can do to get more of what we want.

On this vein of thought, my prediction after seeing the good performance  on the actual target (and prior to checking this prediction) was:

Despite improved performance on degradation/L0 ProLU SAEs will have the same or worse on the MSE/L1 curve.
We may also see the higher performing architectures have greater L1/L0

Let's check:
In favor of the hypothesis, while other architectures sometimes join it on the frontier, the Vanilla ReLU is present for the entirety of this Pareto frontier. On the other hand, at lower sparsity levels ${\text{ProLU}}_{STE}$ joins it at the frontier. So the part where this change does not improve performance on the optimization target seems true, but it's not clear that better performance on the actual target is coming from worse performance on the optimization target. 

This suggests a possible reason for why the technique works well:
Possibly the gains from this technique do not come from scale consistency so much as that it forced us to synthesize some gradients and those gradients happened to point more in the direction of what we actually want.

Here is the graph of L1 norm versus L0 norm:

This looks like it's possible that what is working well here is the learned features are experiencing less suppression,  but that may not be the only thing going on fixing this. Feature suppression is still consistent with the scale consistency hypothesis, as consistent undershooting would be an expected side effect if that is a real problem, since regular SAEs may be less able to filter unwanted activations if they are keeping biases near zero in order to minimize errors induced by scale inconsistency. 

More investigation is needed here to create a complete or confident picture of what is cause of the performance gains in $\text{ProLU}$ SAEs. 

Unfortunately, I did not log $\sqrt{L1}$ so I can't compare with that curve, but could load the models to create those graphs in follow-up work.

Acknowledgements

Noa Nabeshima and Arunim Agarwal gave useful feedback and editing help on the draft of this post.

Mason Krug for in depth editing of my grant proposal, which helped seed this writeup and clarify my communication.

How to Cite

@misc{ProLUNonlinearity,
   title = {ProLU: A Nonlinearity for Sparse Autoencoders},
   author = {Glen M. Taggart},
   year = {2024},
   howpublished = {\url{https://www.alignmentforum.org/posts/HEpufTdakGTTKgoYF/prolu-a-nonlinearity-for-sparse-autoencoders}},
}

1. ^{^}