This graph shows how many productive work minutes you get when adding an additional person who works 40 hours a week, to a team that spends 2-60 minutes per week on each team member to keep them in sync with the rest of the team. If you spend 30 minutes of everyone's time per person to keep then in sync with everyone else, then if you try to scale that to 80 people, you have gained no productive hours, as it takes 40 hours just to keep everyone in sync with your last employee:

I think the effect you’re describing is even stronger than this model/graph implies, because the 40 hours you’re gaining are marginal-employee-hours, while the coordination costs are paid in average-employee-hours.

Online training infrastructure: For diffuse (non-concentrated) issues, training the AIs to behave better online (on the actual tasks we are using the AIs for rather than just on separate training data) could be very useful.

Quick note: under distribution shift, online approaches (including training, but also other kinds of validation or iteration) could be very important even for concentrated threats. Because of this, I'd maybe consider online training infrastructure to be higher on the list of software infrastructure interventions (not very confident, though).

If it's easy enough to run, it seems worth re-training the probes exactly the same way, except sampling both your train and test sets with replacement from the full dataset. This should avoid that issue. It has the downside of allowing some train/test leakage, but that seems pretty fine, especially if you only sample like 500 examples for train and 100 for test (from each of cities and neg_cities). 

I'd strongly hope that after doing this, none of your probes would be significantly below 50%.

My rough guess for Question 2.1:

1. ^{^}

   The code itself would, of course, be the biggest example, but I’m not sure how relevant non-whitespace token length is for most formatting

2. ^{^}

   In particular, you’d need a lookup table of at least size $N\times M$, where $N$ is the longest single string you’d want to track the length of, and $M$ is the length of the longest token. I expect $N$ to be on the order of hundreds, and $M$ to be at most about 10 (since we can ignore a few outlier tokens)

3. ^{^}

   linear operations are pretty free, and addition of linearly represented features is as linear as it gets

[Note: One idea is to label the dataset w/ the feature vector e.g. saying this text is a latex $ and this one isn't. Then learn several k-sparse probes & show the range of k values that get you whatever percentage of separation]

 

You've already thanked Wes, but just wanted to note that his paper may be of interest here. 

If you're interested, "When is Goodhart catastrophic?" characterizes some conditions on the noise and signal distributions (or rather, their tails) that are sufficient to guarantee being screwed (or in business) in the limit of many studies.

The downside is that because it doesn't make assumptions about the distributions (other than independence), it sadly can't say much about the non-limiting cases.

Very small typo: when you define LayerNorm, you say $y_i={\sum }_j({\text{Id}}_{ij}-1_i{1}_j)x_j$ when I think you mean $y_i={\sum }_j({\text{Id}}_{ij}-\frac{1_i{1}_j}{768})x_j$ ? Please feel free to ignore if this is wrong!!!

I do agree that looking at $W_O$ alone seems a bit misguided (unless we're normalizing by looking at cosine similarity instead of dot product). However, the extent to which this is true is a bit unclear. Here are a few considerations:

• At first blush, the thing you said is exactly right; scaling $W_{in}$ up and scale $W_O$ down will leave the implemented function unchanged.
• However, this'll affect the L2 regularization penalty. All else equal, we'd expect to see $\parallel W_{in}\parallel =\parallel W_O\parallel$, since that minimizes the regularization penalty.
• However, this is all complicated by the fact that you can also alternatively scale the LayerNorm's gain parameter, which (I think) isn't regularized.
• Lastly, I believe GPT2 uses GELU, not ReLU? This is significant, since it no longer allows you to scale $W_{in}$ and $W_O$ without changing the implemented function. 

We are surprised by the decrease in Residual Stream norm in some of the EleutherAI models.
...
According to the model card, the Pythia models have "exactly the same" architectures as their OPT counterparts

I could very well be completely wrong here, but I suspect this could primarily be an artifact of different unembeddings. 

It seemed to me from the model card that although the Pythia models have "exactly the same" architecture, they only have the same number of non-embedding parameters. The Pythia models all have more total parameters than their counterparts and therefore more embedding parameters, implying that they're using a different embedding/unembedding scheme. In particular, the EleutherAI models use the GPT-NeoX-20B tokenizer instead of the GPT-2 tokenizer (they also use rotary embeddings, which I don't expect to matter as much).

In addition, all the decreases in Residual Stream norm occur in the last 2 layers, which is exactly where I would've expected to see artifacts of the embedding/unembedding process^[1]. I'm not familiar enough with the differences in the tokenizers to have predicted the decreasing Residual Stream norm ex ante, but it seems kinda likely ex post that whatever's causing this large systematic difference in EleutherAI models' norms is due to them using a different tokenizer. 

1. ^{^}

   I also would've expected to see these artifacts in the first layer, which we don't really see, so take this with a grain of salt, I guess. I do still think this is pretty characteristic of "SGD trying its best to deal with unembedding shenanigans by doing weird things in the last layer or two, leaving the rest mostly untouched," but this might just be me pattern-matching to a bad internal narrative/trope I've developed.