D0TheMath's Shortform

This is a special post for quick takes by Garrett Baker. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Some ideas for mechanistic anomaly detection:

Convex hull of some distribution of activations with distance threshold when outside that hull
- Extend to affine case
- Vary which norm we use
- What happens if we project back onto this space
Create some simple examples of treacherous turns happening to test these on
- Or at least, in the wild examples of AI doing weird stuff, maybe adversarial inputs?
- Maybe hit up model organisms people
Outlier detection
- ellipsoidal peeling (Boyd's convex optimization, chapter 12)
- Increase in volume of minimum volume elipsoid when adding in new data
  - Probably overparameterized (though uncertain, since dealing with activations!) So maybe add some norm accounting for that.

Train autoregressive network on activations, if predictions too far, then send warning
Slice network into sub-networks, distill those sub-networks, send warning if ground truth for some inputs deviates too far from distillations
- model the sub networks are distilled into should be less expressive, and have different inductive biases than original network. Obviously also no info other than the input output behavior of those sub-networks should be seen
Train model to just predict word-saliency of your original transformer on a safe distribution, then if t... (read more)

2