Cross-Model Activation Generalizability Isn't Strong (Yet)
TL;DR * Tested activation similarities across different LLM families (Llama, Gemma, Qwen, Pythia) at small scale (1~3B) * CKA Similarity : Cross-architectural activation similarity is statistically real, but weak. Within-family activations are much stronger (4~9x) * Linear Transferability : Trained linear bridges for linear activation transfers for binary classification and...
Apr 66