By design, LLMs perform nonlinear mappings from their inputs (text sequences) to their outputs (next-token generations). Some of these nonlinearities are built-in to the model architecture, but others are learned by the model, and may be important parts of how the model represents and transforms information. By studying different aspects...
There have been multiple recent calls for the automation of AI safety and alignment research. There are likely many people who would like to contribute to this space, but would benefit from clear directions for how to do so. Stemming from a recent SPAR project and in light of limitations...
By design, LLMs perform nonlinear mappings from their inputs (text sequences) to their outputs (next-token generations). Some of these nonlinearities are built-in to the model architecture, but others are learned by the model, and may be important parts of how the model represents and transforms information. By studying different aspects of this nonlinear mapping, we can aim to understand how these models make sense of their inputs and respond to them.
An example of such learned nonlinearities is that of activation plateaus—a phenomenon where interpolating from one input sequence to another can produce minimal downstream changes for most of the interpolation, with a sudden rapid 'phase change' somewhere in the middle. This raises... (read 2134 more words →)
Very glad to hear you're okay!
Point Out Anything Suspicious
I've found this highly valuable. Doctors/nurses almost indubitably know more than I do, but also indubitably have way less time to obsessively stare at at scans and charts for hours on end.
And congrats on the wedding!
PIBBSS definitely does some mech interp, and I believe AI safety camp has some mech interp projects.
For people looking for MATS-like programs in other locations, with different timelines, etc. this page is a great resource for finding other training programs, a number of which (PIBBSS, Pivotal, LASR Labs, others) include mech interp research: https://www.aisafety.com/map
There have been multiple recent calls for the automation of AI safety and alignment research. There are likely many people who would like to contribute to this space, but would benefit from clear directions for how to do so. Stemming from a recent SPAR project and in light of limitations of current systems, we provide a brief list of concrete projects for improving the ability of current and near-future agentic coding LLMs to execute technical AI safety experiments. We expect each of these could be meaningfully developed as short-term (1 week to 3 months) projects.
This is in no way intended to be a comprehensive list, and we strongly welcome additional project ideas in the comments.
Note: Due... (read 2336 more words →)
Hello! Long-time lurker, planning to post research results on here in the near future. I'm a currently a PIBBSS research fellow, working on LLM interpretability relating to activation plateaus and deception probes. I'll be joining Anna Leshinskaya's Relational Cognition lab in the fall as a postdoc, working on moral reasoning in LLMs. Feel free to reach out if you have any ideas, questions, etc. on any of these topics!