Background

(specific information will be sparse here. This is meant to give context for the Takeaways section of the post)

Our group (Garrett, Chu, and Johannes) have worked with John Wentworth in the SERI MATS ~~2 Electric Boogaloo~~ program for three weeks, meaning it's time for a Review & Takeaways Post!

First week was Project Selection, and the first day was spent thinking about strategies for coming up with good projects. We chose to find a general method for figuring out True Names of mathy-feely-concepts-in-your-brain (such as roundness, color decomposition^[1], or telling whether a piece of cloth is in a pile) with the goal that such a method would allow for figuring out true names for concepts like optimization, corrigibility, agency, modularity, neural network representations, and other alignment-relevant concepts.

Then we read Jaynes, and talked to TurnTrout, and concluded this project sucked. So we went back to Project Selection 2.0!

We came out of Project Selection 2.0 renewed with vigor, and a deeper understanding of the problems of alignment. Our new project was finding a better version of information theory by adapting logical induction or infra-Bayesianism.

Then we talked to Eliezer Yudkowsky, he asked for a concrete example of how this would solve alignment, and we didn't have a good example. So we went to Project Selection 3.0.

We came out of Project Selection 3.0 with even more vigor, and an even deeper understanding of the problems associated with alignment... and a clever idea.

Finetuning LLMs with RL seems to make them more agentic. We will look at the changes RL makes to LLMs' weights; we can see how localized the changes are, get information about what sorts of computations make something agentic, and make conjectures about selected systems, giving us a better understanding of agency.

Nobody has convinced us this is a bad use of our time, though we'd like to see people try.

Takeaways

Big ASS Tree

We learned lots of things over the course of figuring out all our ideas sucked. During the project selection phase we had a cool idea for a way to generate project ideas: The Alignment Safety Search Tree (ASS Tree for short). The idea comes from Mazes and Duality; the goal was to explore the space of problems and constraints before trying to propose solutions.

You start by writing "Alignment" up at the top of your whiteboard. This ...

brooksshowell

brooksshowell