This is a special post for quick takes by sophont. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Is alignment even possible? I know we all like to believe that it is, but given a model, and given that the model has the ability to modify itself, won't it drift out of control over time no matter what? I'm not sure if it's entirely plausible that alignment of models - regardless of architecture - will ever exist, because a model would have to be robust, both to its internal modification, and also to external modification i.e. be able to preserve its goals against adversarial actors. Is there any reason to think alignment of ASI is even possible under these conditions and should we even try? Maybe we should throw in the towel and advocate that it simply shouldn't be built.
I've called this The alignment stability problem. It's also closely related to the diamond maximizer problem.
My hope, and I think the common hope from people who think about this problem, is that a reflectively stable intelligence will take active measures to prevent its goals or values from changing as it learns.
I think it's pretty clear that this is the rational action from an intelligence that has a single, well-defined goal. But I think it's pretty unclear how this dynamic works if the intelligence has a number of loosely defined and competing goals or values. Humans and LLMs have this motivational structure. This hasn't been worked through thoroughly by anyone I know of.
One formal attempt to work this through is called tiling theorems, I believe.
I think the problem is solvable if you somehow map the mix of goals into one goal that is a well-defined combination of all the subgoals. However, that is pretty different from the types of alignment we are trying for in LLM agents, and probably will try for in LLM AGI. So, I think this is an open problem, among others, with our current path to alignment.
I wonder whether anyone has done a proper job of researching whether it's even possible to capture human preferences. Naively speaking, the mind body problem and the question of free will are unsolved so it would seem that depending on the answer to these questions, we may only ever be able to make a reasonable guess. And given the amount of information required to simulate the human brain, unless we have an incredible amount of compute, it seems unlikely we're able to deterministically predict (if this is universally possible) what people want in different scenarios prior to ASI. Is there sociology or psychology research that tries to evaluate the baseline minimum amount of information required to predict what people want in different contexts or in general? If I know someone's MBTI and Big 5, can I guess what they want better than coin flip odds in binary decisions? More generally, can I guess what they want regarding food, relationships, conversation, location, lifestyle, etc.? Marketing relies on answering these questions, but it seems somewhat shallow. Of course you can sell ice cream to a child because they love fat and sugar due to physiologically inbuilt drives. But can you predict in 10 years that they have become vegan due to their strong moral beliefs and now love edamame sprinkled with salt? Has anyone trained a preference prediction model or attempted to finetune a model to predict preferences? If anyone knows someone working on this, would love to get in touch.