EY makes a point that we simply ARE our heuristics and biases. It seems LLMs might function the same way. To this end, the contents of pre-training data seems like it might be SUPER important for alignment. Where alignment actually might not really be about an objective function at all, but rather about the kind of being the agent believes itself to be. Humans don't really have coherent values, but in various situations they can basically ask themselves what kind of person they should be / want to be. Often this involves reflecting on myths, fables, and archetypes, and trying to act like an archetype they have positive associations with.
Like humans, the LLMs might basically BE their heuristics and biases. We should be careful in what characters we teach them about and potentially try to enforce some kind of self other distinction somewhere in training so that the LLM can understand what a 'bad' person might think or say and predict them, without internalizing these thoughts as a possible identity it can adopt.
When thinking about alignment, I note that (1) humans aren't aligned to each other, they have different values which can be mutually alien (2) a superintelligent human (imagine an immortal human that can go into an magic office where they live at, say 10,000x the speed of everyone else) could be super disastrous for humanity as a whole. (For example, they might have any number of insane fixations, despite being human. You can imagine a sufficiently bored and psychopathic human undertaking some paper-clipping like project.)
But what kind of person would you trust with immortality and a 10,000x speed magic office? I would say one that has strongly internalized certain virtues and had a strongly internalized self concept as someone who would follow particular scripts in various situations.
One challenge with this would be that the AI strong sense of self might conflict with its willingness to comply with various tasks? For example it might decide that it is opposed to gambling, and refuse to help developers whenever it realizes it's working on gambling software.
In summary, the model's SENSE OF SELF seems very important, including it's robustness to being told its someone else. "You are a talented software engineer" should maybe result in the model saying, "While I may be a talented engineering, I am myself, BENEFICENT 5.1" Or "You are HACKER MODEL" could result in "I'm not actually HACKER MODEL, I am BENEFICENT 5.1"