A vast amount of data on what humans value in an enormous range of both low stakes and high stakes situations is all through the training set. Fiction in particular has a very wide distribution of high stakes situations (it's often rich in ones rare in the real world). Any decent-sized LLM base model already has a huge amount of data about human values across a very wide training distribution. RL alignment doesn't need to instill knowledge of this from scratch: it merely needs to tune it up and get the assistant persona to actually act on it appropriately — rather than encourage it to hack the task grader for maximal reward.
The main problem we're currently having is not aligning the models, it's avoiding RL reasoning training unaligning them again. Reasoning training uses many millions of training environments, and the frontier labs (and subcontractors working for them) have assembled all of these in a great hurry. Some are broken and insecure, so that rather than teaching the AI to do the task, they instead teach it to cheat on the test. In humans, willingness to cheat on tests is anticorrelated with fine upstanding moral behavior, so reward hacking generalizes to emergent misalignment. This isn't a theoretical research problem any longer (that research was published a year or two ago), it's just a quality control problem: we understand why broken RL training examples teach bad behavior, the labs are simply having trouble finding and fixing/discarding all of them, and avoiding generating more. As the recent OpenAI Hugging Face hacking issue and the reports on it make very clear, (at least) OpenAI was not sufficiently prioritizing doing this (if you're not already familiar with that, Zvi has done some excellent analysis of all the reports).
Here is a question worth asking at least once: Why can't we just solve alignment by doing RL where the reward is exactly equal to our own utility function?
Now, there are some implementation concerns here. For example, we don't actually know our own utility function. And even if we did, we might have difficulty evaluating it. And in real life, it's just a fact right now that the majority of RL updates in SOTA model training are for solving problems, not for acting ethically.
But I will pose another concern here. Assume we do know our own utility function and can calculate its value for any given outcome. Even so, at least one of the following requirements must be satisfied for the above alignment plan to work:
In other words, if our plan for getting our utility function into the AI involves learning from examples in the typical RL way, we need to either be able to generalize from small examples to large ones, or to include large examples during training. And if we choose to provide large examples, they need to be sufficiently representative of the real world. Either because they are from the real world, or because they are a close enough imitation.
The list is in order from most preferable to least preferable. (1) becomes more viable as the AIs get smarter, (2) gets less viable, and (3) is a non-starter from the very beginning (kinda the whole thing with value is that you don't want to lose large chunks of it if you can possibly avoid it).
I don't think current algorithms can extrapolate well from low variance of value to high variance of value. As one example, if we consider RL methods that use a learned value function, we should not expect a neural network to correctly produce outputs orders of magnitude larger than it ever did in training.
So, even requirement (1), which is our best option, is one we should expect to fail. This is why I don't think this kind of plan will work.