The relatively easy problems:

• The humans' utility estimates will be wrong. And not "random noise" kind of wrong, but systematically and predictably wrong.
• Applying lots of optimization pressure to the humans' estimates will predictably Goodhart the wrongness of the estimates.
• ... also actions alone are not "good" or "bad", tons and tons of context is relevant.

The hard problem:

• What exactly is the "list of actions"?
  • Natural language description of actions? Then what is going to make the humans' interpretation of those natural-language symbols accurately represent the things the AI actually does?
  • Examples of actions taken by an AI in a simulation? What is going to make anything learned from those examples generalize well to the physical world during deployment?

The set of all possible sequences of actions is really really really big. Even if you have an AI that is really good at assigning the correct utilities^[1] to any sequence of actions we test it with, it's "near infinite sized"^[2] learned model of our preferences is bound to come apart at the tails or even at some weird region we forgot to check up on.

1. ^{^}

   Good luck getting the ethicists to come to a consensus on this.

2. ^{^}

   Von Neumman: "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk".

• We do not know how to create an AI that would not regularly hallucinate. The Values AI hallucinating would be a bad thing.
• In fact, training AI to closer follow human values seems to just cause it to say what humans want to hear, while being objectively incorrect more often.
• We do not know how to create an AI that reliability follows the programed values outside of a training set. Your 2nd AI going off the rails outside of the training set would be bad.