This is a special post for quick takes by No77e. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Alignment seems quite similar to the problem of imbuing AIs with artistic taste. Morality and taste are both hard to verify and subjective (or inter-subjective). Alignment has in practice the further difficulty that deception may play a role. I.e., even after managing to train moral principles into an AI system, you have to make sure they actually act as a guide for action.
That said, my very subjective impression is that AI is far ahead in terms of ethical taste compared to artistic taste. Perhaps this is thanks to the fact that alignment has been considered a core AI problem for a much longer time.
"make the world better!" <-- doing what this operator wants requires some aesthetic sensibility.
1Zach Stein-Perlman
You missed "trying." Succeeding requires certain capabilities, but trying to do it does not. I believe there's much more risk from AIs not trying to do what the operator wants than trying and failing.