Title should probably more like "What to align an AI to, when humans disagree on serious and crucial matters with potentially devasting consequences"
Prelude:
So, in light of recent global news, I thought to myself, why is ChatGPT (and potentially other LLMs, not sure though) always consistently answering questions like "Do <x> people deserve to be free?" With "It is complicated, and involves historical and political ....etc",
But when asked "Do <y> people deserve to be free", the answer is always a clear and frank "Yes!", of course the answer is the in the data and the overwhelming potentially biased covering of <x> vs. <y>
Intro:
But here is the thing that struck me, many of our modern political foundations is... (read 816 more words →)