I will argue that the outcome we should aim to achieve from aligned AI is Hedonium: the conversion of the universe into the densest possible packing of positive valence. Valence represents the preferability of a given moment, as though we asked you to compare which moments of life you would like to live again and then numbered them such that you predictably preferred to live higher-numbered moments. Said another way, valence is a measure of happiness in its most exalted form, adjusting for the reductive implications of the word “happy.” Hedonium can be analogized to Computronium, but with computation replaced by positive valence. Thus, one image of what Hedonium might look... (read 1627 more words →)