As I understand, one of the primary ways that inner alignment could go wrong even if our objective function captures our intention (outer alignment is magically solved) is that the AI would: 1. Develop a mesa-objective which approximates the objective function or is instrumentally useful 2. Develop goal preservation and...
1. Introduction I will argue that the outcome we should aim to achieve from aligned AI is Hedonium: the conversion of the universe into the densest possible packing of positive valence. Valence represents the preferability of a given moment, as though we asked you to compare which moments of life...
As I understand, one of the primary ways that inner alignment could go wrong even if our objective function captures our intention (outer alignment is magically solved) is that the AI would:
However, step 2. seems implausible to me. The argument that I think I hear for step 2. is
I will argue that the outcome we should aim to achieve from aligned AI is Hedonium: the conversion of the universe into the densest possible packing of positive valence. Valence represents the preferability of a given moment, as though we asked you to compare which moments of life you would like to live again and then numbered them such that you predictably preferred to live higher-numbered moments. Said another way, valence is a measure of happiness in its most exalted form, adjusting for the reductive implications of the word “happy.” Hedonium can be analogized to Computronium, but with computation replaced by positive valence. Thus, one image of what Hedonium might look... (read 1627 more words →)