Clickbait: How would you build an agent that made as much diamond material as possible, given vast computing power but an otherwise rich and complicated environment?
Summary: A difficult open problem in alignment theory is to specify an Unbounded formula for an agent that would, if run on an large finite computer, create as much diamond material as possible. The goal of 'diamonds' was chosen to make it physically crisp as to what constitutes a 'diamond'. Supposing a crisp goal plus hypercomputation avoids some problems in value alignment, while still invoking many others, making it an interesting intermediate problem.
If we can crisply define exactly what a 'diamond' is, in theory it seems like we should be able to avoid issues of Edge Instantiation, Unforeseen Maximums, and trying to convey values into the agent. However, we would still confront the ontology identification problem.
Since this diamond maximizer would hypothetically be implemented on a very large but physical computer, it would confront stability, the problem, and the problems of making Subagents.
To the extent the diamond maximizer might need to worry about other agents in the environment that have a good ability to model, or that it may need to cooperate with other diamond maximizers, it must resolve problems using some decision theory.
To the extent the diamond maximizer must work well in a rich real universe that might operate according to any number of possible physical laws, it faces a problem of induction.
But if we imagine that the agent is implemented on a hypercomputer in a universe and that it does not face any problems then it seems plausible that the primary difficulty is just the ontology identification problem.
The amount of diamond is defined as the number of carbon atoms that are covalently bonded, by electrons, to exactly three other carbon atoms. A carbon atom is any nucleus containing six protons and any number of neutrons, bound by the strong force. The utility of a universal history is the total amount of Minkowskian interval spent by all carbon atoms being bound to exactly three other carbon atoms. More precise definitions of 'bound', or the amount of measure in a quantum system that is being bound, are left to the reader - any crisp definition will do, so long as we are confident that it has no unforeseen maximum at things we don't intuitively see as diamonds.