Computronium is a "theoretical arrangement of matter that is the most optimal possible form of computing device for that amount of matter. "

Relevance to Friendly AI

In a thought experiment similar to the Paperclip maximizer, if an artificial general intelligence has a terminal value (end-goal) which to make a pure mathematical calculation like solving the Riemann Hypothesis, it would convert all available mass to computronium (the most efficient possible computer processors).

In fact, a similar outcome would also apply to many other goals: So long as greater optimization power can be boosted with more computing power, and so long as dedication of resources to creating computronium does not detract from the goal (e.g., by taking up matter, time, or effort that can better be used in other ways), computronium may be valuable to attaining the goal. A purely mathematical goal, like proving the Riemann Hypothesis, is completely focused on computation and so most directly illustrates the concept.

Theories that valorize intelligence as such (such as that of Hugo de Garis [2005], or Eliezer Yudkowsky before 2001) may consider the conversion of all matter to computronium (running an AGI) to be a positive development, as this would provide the most powerful possible infrastructure for intelligence.

References

Hugo de Garis, 2005. The Artilect War: Cosmists vs. Terrans: A Bitter Controversy Concerning Whether Humanity Should Build Godlike Massively Intelligent Machines. Etc Publications.