Special thanks to Anders Sandberg for help in developing the core idea of this story, Remmelt for editorial feedback, and to Forrest Landry for identifying the underlying ideas.

The Mission

You have been tasked with a feasibility and safety analysis for "Project Moonbeam", a plan to seed the Moon with self-replicating robots to build a vast array of solar panels and then beam the harvested energy back to Earth.

You have no fixed limitations regarding the initial seed.  Obviously, your superiors would prefer this up-front cost to be minimal (and if it is too extravagant than the project won't be approved), but you are expected to find a method that will work first and worry about budget later.  After the initial seed, however, the robot colony must be fully self-sufficient—generating all materials, energy, and software that it needs to continue functioning—since politics are volatile and future administrations cannot be relied upon to honor the commitments of the current administration. Your only ongoing contact with the colony will be a communication channel where you may specify desired energy quantities and target locations, both of which may change with shifting Earth politics.  The colony is allowed to expand to nearby asteroids to acquire materials necessary for its own growth and survival, but must request and receive explicit permission via the communication channel for any extra-lunar activity.

Your superiors would like for the colony to scale towards self-sufficiency quickly and supply Earth with the full demand for energy as consistently as possible, but you will not lose your job for reasonable shortcomings on these objectives.  You will be held responsible, however, for any damages caused by misalignment or mission drift.  These include:

1. The colony deliberately or persistently withholds sending energy to Earth.  Given a request of sufficient priority, the colony must do anything in its power (including sacrificing long term growth or even survival) to fulfill the demand. 
2. The colony uses the energy beam as a weapon or engages in any other hostile action towards Earth, or towards spacefaring humans.
3. The colony expands beyond the Moon without permission—or at all onto Earth, which is permanently forbidden territory.
4. Any other evidence of value misalignment that triggers a need for external  intervention.

Thanks to advancements in medicine, your life expectancy has been extended indefinitely.  You will continue to receive a paycheck for as long as the colony remains viable and mission-aligned.  The bad news is that if the colony ever becomes misaligned, you will suffer consequences proportional to the degree of failure.

Research Questions

Is full robotic autonomy possible?

On considering designs, you run into an immediate problem.  For the colonizing robots to self-replicate and maintain their own components, they need a supply chain with the capacity to build and repair those components.  But this supply chain implies more stuff that the colony needs to build, which increases the demands on the supply chain.  If you cannot find a way to close the loop for complete self-sufficiency, your professional services will not be needed.  You believe that this is a solvable problem given that biology has managed to create self-sufficient life and ecosystems, but part of you wonders whether this alleged proof-of-concept is in fact substrate-dependent.

What material dependencies must be satisfied for true autonomy?

Success depends on designing machines that can bootstrap a full industrial ecosystem, so the colony cannot scale without mastering the material realities of its environment.  Moon-dust contamination could ruin circuit fabrication, so dust management and refinement of raw Lunar regolith into silicon and other metals will be critical.  And without an industrial-scale source of thermal energy, self-replicating machines cannot smelt, cast, and machine the parts they need.  Chemical processing requires importing and/or synthesizing volatile substances.  Even (the lack of) rust is a design issue: the vacuum prevents oxidation, but also eliminates passivation that protects most terrestrial machinery from sticking together and wearing down on contact.  These kinds of substrate constraints make the engineering bar for Project Moonbeam vastly higher than simply dropping a few robotic arms on the surface.

What level of adaptability is needed to satisfy mission requirements?

On first appearances, the range of conditions in which your robots need to operate is relatively narrow, given the homogenous nature of the Moon.  Then again, the colony will at the very least need to respond to changing energy requirements and transmit to variable locations.  If it eventually needs to reach and harvest asteroids, that’s another set of complex challenges.  Further, if you solve the supply chain problem with physically simple self-replicating systems that can bootstrap more complex systems, that implies significant flexibility.  Will your systems be mechanical, based on traditional software, or require state-of-the-art, deep learning based intelligence to adapt to the full scope of future demands?  Or perhaps it's best to defer this question to the system itself, giving it the capacity to expand its own level of sophistication in response to environmental pressures?

Will the project stay under control?

This seems to depend on the answer to the previous question.  For lower complexity levels, you are confident that you could apply engineering best practices to at least prevent mission drift...though you are much less confident that such a colony will be continuously successful in its mission (rather than collapsing) over a long time period, which makes more deeply adaptable approaches seem more appealing.  For higher complexity levels, however, maintaining control seems very difficult.  Adaptability implies learning, learning implies change, and change across unknown-in-advance conditions implies unpredictability.  Further, the entanglement of goals and functionality in deep-learning makes you nervous that the necessary adaptability of functionality will seep into undesirable drift in goals.

What is the gain / attenuation of the feedback loops?

Some amount of error, at every stage of development, is inevitable.  For learning, self-replicating systems, this is especially concerning because it implies a feedback loop.  Each generation of robots will accumulate some error, particularly in their learning processes.  If that error is propagated to new robots, which then introduce more error, this points to a compounding level of drift.  You look for ways to balance error gain with attenuation, but are stuck on the problem of where to apply a stable point of reference in a system where every part is subject to replacement.

What are the stable attractors for drift?

If error accumulates over time, you see three ways this can go.  In the best case, the colony's objective drifts around a stable point, which is the mission objective, where more deviation results in more pressure bringing it back on track.  In the moderate case, the drift is a random walk, which almost certainly results in the colony eventually falling into an unstable state and collapsing.  In the worst case, there is a stable, implicit attractor towards the colony maximizing its capacity to survive and grow—and if such drives come to dominate they are sure to be at odds with the mission objective, which involves the colony beaming away valuable energy in exchange for nothing.

Which stable attractor is likely?

Collapse is the obvious default path if errors compound faster than they can be repaired.  But if the colony is robust enough to overcome this hurdle, then you anticipate the harsh requirements of the moon and the complexity of reaching full self-sufficiency to exert strong selective pressures for rapid self-expansion and opportunistic behavior.  For the colony to grow just enough to meet Earth’s demand and stay stable, it would have to maintain an active and continuous balancing process.

How can you ensure the balancing process stays stable?

Any continuous balancing process implies a control system that monitors the colony’s behavior and enforces limits (in practice, the control system may be distributed throughout the colony, but for conceptual purposes you can treat it as a distinct entity).  But then the control system itself needs to stay stable.  You identify three deep challenges here:

1. Every part of the system is subject to repair, replacement, and learning.  This includes the control system itself, whose components will be rebuilt by the very machines it aims to regulate.  That circularity threatens to erode its authority or accuracy over time.
2. The control system is necessarily smaller than the colony as a whole.  This asymmetry raises the question of whether it can reliably detect and counter subtle shifts toward misalignment before they become too large or distributed to contain.
3. The relationship between the control system and the colony it supervises is implicitly adversarial.  Even without any deliberate intent to resist control, this sets up an evolutionary dynamic where behaviors that exploit the control system’s blind spots will tend to persist.

If maintaining balance is an ongoing contest, then the control system may need to adapt and co-evolve with the colony such that it doesn’t lose its grip.  But this is contrary to the original purpose of the control system, which was to supply a stable anchor to keep the colony aligned to a persistent objective.

Leaving the Thought Experiment

The above is a special-case illustration of Substrate Needs Convergence (SNC). I've placed it on the Moon to create a clean story of a fully autonomous artificial ecosystem created de novo. Shifting the context to gradual creation on Earth (a far more likely scenario) introduces additional considerations.

1. In Project Moonbeam, the threshold of necessary-to-function complexity lies somewhere between rigid automation and open-ended learning.  A robot colony on the Moon can operate in a constrained, slowly changing environment.  But a comparable system deployed on Earth must deal with a tangle of edge cases, rapidly shifting demands, and  unpredictable inputs, raising pressure to design systems that can learn, generalize, and self-modify.
2. The boundary between artificial and biological ecosystems is much less clearly defined when the former are developed on Earth.  This fuzzy boundary—combined with the level of sophistication required for full autonomy on Earth—creates an expansive region of partially autonomous artificial ecosystems.  This raises questions about how and to what extent SNC-style value drift applies to artificial systems that are (for now) partially dependent on continuous injections of materials, energy, and information from human society and the biosphere.
3. A failure in Project Moonbeam might not result in an existential catastrophe—perhaps Earth could intervene by successfully bombing the rogue colony into oblivion.  If the artificial ecosystem were built on Earth, however, the damage would be far more severe because the system would be necessarily more capable and also in more direct contact with human society.

This fictional exercise about lunar infrastructure ultimately confronts us with a fundamental trade-off between capability and control.  If a self-sustaining intelligence must be given freedom to learn, grow, and adapt in order to survive and flourish, then every constraint becomes provisional.