Just wanted to say this question pinged my "huh, this is a neat question" detector. (I consider myself pretty confused about the broader topic so don't have good answers, but found your question neat because it tackled a problem at a level that where it still personally feels meaty to me)
Doublechecking my understanding of your "implied question" – it sounds like what you want is a reward function that is simple, but somehow analogous to the complexity of human value? And it sounds like maybe the underspecified bit is "you, as a human, have some vague notion that some sorts of value-generation are 'cheating'", and your true goal is "the most interesting outcome that doesn't feel like Somehow Cheating to me?"
Some thoughts I have at the moment:
If I'm taking your current (implied?) assumptions of "one cell == minimum viable value-weight", and "the goal is to have as simple a function you can that sort of 'feels like it's getting at something analogous to human value", I think the answer of "maximize the number of unique states that happen before things start looping" (maybe with a finite board, so that gliders-guns can't "game" the system by generating technically infinite 'variety'?)
In this case it might mean that the system optimizes either for true continuous novelty, or the longest possible loop?
I do suspect that figuring out which of your assumptions are "valid" is an important part of the question here.
Here's an inelegant answer that might work as a good proxy:
Hand-code a list of substructures that you identify as "good:" Gliders, glider guns, maybe some interesting shapes, etc. Make this list at least 10 items long.
Score is the product of the total numberXduration of each type of substructure. Use a large but finite grid, with a large but finite time.
I think this answer is better than it sounds. I think that value is complex; flourishing is not something that should be possible to specify simply in the real world, so why should it be possible in GoL? Moreover, flourishing in the real world involves a combination of order and disorder: Macro-structures like people and societies and spaceships (order) but they have to be different from each other and combined and recombined in lots of new ways (disorder).
I think one criticism of this answer is that it too could potentially be "gamed" by some big grid of densely packed substructures from the list. However, in practice I don't think this will be a problem because I doubt it will be possible to construct such a grid. Nevertheless I think thinking more about this question for an hour would produce a better answer.
Some friends tried (inconclusively) to apply AlphaZero to a two-player GoL. I can put you in touch if you want their feedback.
I'm thinking of designing a reinforcement learning environment based on Conway's Game of Life (GoL). In it, at every timestep, an agent can change the state of some cells.
As is the case with most interesting RL problems, agent behaviour would be determined by the reward function.
In this scenario, I see some issues with simple reward functions:
1) Total life:
Something like this glider gun would represent a technically correct unbounded score, with a stream of lonely travellers in an interstaller abyss. Say we want to stick to a finite memory.
2) Highest density:
This is generation 50 of the Max spacefiller. The agent might find some way of maintaining the striped pattern from the middle. Uninspiring, to say the least; like
***spoiler alert*** being batteries in The Matrix.
Other reward functions might consider variation over time.
3) Some function with a penalty for static life:
Consider Karel's p177:
This oscillator has a period of 177 time steps. There's another one with p312, but I couldn't find a good-enough gif of it. Such patterns would likely game this reward specification too.
I've thought of a few other simple functions, which all look flawed in some obvious ways.
That's not to say there is some ungameable reward function. But I wonder, if each cell symbolically represented some small unit of sentience (say a single person, or a family, or a planet) what would flourishing look like?