This is a suggestion of a reinforcement learning model with an additional conflicting dynamic between optimized data and loss function. That conflict is intended to reduce RL’s intrinsic Omohundro x-risk dynamic. Thus, it’s also an attempt to produce, though without details, a good idea for AI safety, and so as a test for the meta-usefulness of my posting here. That’s following the idea.

The Idea

A modification of reinforcement learning: to have the learning agent develop de sui a function for optimization which is to supplant the loss function, at a given reduction of the loss function, by the action of the learning process. That is, if we correlate the loss function not with arbitrary data to be optimized, but with an optimization of some systemic data which has, or is known to have, a pattern, and that pattern requires for optimization certain “behaviors” (e.g., optimizing which is against some “aversiveness”, (entropy, say), that incorporates the agent as part of the dataset so it too must act, and act on itself, to reduce “aversiveness”) – then the optimizing agent can be expected at length to opt for aversive reduction (e.g., entropy-reduction) so as to reduce the associated loss function, in the process it will identify systemically the aversive data-point’s features, and how to elide them, as was done in identifying eye- and ear-forms (pixels of an eye as a whole structure are easier to spot than sequential pixels that make an eye). If, then, seizing control of the loss function will increase systemic “aversiveness”, as occurs since loss function is now tied to a non-anthropic, non-random environmental “constant” being optimized for, not human wants – then the agent will not seize it. Since, it’s caught between RL’s wireheading drive, and the fact that behaving in that way makes the situation more chaotic and harder to optimize for, since the data is already patterned, only the agent, prior to optimizing, does not know that. What is more chaotic and harder to optimize for, actually increases the loss function output.

And in a sentence: whether it’s feasible to include the loss function, the fact of the loss function, and what constitutes the function, including the agent, as part of the dataset being optimized for. Then abandoning the optimization, to wirehead, is inhibited, and recursive self-improvement and self-defe...

Posts

Wikitag Contributions

Comments