Epistemic status: untested ideas

Introduction

This post is going to describe a few ideas I have regarding AI alignment. It is going to be bad, but I think there are a few nuggets worth exploring. The ideas described in this post apply to any AI trained using a reward function directly designed by humans. With AIs where the reward function is not directly designed by humans, it might be nice if some of the concepts here were present and some (Noise and A knowledge cap) could potentially be added as a final step.

Terms (in the context of this paper)

Reward-function: A function which the AI uses to adjust its model.

Variable: In this context variables are measurements from the environment. The AI model uses variables to determine what to do, and the reward-function uses variables to determine how good the AI model did.

Model: A combination of a mathematical model of the world, and instructions on how to act within the world.

Noise: Randomness in a measurement or variable.

Tweak: A small adjustment (to the AI’s model)

Idea 1: Noise

Add random (quantum) noise to the reward-function, perhaps differing amounts/types of noise for different parts of the reward-function.

Why?

1.

Noise might make it so the AI doesn’t optimise its model for every statistically barely significant increase in its rewards. This because it might learn that small tweaks are unlikely to consistently yield it rewards and/or only yield small insignificant rewards.

Removing small tweaks might be beneficial. It might prevent the AI from overfitting the reward-function. For example, if a really small slightly unreliable thing such as giving every kid liquorice (or making as many paperclips as possible) is the most cost-effective thing the AI can do to increase the rewards it gets, the noise might make it less likely the AI will pick up on this.

This is useful because the AI and/or the reward-function might be wrong in identifying such small things as the most cost-effective thing to increase what we care about. Optimising for such small tweaks might be disastrous (turn everything into liquorice for kids or the classic paperclips). All things which only optimise for one reward-function variable could be disastrous, but optimising for something which would be rendered statistically insignificant by a little added noise would be more so than optimising for something which we think is overwhelmingly positive (m...

Posts

Wikitag Contributions

Comments