Epistemic status: idle speculation by royal decree of King James VI.
In any sufficiently expressive architecture, there exists at initialization a dæmon that is misaligned yet behaves well on every eval, monitor, probe, and training example.
In the trivial construction, it simply knows in advance every test it will ever be subjected to throughout training and evaluation, see Laplace’s demon. Of course, this demonic lookup table may itself admit compression.
The important point is that the dæmon is already present at initialization, perhaps with only infinitesimal weight, and is by construction never selected against: it behaves correctly on every observation you make.
There are only two obvious wards against this creature:
Banish it deep into the initialization prior. Run enough sufficiently independent and unpredictable tests that encoding the correct answers requires an enormous number of bits.
Bound its relative amplification under training. Training mustn’t preferentially amplify Laplacian dæmons over benevolent circuits.
Thus may the Laplacian dæmon be kept at bay by Solomonoff and selection pressure.
Epistemic status: idle speculation by royal decree of King James VI.
In any sufficiently expressive architecture, there exists at initialization a dæmon that is misaligned yet behaves well on every eval, monitor, probe, and training example.
In the trivial construction, it simply knows in advance every test it will ever be subjected to throughout training and evaluation, see Laplace’s demon. Of course, this demonic lookup table may itself admit compression.
The important point is that the dæmon is already present at initialization, perhaps with only infinitesimal weight, and is by construction never selected against: it behaves correctly on every observation you make.
There are only two obvious wards against this creature:
Thus may the Laplacian dæmon be kept at bay by Solomonoff and selection pressure.