[ Question ]

Evidence other than evolution for optimization daemons?

byLiam Donovan1mo21st Apr 20198 comments


The idea of consequentialist agents arising in sufficiently strong optimizing systems intuitively makes sense to me. However, I don't have a good mental model of the differences between a world where optimization daemons can arise and a world where they can't (i.e. what facts about the world provide Bayesian evidence for the concept of ODs). The only example I've seen is the evolution of humans, but I find it concerning that I can't make any other predictions about the world based on the idea of ODs.

What other Bayesian evidence/potential intuition pumps exist for the possibility of optimization daemons arising?

[removed discussion of religion to make the question more clear/straightforward]

New Answer
Ask Related Question
New Comment

1 Answers

I think a lot of the intuition right now is "there is an argument that inner optimizers will arise by default; we don't know how likely it is but evolution is one example so it's not non-negligible".

For the argument part, have you read More realistic tales of doom? Part 2 is a good explanation of why inner optimizers might arise.