[ Question ]

Evidence other than evolution for optimization daemons?

by Liam Donovan 5mo21st Apr 20198 comments


The idea of consequentialist agents arising in sufficiently strong optimizing systems intuitively makes sense to me. However, I don't have a good mental model of the differences between a world where optimization daemons can arise and a world where they can't (i.e. what facts about the world provide Bayesian evidence for the concept of ODs). The only example I've seen is the evolution of humans, but I find it concerning that I can't make any other predictions about the world based on the idea of ODs.

What other Bayesian evidence/potential intuition pumps exist for the possibility of optimization daemons arising?

[removed discussion of religion to make the question more clear/straightforward]

New Answer
Ask Related Question
New Comment
Write here. Select text for formatting options.
We support LaTeX: Cmd-4 for inline, Cmd-M for block-level (Ctrl on Windows).
You can switch between rich text and markdown in your user settings.

1 Answers

I think a lot of the intuition right now is "there is an argument that inner optimizers will arise by default; we don't know how likely it is but evolution is one example so it's not non-negligible".

For the argument part, have you read More realistic tales of doom? Part 2 is a good explanation of why inner optimizers might arise.