Very nice post, thank you for the mathematical rigor!
I do have a suggestion for improvement, as I'm a bit concerned that making reserved powers a list rather than a function could present some problems, since it's very difficult to enumerate every possible thing that could go wrong. (The posts's proposed alternative of letting the agent learn about the reserved powers list is also risky without safeguards such as the ones I propose at the bottom of this comment.)
Here are a few potentially problematic scenarios:
1. Storage cleanup. An AI that has been task... (read more)
Very nice post, thank you for the mathematical rigor!
I do have a suggestion for improvement, as I'm a bit concerned that making reserved powers a list rather than a function could present some problems, since it's very difficult to enumerate every possible thing that could go wrong. (The posts's proposed alternative of letting the agent learn about the reserved powers list is also risky without safeguards such as the ones I propose at the bottom of this comment.)
Here are a few potentially problematic scenarios:
1. Storage cleanup. An AI that has been task... (read more)