I am glad that a term like "bounded-rational" exists. If it's been discussed someplace very thoroughly then I likely don't have very much to add. What are some proposals for modeling bounded Bayesianism?
I think what I'm saying is consistent with your bullet points, but I would go further. I'll focus on one point: I do not think it's possible for a bounded agent to be epistemically pure, even having sacrificed most or all of its instrumentally rational capability. Epistemic impurity is built right into math and logic.
Let me make the following assumption about our bounded rational agent: given any assertion A, it has the capability of computing its prior P(A) in time that is polynomial in the length of A. That is, it is not strictly agnostic about anything. Since there exist assertions A which are logical consequences of some axioms, but whose shortest proof is super-polynomial (in fact it gets much worse) in the length of A, it seem very unlikely that we will have P(A) > P(Axioms) for all provable assertions A.
(I think you could make this into a rigorous mathematical statement, but I am not claiming to have proved it--I don't see how to rule out the possibility that P always computes P(A) > P(Axioms) (and quickly!) just by luck. Such a P would be very valuable.)
I believe you are correct. A bounded rational agent that is not strictly agnostic about anything will produce outputs that contain logical inconsistencies.
For a superintelligence to avoid such inconsistencies it would have to violate the 'not strictly agnostic about anything' assumption either explicitly or practically. By 'practically' I mean it could refuse to return output until such time as it has proven the logical correctness of a given A. It may burn up the neg-entropy of its future light cone before it returns but hey, at least it was never wrong. A bounded rational agent in denial about its 'bounds'.
This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.