One assumption that stands out to me as a little questionable is the idea that Cindy will, with infinite simulated time to think, eventually manage to come up with a solution to the alignment problem. (This is compounded by the fact that she's regularly brain-wiped and can only preserve insights by cramming them into the 1 gigabyte of scratch paper afforded to her.)
It just occurred to me that this post serves as a fairly compelling argument in favor of a modest epistemology, which in 2017 Eliezer wrote a whole book arguing against. ("I think I'm doing this for the good of the tribe, but maybe I'm just fooling myself" is definitely an "outside view".) Eliezer, have you changed your mind since writing this post? If so, where do you think your past self went awry? If not, how do you reconcile the ideas in this article with the idea that modest epistemology is harmful?
Can I get a link or two to read more about this incident?