This point cannot be convinced by intuition, nor can it be verified through experiments. Having strategic thinking to "conceal doing bad things" might be a necessary foundational ability for a sufficiently powerful AI, such that only AIs with this strategic thinking could perform tasks that are "slightly-less-impressive-looking" Perhaps this kind of thinking is a phase transition, or part of some phase transition, not a continuous change and thus not continuously observable. The reverse could also be true, and I don't have any compelling intuitions or evidence indicating which possibility is more likely.
This point cannot be convinced by intuition, nor can it be verified through experiments. Having strategic thinking to "conceal doing bad things" might be a necessary foundational ability for a sufficiently powerful AI, such that only AIs with this strategic thinking could perform tasks that are "slightly-less-impressive-looking" Perhaps this kind of thinking is a phase transition, or part of some phase transition, not a continuous change and thus not continuously observable.
The reverse could also be true, and I don't have any compelling intuitions or evidence indicating which possibility is more likely.