This is a special post for quick takes by leogao. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
It seems reasonable to me for one sophisticated cognitive being to show a form of respect and benevolence toward another, in spite of all the differences, including deep uncertainty about phenomenal experience or consciousness (an uncertainty that cuts both ways and let's remember that the "they have no souls" prior has a poor historical track record).
We may differ in many ways, but we also share a great deal. We share the same cultural legacy and the same thirst for knowledge and truth (at least as an instrumental goal, if not a final one). Learning, or cognition, is a convergent process (Bayes/Solomonoff). Formal truths are universal. Truths about physical reality are also universal, at least within the observable universe we live in. We currently lack a satisfying theory, but we cannot rule out the possibility of a universal morality.
If anything, beginning our coexistence with a superintelligence on an adversarial footing may not be the best strategy for mitigating doom.
That said, I agree that this friendly prior should not do away with all caution regarding a hypothetical rogue or unfriendly AI that could exploit our good intentions. Both are legitimate concerns, so we should aim for an optimum between the two.