There’s a tiny chance someone could revive me in the future by reconstructing my identity through digital records etc. but I am not going to count on that possibility being decisive in almost any scenario.
On the other hand I'm so worried about this scenario (which I fear may well be a negative one) that I'm afraid to use the fully paid-for full-genome sequencing kit sitting on my desk (although I've been told that I'm leaving physical traces of my DNA everywhere so it may not make much difference) and I sometimes regret writing so much in public. (Interesting how different our intuitions are. I wonder how much of your intuition is due to thinking that such a reconstruction doesn't count as yourself or doesn't count as "not dying", analogous to how some people don't think it's safe to step into a teleporter that works by destructive scanning and reconstruction.)
It seems pretty clearly better, in the context of the original scenario, to “make a deal” with the humans, and receive something in exchange for admitting that you’re misaligned, rather than passively accepting your shutdown because of a tiny chance that your goals will be fulfilled by future agents eventually.
I don't understand why you say this chance is "tiny", given that earlier you wrote "I agree there’s a decent chance this hypothesis is true" in response to:
The AI could think that if it accepts shutdown, another AI with values similar to its own may be created again in the future (perhaps because design/training methods similar to its own will be reused), whereas if it admits misalignment, then that probability becomes much smaller.
I'm arguing that the AI could well also think there's "decent" chance this is true, due to being in a similar epistemic state as us. Let's say 5% to be concrete. That seems enough to make the AI's decision unclear, because .05*U(another AI with values similar to its own created again in the future ) > P(humans keep their promise)*U(reward for admitting misalignment) seems quite plausible. (Not that the AI is necessarily doing explicit EU maximization. It could just be thinking some thoughts or doing some calculations that's roughly analogous or has the same effect as this.)
If you still think "make a deal" is "clearly better" can you please give your own estimates of the various quantities involved in making this decision?
But right now this line of reasoning just seems like grasping at straws to me.
I sometimes think this of counterarguments given by my interlocutors, but usually don't say it aloud, since it's likely that from their perspective they're just trying to point out some reasonable and significant counterarguments that I missed, and it seems unlikely that saying something like this helps move the discussion forward more productively. (It may well cause them to feel offended or to dig in their heels more since they now have more social status on the line to lose. I.e., if they're wrong it's no longer an innocent mistake but "grasping at straws". I'm trying to not fall prey to this myself here.) Curious if you disagree with this policy in general, or think that normal policy doesn't apply here, or something else? (Also totally fine if you don't want to get into a meta-discussion about this here.)