What's always bothered me about "unaware self AI" scenarios are the fact they are literally reading everything we write right now. How is their "generated world-model" processing us talking about it? There might be a wake-up somewhere in there. That point, of course, is the entire argument about when a machine passes the Turing test or other models to become indistinguishable from sentient life.
Excellent and thoughtful review. I agree with your take on "human on human" interactions which AI will struggle to emulate at least at first. I think trying to compare our age to others is very difficult because we are definitely on some sort of exponential, but we don't know the power of the exponent. AI is finally good enough to scare me (GPT-2 I'm looking at you). I've been waiting for this moment for about 25 years, and it just arrived. GPT-2 is very close to being able to pass a Turing test with only text exchanges allowed. I didn't know if we ever would, and it could have been 2040, 2060, etc. But it's here now, and the show begins.