Excellent and thoughtful review. I agree with your take on "human on human" interactions which AI will struggle to emulate at least at first. I think trying to compare our age to others is very difficult because we are definitely on some sort of exponential, but we don't know the power of the exponent. AI is finally good enough to scare me (GPT-2 I'm looking at you). I've been waiting for this moment for about 25 years, and it just arrived. GPT-2 is very close to being able to pass a Turing test with only text exchanges allowed. I didn't know if we ever would, and it could have been 2040, 2060, etc. But it's here now, and the show begins.
What's always bothered me about "unaware self AI" scenarios are the fact they are literally reading everything we write right now. How is their "generated world-model" processing us talking about it? There might be a wake-up somewhere in there. That point, of course, is the entire argument about when a machine passes the Turing test or other models to become indistinguishable from sentient life.
That's very interesting. My model is precisely the opposite, that free will is an illusion if we accept any of the theories. From the moment the universe began, it can only proceed on one path or all paths (each quantum event triggers a split so all possible actions happen). Either way, free will is an illusion.
I live my life entirely free from this conclusion, because not believing in free will will soon land you in a place where the doors don't open from the inside.