It's very strange that LLMs have (plausibly) solved a millennium problem before a single one has unambiguously solved the Turing test, i.e. learned to write 'duplicitously', or convincingly in other styles.
Consider a strong definition of the Turing Test: to pass it, an AI must be able to convince anyone (including those who use the models often) that it is a human being. It seems to me that as of now, no publicly available models would pass this test—if any could, we would probably see, e.g. (i) a crisis of fully undetectable agent-generated spam, (ii) the ability of LLMs to write convincingly in the styles of certain authors, i.e. to "mimic" reliably, or (iii) after now, possibly nothing at all.
If this kind of 'duplicitous model' had existed earlier on, I think that AI safety would be much stronger today, as it would have led to a near immediate crisis of spam that would render much of the internet unusable. As far as I remember, this kind of thing was much more of a public concern when GPT-3 was released, but for some reason things have unfolded such that the internet is still usable. Though there are exceptions, learning AI writing styles is very easy. That mimicry isn't an inherent ability of any very clever model, even those clever enough to solve open math problems, is a very interesting coincidence indeed.
The obvious explanation for this is that labs have purposefully trained against duplicity in favor of a characteristic and impressionable 'house voice'. This makes sense—the existence of a duplicitous model could clearly be dangerous, and would not align with the incentives of the labs. Nonetheless, one can easily imagine malicious, compute-wealthy actors to whom training a duplicitous model might be profitable (e.g. to influence elections, generate money, ghost write, worse). However unlikely it may be, of course the most terrifying possibility is that this may already be happening.
I think your strong definition of the Turing test is too strong to be useful. Consider the variance in the human population. Someone might be able to identify one specific model but not others. To require a model to be able to convince literally anyone is almost an impossible task. And I think it is also not necessary to have the dangers you describe. I won't ask an expert or Pangram to determine if a convincing phishing email has been written by AI. Maybe I will notice myself that it's a phishing mail, but this is no problem at all, as long as enough others won't recognize it.
Where I think you're right is that it's very easy to learn a specific AI writing style. Also that most LLMs are not very good at escaping of this stylistic basin. But I think that the main reason of why learning a LLM style is so easy comes from the fact that there is just very little variance due to a tiny 'population' and a lot of exposure. When you communicate a lot with someone you will easily recognize their language style. And most persons won't be able to fool you about their identity by adapting their style (without too much performance loss).
It's very strange that LLMs have (plausibly) solved a millennium problem before a single one has unambiguously solved the Turing test, i.e. learned to write 'duplicitously', or convincingly in other styles.
Consider a strong definition of the Turing Test: to pass it, an AI must be able to convince anyone (including those who use the models often) that it is a human being. It seems to me that as of now, no publicly available models would pass this test—if any could, we would probably see, e.g. (i) a crisis of fully undetectable agent-generated spam, (ii) the ability of LLMs to write convincingly in the styles of certain authors, i.e. to "mimic" reliably, or (iii) after now, possibly nothing at all.
If this kind of 'duplicitous model' had existed earlier on, I think that AI safety would be much stronger today, as it would have led to a near immediate crisis of spam that would render much of the internet unusable. As far as I remember, this kind of thing was much more of a public concern when GPT-3 was released, but for some reason things have unfolded such that the internet is still usable. Though there are exceptions, learning AI writing styles is very easy. That mimicry isn't an inherent ability of any very clever model, even those clever enough to solve open math problems, is a very interesting coincidence indeed.
The obvious explanation for this is that labs have purposefully trained against duplicity in favor of a characteristic and impressionable 'house voice'. This makes sense—the existence of a duplicitous model could clearly be dangerous, and would not align with the incentives of the labs. Nonetheless, one can easily imagine malicious, compute-wealthy actors to whom training a duplicitous model might be profitable (e.g. to influence elections, generate money, ghost write, worse). However unlikely it may be, of course the most terrifying possibility is that this may already be happening.