To people who downvote, it would be much more helpful, if you actually wrote a reply. I'm happy to be proven wrong.
I would feel much more concerned about advances in reinforcement learning, rather than training on large datasets. As surprising as some of the things that GPT-3 and the like are able to do, there is a direct logical link between the capability and the task of predicting tokens. Detecting and repeating patterns, translations, storytelling, programming. I don't see a link between predicting tokens and overthrowing the government or even manipulating a single person into doing something. There is no reward for that, I don't particularly see any variation of ...
Maybe there is some consolation in that if the humanity were to arrive at something approaching AGI, it would rather be better for it to do so using an architecture that's limited in its ultimate capability, demonstrates as little natural agency as possible, ideally that's a bit of a dead end in terms of further AI development. It could serve as a sort of vaccine if you will.
Running with the singularity scenario for a moment, I have very serious doubts that a purely theoretical research performed largely in a vacuum will yield any progress on AI safety. Th... (read more)