The best response I've heard against the simulation hypothesis is "If we're simulated, why aren't we asked to do meaningful work?" Think about why we would want to simulate personalities in our near to medium term future: to answer questions, to help us work faster, to entertain us, to be our companions, etc.
So why aren't we asked to complete the equivalent of Mechanical Turk questions? Is compute is so unfathomably cheap that they there's no economic reason we would need to "pay rent" so to speak? Even if that's the case, why don't our creators want to interact with us directly? In other words, if we're all video game characters, who is the player character? Why do I have consciousness, if I'm just an NPC?
And on and on. The crux is whether you believe that in the world that allows us to be simulated, it's more likely that our creator just wants to view us through a thick pane of glass than they'll want to play with us. My intuition is very much on the later.
Let me go one further, here: Suppose an GPT-N, having been trained on the sum total academic output of humanity, could in-principle output a correct proof of the Riemann Hypothesis. You give it the prompt: "You're a genius mathematician, more capable than any alive today. Please give me a complete and detailed proof of the Riemann Hypothesis." What would happen?
Wouldn't it be far simpler for GPT-N to output something that looks like the proof of the Reimann Hypothesis rather than the bona-fide thing? Maybe it would write it in a style that mimics current academic work far better than GPT-4 does now. Maybe it would seem coherent on first glance. But there's a lot of ways to be wrong and only a few to be correct. And it's rewarded the same amount it produces a correct result versus one that seems correct unless probed extensively. Wouldn't it be most parsimonious if it just ... didn't spend it's precious weights to actually create new mathematical insights and just learned to ape humans (with all their faults and logical dead-ends) even better. After all that's the only thing it was trained to do.
So yes, I expect with the current training regime GPT won't discover novel mathematical proofs or correct ways to make nano-tech if those things would take super-human ability. It could roleplay those tasks for you, but it doesn't really have a reason to actually make those things true. Just to sound true. Which isn't to say it wouldn't be able to cause widespread damage with just things humans can and already do (like cyber terrorism or financial manipulation). It probably won't be pie in the sky things like novel nano-tech.
I liked the format, but let me pick on a particular point. What makes you confident that in seven years China will be meaningfully ahead of the West? My intuition is that the West still has the best education and economic centers to drive R&D and those have significant moats that don't get shaken up that quickly. You're pretty vague about your justifications other that impressive levels of progress. I see it as a "rising tides float all boats" situation where progress is being accelerated everywhere by open sharing, an economically conducive environment for AI research, and availability of compute.
This took a weird turn in an article that I thought would explore the basic scientific challenges of nanotech and why we don't have it yet. I do think that the erosion of religion has some negative externalities in modern society (ie. lack of easy meaning and direction in life and downfall of local community kinship), but no I don't think that is the main reason why we don't have nanotech specifically. I don't even think that's the primary reason we are more polarized politically now (my current thoughts lean towards changes in information consumption, communication, and trust in institutions).
But specifically nano-printers? Of course people want that, as much as people want quantum computing, fusion energy, brain-computer interfaces, and life-extension. The benefits are obvious, from the money-making opportunities alone. Maybe reality is just disapointing: it's a harder problem than people originally expected without any economically viable intermediate steps (the kind that bolster AI reasearch now) so progress is stuck in a quagmire.
I agree that most of the recent large model gains have been due to the surplus of compute and data, and theory and technique will have to catch up eventually ... what I'm not convinced on is why that would necessarily be slow.
I would argue there's a theory and technique overhang with self-supervised learning being just one area of popular research. We haven't needed to dip very deeply yet since training bigger transformers with more data "just works."
There's very weak evidence that we're hitting the limits of deep learning itself or even just the transformer architecture. Ultimately, that is the real limiter ... certainly data and compute are the conceptually easier problems to solve. Maybe in the short-term that's enough.
Sutton and Barto's Reinforcement Learning textbook: http://incompleteideas.net/book/the-book-2nd.html
Elements of Statistical Learning: https://hastie.su.domains/ElemStatLearn/ as well as the more introductory version: https://www.statlearning.com/
Deep Learning textbook: https://www.deeplearningbook.org/
Bayesian Data Analysis: http://www.stat.columbia.edu/~gelman/book/
Mongolian BBQ is completely divorced from the ethnic cuisine of Mongolia, which is heavy in meat and fat and (within my limited knowledge) not very flavorful. I've watched a food vlog and ... yeah, it's fatty lamb boiled in plain broth. Food is subjective, but I'm sure that wasn't what you were thinking.
What we know as Mongolian BBQ was actually invented in Taiwan on the 1950's and given that name for a pretty arbitrary reasons. (As an aside how many food names with a country name are actually from that place? Usually it's tenuous connections at best like how Hawaiian pizza was named because of pineapples and actually invented in Canada).