I will use orthonormal definition of transformative AI: I read it as transformative AI would permanently alter world GDP growth rates, increasing them by 3x-10x. There is some disagreement between economists that is the case, i.e the economic growth could be slowed down by human factors, but my intuition says that's unlikely: i.e human-level AI will lead to much higher economic growth.

The assumption that I now think it is likely to be true (90% confident), that's possible to reach transformative AI by using deep learning, a lot of compute and data and the right architecture (which could also be different from a Transformer). Having said that, to scale models 1000x further there is significant engineering effort to be done, and it will take some time (improving model/data parallelism). We are also reaching the point where spending hundreds of millions of dollars of compute will have to be justified, so the ROI of these projects will be important. I have little doubt that they will be useful, but those considerations could slow down the exponential doubling of 3.4 months. For example, to train a 100 trillion parameters GPT model today (roughly 1 zettaflops day), it would require ad-hoc supercomputer with 100,000 A100s GPUs running for 100 days, costing a few billion $. Clearly, such cost would be justified and very profitable if we were sure to get a very intelligent AI system, but from the data we have now we aren't yet sure.

But I do expect 100 trillions models by the end of 2024, and there is a little chance that they could already be intelligent enough to be transformative AIs.

[10/50/90% || 2024/2030/2040] - 50% scenario: Microsoft is the first to release a 1 trillion parameter model in 2020, using it to power Azure NLP API. In 2021 and 2022, there is significant research into better architectures and other companies release their trillion parameters models: to do that they invested into even larger GPUs clusters. Nvidia datacenter revenue grows >100% year over year, and the company breaks the 1 trillion $ marketcap.

By end 2023, a lot of cognitive low-effort jobs, such as call centers, are completely automatized. It is also common to have personal virtual assistant. We also reach Level5 autonomous driving vehicles, but the models are so large that have to run remotely: low-latency 5G connections and many sparse GPUs clusters are the key. In 2024, the first 100 trillion model is unveiled and it can write novel math proofs/papers. This justifies a race towards larger models by all tech companies: billion $ or 10 billion $ scale private supercomputer/datacenter are build to train quadrillion-scale models (2025-2026). The world GDP growth rate start accelerating and it reaches 5-10% per year. Nvidia breaks above 5 trillion $ of marketcap.

By 2028-2030, large nations/blocks such as EU, China and US invest 100s billion $ into 1-5 Gigawatt-class supercomputers, or high-speed networks of them, to train and run their multi-quadrillion-scale model which reach superhuman intelligence and bring transformative AI. The world GDP grows 30%-40% a year. Nvidia is a 25 T$ company.

Having demoable technology is much different than having reliable technology. Take the history of driverless cars. Five teams completed the second DARPA grand challenge in 2005. Google started development secretly in 2009 and announced the project in October 2010. Waymo started testing without a safety driver on public roads in 2017. So we've had driverless cars for a decade, sort of, but we are much more cautious about allowing them on public roads.

Unreliable technologies can be widely used. GPT-3 is a successor to autocomplete, which everyone already has on their cell phones. Search engines don't guarantee results and neither does Google Translate, but they are widely used. Machine learning also works well for optimization, where safety is guaranteed by the design but you want to improve efficiency.

I think when people talk about a "revolution" it goes beyond the unreliable use cases, though?

[10/50/90% = 2021/2022/2025] 50% chance scenario: At the end of 2020, Google releases their own API with a 100x larger model. In 2021, there is a race for who could scale it to an even larger model, with Manhattan project level investments from nation states like China and the US. Physicists, computer scientists and mathematicians are working on bottlenecks privately, either at top companies or in secret nation-state labs. People have already built products on top of Google's API which make researchers and engineers more productive. At the end of 2021, one country (or lab) reached a GPT-5 like model size but doesn't publish anything. Instead, researchers use this new model to predict what a better architecture for a 100x bigger model should be, and their productivity is vastly improved. At the same time, researchers from private labs (think OpenAI /DeepMind) move to this leading country (or lab) to help build the next version (safely?). In 2022, they come up with something we would call both transformative AI and GPT-6 today.