I'll have use for a model with a time-horizon 100x bigger than now in three years, I don't know if I will have use for a model with a time-horizon 10,000x bigger than today in six years
This is where I get lost, here. Isn't "there will be a model with a 10,000x bigger time-horizon" equivalent to "the singularity will have happened"?
Some people argue that the time horizon won't keep growing at the same pace, and it will plateau, and others argue that it will and we'll get a technological singularity, but if an LLM can do anything that would take a moderately competent human five years, then that does seem like the end of our current mode of civilization.
In other words, I don't see a set of possible worlds where LLM time horizons get too long to be marketable to hobbyist engineers and that lack of marketability is still a concern.
This is far from being why so many people don't believe in AI superintelligence.
What is needed is to push the public to look at straight lines in a log-(log-)chart.
Any financial advisor worth their salt will tell you "past performance does not guarantee future performance". Inductive reasoning is famously not the most robust way of understanding the world. The fact that Moore's law held up does not mean, in any way, that all trends in the world of compute will hold up.
All they need to do is to understand benchmarks and to use AI, in its best form, for themselves every 6 months.
Benchmarks are extremely hackable. And AI's intelligence keeps being extremely jagged.
There is no actual proof or robust reasoning for ASI. It's just speculation. Which is fine, there are many things in the world we can't determine exactly. But let's not pretend it's a fact.
People will say:
I will keep updating the list above over time.
I believe the public has been using very bad heuristics to decide how much they should care about the field of artificial intelligence. The goal of this essay is to try to explain why having a world model of imminent Superintelligence isn't in opposition with the way the Labs behave.
The audience I expect to read this text are Less Wrong readers and that people who much better communicators than myself can repackage the argument to normies.
The capitalist class treats AI as normal technology
The reality is that the entire capitalist class, with some rare exceptions (like Masayoshi Son, who was ASI pilled back in 2010), look at revenue, not capabilities. And for a variety of reasons revenue is extremely lagging of AI capabilities:
If a given AI Lab wants to get to the Superintelligence, and to get there first, they expect they will have exponentially growing training costs to train the Superintelligence. And even though they could fund their increasing training costs with their exponentially growing revenue, they know they'd lose to some other lab that would accept to also have exponentially growing losses, funded by capitalists.
What happens is that capitalists will want the labs to beat the very rosy expectations they set, for example, leaking financials to The Information.
Capitalists can and do look ahead, but they will always have a hard time paying attention to the exponential. But if the AI Lab CFO says things such as:
Capitalists can pencil down some math and invest into OpenAI at $500B valuation or to Anthropic at $300B valuation, or something like that.
Even if internally your goal is to create the Superintelligence, ask it to create 100 new world-changing drugs, patent them, and get unbelievably rich, you can't tell the capitalists that. Or if you tell them, they won't believe. You need to tell them you'll take a cut of eCommerce sales.
But capitalists are smart, this means that if you tell them you'll put ads in ChatGPT, you need to actually add ads to ChatGPT one year later, otherwise they'll question your execution and their revenue expectations will disappoint them.
Because creating the Superintelligence is really expensive and it might require AI Labs to raise hundreds if not trillions of equity capital from society, they will need to increasingly play this game.
Adding monetization that will be meaningless when the Superintelligence arrives is a cost they AI Labs are willing to pay to create the Superintelligence.
The Labs want to be prepared if AI is a normal technology
If xAI ends up creating universal high income: great! If xAI ends up killing everyone, not great, but who will be left to care? But in the worlds where AI ends up being normal technology, those will be the ones where it makes the most sense to be prepared if AI is normal technology.
In reality, being prepared for AI being normal technology is easy. If you are Sam Altman and you are busy securing compute, going to podcasts, talking to your researchers, and broadly enabling everyone to create AGI, you might think "Gosh, how pointless it is to spend time creating health care features for ChatGPT when in 2 years GPT-6.5-CODEX-xhigh will be able to one-shot it", but in the great scheme of things, the cost of hiring a couple hundreds engineers and putting Fudji Simo to create ChatGPT Health and put ads in ChatGPT isn't immense and you can pay them in equity anyway. Imagine if Scaling Laws hit a wall and you didn't do these things and you lose to Google!
More importantly, many of these decisions that have formed people's views were made during the past eighteen months where it was much less clear compared to today how much line of sight we have to create the Superintelligence. Sam Altman has recently said:
Some AI bears will have it both ways: heads AI doesn't work because they hire people, tails AI doesn't work and Sam Altman needs to cut costs because his company is unprofitable.
Some other reasons why the labs want to be prepared if AI is normal technology:
The Labs want to be prepared if the Superintelligence doesn't deliver superior economics
Given how much the labs talk about their fears in public (see Dario Amodei's The Adolescence of Technology), I find it underdiscussed how little is talked about the possibility about the Superintelligence being a commodity.
The debate around the Superintelligence almost entirely assumes you need to "escape the permanent underclass" or contributions like Dwarkesh Patel and Phillip Trammell Capital in the 22nd Century. Dwarkesh and Phillip's implied view is that one hundred year post-singularity, there will still exist enough interesting things for capital to compound at accelerated rates, instead of the forces of competition pushing all the prices to zero, because there are no longer many things people want.[1]
The labs model is predicated at always being demand for SOTA, similarly to TSMC. Unlike TSMC, China SOTA is 6 months behind, and do it yourself AI is 18 months behind, and the rate of change is 4x faster. I assign a probability higher than 50% that in 2028, I will be using an older open-source model instead of paying market prices for the State of the Art.
As the value of selling proto-AGI through an API becomes commoditzed, it's likely that the labs will need to transition to creating science themselves, patents themselves, and have internally built AI they don't share to the public.
The labs obviously know this, but the transition could be far from perfect. The one best prepared is Google. Isomorphic Labs already exists and already is patenting and creating the muscle to make money off AI-created science. Even there, I doubt Isomorphic Labs will be even considered a top-20 pharma company in 2030. At the same time, while I think I'll have use for a model with a time-horizon 100x bigger than now in three years, I don't know if I will have use for a model with a time-horizon 10,000x bigger than today in six years. I might prefer AI that is cheaper and faster. We could hit bumps in the monetization road.
Once again, I don't pretend to have answers.
The key point is that it makes sense for labs to hedge. The cost of hedging is small in the great scheme of things. But it creates apparently inconsistent behavior.
The Labs think they desperately need to win
A lot of AI Labs have deep distrust between each other, between them and China, and so forth. Anthropic was created because the founders didn't trust OpenAI. xAI was created because Elon asked for AI to be paused and no one heard him (and he doesn't trust OpenAI). Meta Superintelligence Labs was created because Mark doesn't trust Google. OpenAI was created because Elon didn't trust Google and Page. Safe Superintelligence was created because Ilya didn't like OpenAI research path (and likely he also doesn't trust OpenAI). [2]
And all the Lab leaders, wholeheartedly, believe they are about to create the Superintelligence and the prize is only there for whoever gets there first (assuming singularity/recursive-self improvement).
Anthropic is right now betting the ranch they'll get there. Our Effective Autruist overlords at Anthropic, quite likely, would like that we could slow down the development of the Superintelligence for society to be ready. Dario Amodei said that he would coordinate with Google DeepMind, if the race was only between the two.
From the AI perspective, no cost is high enough to increase the likelihood they will be the ones creating the Superintelligence and getting to control it.
The costs of creating the Superintelligence are increasing exponentially
This point is obvious for anyone who knows one thing or two about the scaling laws. See Gwern, Leopold, and Dwarkesh.
The costs of scaling state of the art artificial intelligence are increasing by 10x every two years, with no end in sight. Last year, OpenAI fundraised something like $40B. This year they are fundraising just in the first month of the year 2.5x that, and they plan the largest initial public offering in history later this year. That's because the costs of creating the Superintelligence are increasing to the point that soon even Google DeepMind will have difficulty funding it.
The implications are two-fold:
If the investments in artificial intelligence continue constant without increasing, AI timelines are much longer than most on the labs and on this website expect.[4]
In reality, every year that we 3x the compute used to train state of the art models and the slot machine prints intelligence, society will return to increase the intelligence, either through the capitalist system, or through state intervention.
The fallacy is caused by past misguided techno-optimism (and lies)
The problem the AI Safety community needs to overcome is the system one thinking that was created in many tech observers, to fade most, if not all, claims made by tech people.
Elon Musk has said that Full Self-Driving was imminent for 10 years. He's now, once again, saying that. No one believes him. But FSD is imminent.
Vinod Khosla said, in 2017, that in five years, radiologists will be obsolete. Not only radiologists were not obsolete in 2017, but the employment in Diagnostic Imaging Centers in the United Stated has outpaced the overall employment growth in the overall economy. But AI that can make radiologists obsolete is imminent.
The heuristics many people have created for themselves is "Tech bros often lie about the capabilities and about the future because they are trying to sell you something or to raise money from you. I can avoid understanding the underlying details about technology XYZ only if I look at how they behave."
This is a fallacy. I am calling it "If the Superintelligence was near fallacy".
What is needed is to push the public to look at straight lines in a log-(log-)chart.
What is needed is to explicitly call out the fallacy in the public discourse.
What is needed is to tell people they don't need to ever hear Sam Altman. All they need to do is to understand benchmarks and to use AI, in its best form, for themselves every 6 months.
The hardest thing about AI Safety is that it's an extremely optimistic and extremely pessimistic view of the future. Most people don't get it and it's needed to be extremely candid about that.
I hope that by documenting the 'If the Superintelligence was near fallacy', we can start to have better conversations.
There will always exist arguments over how much people want and the nature of the things there are still there to create. I would argue a compressed 21st century, like Dario Amodei describes in machines of loving grace is possible, a compressed 3rd millenia is unlikely.
Lots of this lack sources! If I was writing for the NYT, I wouldn't write these things. It's hard to say these things in such uncertain terms as I am saying. Don't quote me on that. That's just me reading in the between lines!
You could argue that this is how all the big atrocities in History started because someone thought their cause was fair and no price was high enough to get there. I would argue many AI leaders are victims of thinking they will singlehandedly solve History. But that's not the point of the essay.
Recursive-self improvement is a dark horse here! Anthropic seems to think they can get there by investing dozens of billions, not hundreds of billions of dollars. And RSI is the base expectation of many really good AI observers.