In the beginning of the universe, everything was equal. Then it exploded. Quantum inequalities became galactic. Stars coalesced, and beamed energy into the voids left behind. Between one hot young star and the void was a planet we now call Earth. On Earth, energy gradients drove chemical reactions. One reaction...
In a dry field, a band of feral children crowd around a dim glow. The fire is small, but it’s the first they’ve ever made. Until now, most doubted it possible. Fire, from rubbed sticks? But the flicker and crackle is proof. One child raises her voice. “This is dangerous....
(Crossposted to the EA forum) Abstract The linked paper is our submission to the Open Philanthropy AI Worldviews Contest. In it, we estimate the likelihood of transformative artificial general intelligence (AGI) by 2043 and find it to be <1%. Specifically, we argue: * The bar is high: AGI as defined...
In the beginning of the universe, everything was equal.
Then it exploded. Quantum inequalities became galactic. Stars coalesced, and beamed energy into the voids left behind.
Between one hot young star and the void was a planet we now call Earth. On Earth, energy gradients drove chemical reactions. One reaction was special. It became life.
Over time, life grew in power. It evolved many tricks to agglomerate power:
- Self-replication
- Exclusion (e.g., cell membranes)
- Cooperation (e.g., symbiosis)
- Acquisition (e.g., incorporation of prokaryotes as organalles)
- Communication (e.g., quorum sensing)
- Agglomeration (e.g., multicellularity)
- Specialization (e.g., organs)
- Socialization
- Language
- Government
Eventually humans arose as the apex species.
Humans are not the most numerous species. That honor might go to Pelagibacter Ubique,... (read more)
In a dry field, a band of feral children crowd around a dim glow. The fire is small, but it’s the first they’ve ever made. Until now, most doubted it possible. Fire, from rubbed sticks? But the flicker and crackle is proof.
One child raises her voice.
“This is dangerous. It might set the field ablaze. Let’s stop.”
The others dismiss her.
“I’ve never seen a field catch fire.”
“This fire is too small to do anything.”
“If it gets bigger, we’ll be more careful.”
“If the field catches fire, we’ll learn to put it out.”
The children wonder how big a fire can get. They light a second fire, larger than the first. They light a third fire, larger... (read 587 more words →)
(Crossposted to the EA forum)
The linked paper is our submission to the Open Philanthropy AI Worldviews Contest. In it, we estimate the likelihood of transformative artificial general intelligence (AGI) by 2043 and find it to be <1%.
Specifically, we argue:
Hi Rose and Tom. One small error I noticed:
>AI chips double in FLOP/$ every ~2 years.
If you look at the top black dots from 2008 onward, the slope is more like doubling every 4-5 years.
Unfortunately not soon. API team is aware and mostly in favor, but not the top of their current priority list. I'll see if I can do something to move it forward.
We'll take a serious look at offering this option. We're generally happy to give people more control and transparency; the main worry that might hold us back is death by a thousand papercuts where each extra API option is small by itself but makes the overall experience more confusing for developers when dozens of them pile up. But in this case, might really be worth it. Stay tuned. :)
What would be the best solution for you? A new API parameter that lets you override the date?
Glad to see prefill was disabled for Opus 4.6!
Terrific!
One potential angle: automating software won't be worth very much if multiple players can do it and profits are competed to zero. Look at compilers - almost no one is writing assembly or their own compilers, and yet the compiler writers haven't earned billions or trillions of dollars. With many technologies, the vast majority of value is often consumer surplus never captured by producers.
In general I agree with your point. If evidence of transformative AI was close, you'd strategically delay fundraising as late as possible. However, if you have uncertainty about your ability to deliver, investors' ability to recognize transformative potential, or uncertainty about competition, you might hedge and raise sooner than you need. Raising too early never kills a business. But raising too late always does.
Can you explain what a point on this graph means? Like, if I see Gemini 2.5 Pro Experimental at 110 minutes on GPQA, what does that mean? It takes an average bio+chem+physics PhD 110 minutes to get a score as high as Gemini 2.5 Pro Experimental?
Yes, mostly.
I expect existentially dangerous ASI to take longer than ASI, which will take longer than AGI, which will take longer than powerful AI. Killing everyone on Earth is very hard to do, few are motivated to do it, and many will be motivated to prevent it as ASI’s properties become apparent. So I think the odds are low. And I’ll emphasize that these are my odds including humanity’s responses, not odds of a counterfactual world where we sleepwalk into oblivion without any response.
I wonder if there's a palatable middle ground where instead of banning all AI research, we might get people to agree to ban in advance only dangerous types of ASI.
My current personal beliefs:
- ASI existential risk is very much worth worrying about
- Dangerous ASI is likely the #1 threat to humanity
- In the next few decades, the odds of ASI killing/disempowering us is tiny
- I feel good accelerating capabilities at OpenAI to build technology that helps more people
- I would not support a ban or pause on AI/AGI (because it deprives people of AI benefits, breaks promises, and also accumulates a compute overhang for whenever the ban is later lifted)
- I would happily support a preemptive ban on dangerous ASI