Disagreement with bio anchors that lead to shorter timelines

[-]Tomás B.3y2211

In general, I have noticed a pattern where people are dismissive of recursive self improvement. To the extent people are still believing this, I would like to suggest this is a cached thought that needs to be refreshed.

When it seemed like models with a chance of understanding code or mathematics were a long ways off - which it did (checks notes) two years ago, this may have seemed sane. I don't think it seems sane anymore.

What would it look like to be on the precipice of a criticality threshold? I think it looks like increasingly capable models making large strides in coding and mathematics. I think it looks like feeding all of human scientific output into large language models. I think it looks a world where a bunch of corporations are throwing hundreds of millions of dollars into coding models and are now in the process of doing the obvious things that are obvious to everyone.

There's a garbage article going around with rumors of GPT-4, which appears to be mostly wrong. But from slightly-more reliable rumors, I've heard it's amazing and they're picking the low-hanging data set optimization fruits.

The threshold for criticality, in my opinion, requires a model capable of understanding the code that produced it as well as a certain amount of scientific intuition and common sense. This no longer seems very far away to me.

But then, I'm no ML expert.

[-][anonymous]3y31

It may also need a structured training environment and heuristic to select for generality.

The structured training environment is a set of tasks that train the machine a large breadth of base knowledge and skills to be a general AI.

The heuristic is just the point system : what metric are we selecting AI candidates by. Presumably we want a metric that selects simpler and smaller candidates with architecture that are heavily reused - something that looks like the topology of a brain - but maybe that won't work.

So the explosion takes several things: compute, recursion, a software stack framework that is composable enough for automated design iteration, a bench, a heuristic.

Nukes weren't really simple either, there were a lot of steps especially for the first implosion device. It took an immense amount of money and resources from the time physicists realized it was possible.

I think people are ignoring criticality because it hasn't shown any gain in the history of ai because past systems were too simple. It's not a proven track to success. What does work is bigass transformers.

[-]Tomás B.3y*42

I suppose I expect recursive self-improvement to play out in the course of months not years. And I worry groups like OpenAI are insane enough to pursue recursive self improvement as an explicit engineering goal. (Altman seems to be a moral realist, explicitly says he thinks the orthogonality thesis is false.) From the outside, it will appear instant as there will be a perceived discontinuity when the fact that it has achieved a decisive strategic advantage becomes obvious.

[-][anonymous]3y30

Well again remember a nuclear device is a critical mass of weapons grade material.

Anything less than weapons grade and nothing happens.

Anything less than sudden explosive combination of the materials and the device will heat itself up and blast itself apart with sub kiloton yield.

So analogy wise : current llms can "babble" out code that sometimes even works. They are not trained on RL selecting for correct and functional code.

Self improvement by code generation isn't yet possible.

Other groups have tried making neural networks composable, and using one neural network based agent to design others. It is also not good enough for recursion but this is how autoML works.

Basically our enrichment isn't high enough and so nothing will happen. The recursion quenches itself before it can start, the first generation output isn't even functional.

But yes, at some future point in time it WILL be strong enough and crazy shit will happen. I mean think about the nuclear example: all those decades of discovering nuclear physics, fission, the chain reaction, building a nuclear reactor, purifying the plutonium...all that time and the interesting event happened in milliseconds.

[-]Marius Hobbhahn3y22

I'm honestly not even sure whether this comment is in support of or against my disagreements.

I'm skeptical of the "recursive self-improvement leads to enormous gains in intelligence over days" story but I support the "more automation leads to faster R&D leads to more automation, etc." story which is also a form of recursive self-improvement-just over the span of years rather than days.

[-][anonymous]3y20

Recursion is an empirical fact. It works numerous places. Computing itself that led to AI was recursive - the tools that allow for high end/high performance silicon chips require previous generations of those chips to function. (to be fair the coupling is loose, Intel could design better chips using 5-10 year old desktops and servers and some of their equipment is that outdated)

Electric power generator production for all currently used types require electric power from prior generators, etc.

But yeah it so far hasn't happened in days.

[-]NickGabs3y20

I think making progress on ML is pretty hard. In order for a single AI to self improve quickly enough that it changed timelines, it would have to improve close to as fast as the speed at which all of the humans working on it could improve it. I don't know why you would expect to see such superhuman coding/science capabilities without other kinds of superintelligence.

[-]Lone Pine3y3-1

Furthermore, I think that the time “when AI has the potential to be dangerous” is earlier than my estimate of TAI because I think that this poses a lower requirement than the potential to be economically transformative

Doesn't the AI need at least the capacity to autonomously manufacture computer chips and some kind of robot (nano or otherwise)? If the AI can't do that, it can't survive and reproduce itself after it kills us. If the AI can do that, it would be economically transformative. (Although if we are in a fast takeoff situation, maybe the AI is economically transformative for a total for 23 whole minutes before it kills us.)

[-]Marius Hobbhahn3y*31

I think dangerous doesn't have to imply that it can replicate itself. Doing zero-shot shenanigans on the stock market causing a recession and then being turned off also counts as dangerous in my books.

[-][anonymous]3y32

Note that one dangerous scenario is if the AI is able to make highly convincing arguments to scam humans into giving the AI more power. Also it doesn't matter if it kills us in a way that leaves the AI doomed also, as humans would still be dead.

[-]Lone Pine3y20

I think that an AI that is competent enough to kill us would be competent enough to survive afterwards, and would be savvy enough to not kill us until we became expendable.

[-][anonymous]3y10

Again it doesn't have to work this way at all. Some runaway optimizer trying to drive user engagement could inadvertently kill us all. It need not be intelligent enough to ensure it survives the aftermath.

I mean a deadlier version of covid could have theoretically ended us right here. Especially if it killed by medium term genetic damage or something that let it's victims live long enough to spread it.

[-]Lone Pine3y20

Some runaway optimizer trying to drive user engagement could inadvertently kill us all.

Could you go into more detail about this scenario? I'm having trouble visualizing how ad-AI can cause human extinction.

[-][anonymous]3y32

I am assuming some scam similar to a political party or religion the optimizer cooks up. The optimizer is given some pendantic goal completely unrelated, aka paperclips or make users keep scrolling, and it generates the scam from the rich library of human prior art.

Remember, humans fall for these scams even when the author, say L R Hubbard, openly writes that he is inventing a scam. Or a political party selects someone as their leader who obviously is only out for themselves and makes this clear over and over. Or we have extant writing on how an entire religion was invented from non-existent engraved plates no one but the scammer ever saw.

So an AI that promises some unlikely reward and has already caused people's deaths and is obviously only out for itself might be able to scam humans into hosting it and giving it whatever it demands. And as a side effect this kills everyone. And it's primary tool might be regurgitating prior human scam elements using an LLM.

I don't have the rest of the scenario mapped, I am just concerned this is a vulnerability.

Existing early agents (Facebook engagement tooling) seem to have made political parties extreme, which has lead to a few hundred thousand extra deaths in the USA. (From resistance to rational COVID policies).

The Ukraine war is not from AI but gives a recent example where poor information leads to bad outcomes for all players. (The primary decision maker, Putin, was misinformed as to the actual outcome of attempting the attack)

So a screw up big enough to kill everyone I don't know, but there obviously are ways it could happen. Chains of events that lead to nuclear wars or modified viruses capable of causing extinction are the obvious ones.

[-]Marius Hobbhahn2yΩ120Review for 2022 Review

I still stand behind most of the disagreements that I presented in this post. There was one prediction that would make timelines longer because I thought compute hardware progress was slower than Moore's law. I now mostly think this argument is wrong because it relies on FP32 precision. However, lower precision formats and tensor cores are the norm in ML, and if you take them into account, compute hardware improvements are faster than Moore's law. We wrote a piece with Epoch on this: https://epochai.org/blog/trends-in-machine-learning-hardware

If anything, my disagreements have become stronger and my timelines have become shorter over time. Even the aggressive model I present in the post seems too conservative for my current views and my median date is 2030 or earlier. I have substantial probability mass on an AI that could automate most current jobs before 2026 which I didn't have at the time of writing.

I also want to point out that Daniel Kokotajlo, whom I spent some time talking about bio anchors and Tom Davidson's takeoff model with, seemed to have consistently better intuitions than me (or anyone else I'm aware of) on timelines. The jury is still out there, but so far it looks like reality follows his predictions more than mine. At least in my case, I updated significantly toward shorter timelines multiple times due to arguments he made.

[-]Donald Hobson3yΩ12-1

My best guess bio anchors adaption suggests a median estimate for the availability of compute to train TAI

My best guess is in the past. I think GPT3 levels of compute and data are sufficient, with the right algorithm, to make a superhuman AI.

[-][anonymous]3y10

Willingness to spend: Cotra assumes that the willingness to spend on Machine Learning training runs should be capped at 1% the GDP of the largest country, referencing previous case studies with megaprojects (e.g. the Manhattan Project), and should follow a doubling time of 2 years after 2025.

This seems like a critical decision term.

1. Nvidia charges $33k per H100. Yet the die is a similar size as the 4090 GPU, meaning nvidia could likely still break even were they to charge $1500 per H100.

Just imagine a counterfactual: A large government (China, USA etc) decides they don't enjoy losing and invests 1% of GDP. They pressure Nvidia, or in China's case, get domestic industry to develop a comparable (for AI training only which reduces the complexity) product and sell it for closer to cost.

That's a factor of 22 in cost, or 9 years overnight. Also I'm having trouble teasing out the 'willingness to spend' baseline. Is this what industry is spending now, spread across many separate efforts and not all in to train one AGI? It looks like 1% is the ceiling. Meaning if a government decided in 2023 that training AGI was a worthy endeavor, and the software framework were in place that would scale to AGI (it isn't):

Would 1% of US GDP give them enough compute at 2023 prices, if the compute were purchased at near cost, to train an AGI?

As a side note, 1% of US GDP is only 230 billion dollars. Google's 2021 revenue was 281 billion. Iff a tech company could make the case to investors that training AGI would pay off it sounds like a private investment funding the project is possible.

	Aggressive - bio anchors (Marius)	Best guess - bio anchors (Marius)	Independent impression (Marius)	Ajeya’s best guess (2020)
Algorithmic progress doubling time	1.-1.3 years	1.3-1.6 years	1.3-1.6 years	2-3.5 years
Compute progress doubling time	2.5 years	2.8 years	3 years	2.5 years
Model FLOPS vs. brain FLOPS (=1e15) median	-1	-0.5	-0.2	+1
Lifetime anchor	16%	10%	10%	5%
Short NN anchor	40%	24%	30%	20%
Medium NN anchor	20%	35%	31%	30%
Long NN anchor	10%	17%	13%	15%
Evolution anchor	3%	3%	5%	10%
Genome anchor	1%	1%	1%	10%
10th percentile estimate	<2025	~2025	~2028	~2032
Median(=50%) estimate	~2032	~2036	~2041	~2052
75th percentile estimate	~2038	~2052	~2058	~2085

LESSWRONG
LW

LESSWRONG
LW

75

Disagreement with bio anchors that lead to shorter timelines

75

Ω 27

75

Ω 27

Executive summary

My disagreements

The resulting model

Final words