A single strong down vote by a moderately high karma user dropped this post from 3 to -4; the actual sequence of votes seems to be 1+1+1+1-1-7=-4. I say this only b/c the naive conclusion might be that this post received only down votes.
Update -- substantial revision 2025-01-08, § 0 added 2025-01-12.
These are arguments in favor of the SMI2LESHAI position ie (ubiquitous) super human AI but without space migration, intelligence increase or (radical) life extension for adults alive in 2025. Practical consequences of SMI2LESHAI are also discussed (§ 6|2). There is no comprehensive attempt to convert any one to SMI2LESHAI ism. Each section is relatively independent.
I think I under estimated the extent to which people on LW hold the belief that {vastly super human AI will lead to all sorts of amazing things}.
Instead I believed that people were more likely to hold beliefs roughly along the lines of {things that happen efficiently benefit from inherent easiness}. Eg energy used by the economy is still small relative to energy used by the ecosystem which is extremely small relative to solar energy reaching the earth.
Hence there is no deliberate attempt to refute the (IMO want based belief) that {vastly super human AI will lead to all sorts of amazing things}, but people who hold such a belief should still be able to follow much of the reasoning & accept many of the conclusions.
(Whether they actually follow the reasoning or accept many of the conclusions is of course a different matter.)
Summary. I argue, from the perspective of biology, that super human AI is a very low hanging fruit. I believe that this argument is very solid. I briefly consider reasons why {super human AI}/{vastly super human AI} might not arise. I then contrast AI with other futurist technologies like human brain emulation, radical life extension & space colonization. I argue that these technologies are in a different category & plausibly impossible to achieve in the way commonly envisioned. This also has some relevance for EA cause prioritization.
Aphorism -- Probably AI is mostly a dead end [N0].
In my experience certain arguments are ignored b/c they are too straightforward. People have perhaps heard similar arguments previously. The argument isn't as exciting. B/c people understand the argument they feel empowered to disagree with it. They may believe that complicated new arguments, which they don't actually understand well, have debunked the straightforward arguments even tho the straightforward argument may be easily salvageable or not even debunked in the 1st place!
N0. AI will do incomprehensibly good things for AI. Aphorism = a vague statement that should become precise upon reading the post. Maybe this will help people remember/understand.
The reasons why super human AI is a very low hanging fruit are pretty obvious.
1) The human brain is meager in terms of energy consumption & matter. 2000/calories per day is approximately 100 watts. Obviously the brain uses less than that. Moreover the brain is only 3 pounds.
So we know for certain that human level intelligence is possible with meager energy & matter requirements. It follows that super human intelligence should be achievable especially if we're able to use orders of magnitude more energy & matter which we are.
2) Humans did not evolved to do calculus, computer programming & things like that.
Even Terence Tao did not evolve to do complicated math. Of course you can nitpick this to death by saying humans evolved to do many complex reason tasks. But we didn't actually evolve to do tasks requiring such high levels of mathematical reasoning ability. This is actually why there's such a large variability in mathematical intelligence. Even with 3 pound brains we could all have been as talented as (or even far more talented than) Terence Tao had selective pressure for such things been strong.
3) Evolution is not efficient.
Evolution is not like gradient descent. It's more like Nelder Mead. Much of evolution is just purging bad mutations & selection on standing diversity in response to environmental change. A fitness enhancing gain of function mutation is a relatively very rare thing.
A) Evolution does not act at the level of synapse.
The human genome is far far too short. Instead the genome acts as metaparameters that determine human learning in response to the environment. I think this point cuts both ways which is why I'm referring to it as A not 4. Detailed analysis of this point is far beyond the scope of this post. But I'm inclined to believe that such an approach is not quite as inefficient as Nelder Mead applied at the level of synapse but more limited in its ability to optimize.
TBD
I see only a few reasons why super human AI might not happen.
1) An intelligence obedience tradeoff. Obviously companies want AI to be obedient. Even a harmless AI which just thinks about incomprehensible AI stuff all day long is not obviously a good investment. It would be the corniest thing ever if humans' tendency to be free gave us an insurmountable advantage over AI. I doubt this is the case, but it wouldn't be surprise if there is some (not necessarily insurmountable) intelligence obedience tradeoff.
2) Good ideas are not tried b/c of high costs. I feel like I have possibly good ideas about how to train AI, but I just don't have the spare 1 billion dollars.
3) Hardware improvements hit a wall.
4) Societal collapse.
Realistically I think at least 2 of these are needed to stop super human AI.
In § 2 I argue that super human AI is quite an easy task. Up until quite recently I would some times encounter claims that human brain emulation is actually easier than super human AI. I think that that line of thinking puts somewhat too much faith in evolution. The problem with human brain emulation is that the artificial neural network would need to model various peculiarities & quirks of neurons. An easy & efficient way for a neuron to function is not necessarily that easy & efficient for an artificial neuron & vice verse. Adding up a bunch of things & putting that into ReLu is obviously not what a neuron does, but how complex would that function need to be to capture all of a neuron's important quirks? Some people seem to think that the complexity of this function would match it's superior utility relative to an artificial neuron [N1]. But this is not the case; the neuron is simply doing what is easy for a neuron to do; likewise for the artificial neuron. Actually the artificial neuron has 2 big advantages over the neuron. The artificial neuron is easier to optimize and it is not spatially constrained.
If human brain emulation is unavailable, then a certain vision of mind uploading is impossible. But an AI copying aspects of a person's personality, like the Magi system from that TV show, is not some thing that I doubt [N2].
N1. I've some times heard the claim that a neuron is more like a MLP. I would go so far as to claim that an artificial neuron with input from k artificial neurons & using a simple activation function like ReLu & half precision is going to be functionally superior to a neuron with input from k neurons b/c of greater optimization & lack of spatial constraints. But simulating the latter is going to be way more difficult.
N2. That an AI could remember details of your life better than you could and in that sense be more you than you could possibly be is also worth noting.
Life spans longer than human are definitely possible & have been reported for Bowhead whales, Greenland sharks & a quahog named Ming. But the number of genetic changes necessary for humans to have such long life spans is probably high. And it's unclear whether non genetic interventions will be highly effective given the centrality of DNA in biology.
The energy costs of sending enough stuff into space to bootstrap a civilization are alone intimidating. Perhaps advances like fusion or improved 3D printing will solve this problem.
Won't super human AI make it all possible?
I'm not claiming that human brain emulation, radical life extension & space colonization are definitely impossible. And I'm not proposing that people preemptively give up on these goals. Some elements of human brain emulation will not require the simulation to be accurate at the neuronal level. Radical life extension via genetics seems in principle achievable but maybe not desirable or worthwhile.
But in the case of super human AI we're merely trying to best some thing that is definitely possible & with meager physical inputs.
On the other hand human brain emulation, radical life extension & space colonization may be possible or they may be too physically constrained ie constrained by the laws of physics.
What is the significance of this beyond just the technical points? 1stly this offers an explanation for rapid advancements in AI & prior to that in computational power. Neural networks don't need to have immaculate design -- otherwise human intelligence never would have evolved in the 1st place; starting from a very low level we're only now approaching what already existed in nature. 2ndly this highlights the difference between super human AI & other transhumanist dream technologies. Machines can & do outperform biology, especially if core biological tasks are excluded (§ 2.5), and super human AI should be able to achieve at least as much technologically as humans, but in the case of bottom up adult human brain emulation, radical life extension in adults & spawning substantial extraterrestrial civilization we are dealing with things that are physically very daunting (§ 4, § 5).
People on LW seem to like subjective probabilities [N3]. So I'm also going to play this game.
1) Bottom up adult human brain emulation recreates typical adult human abilities including knowing a prespecified language -- <5%
2) Super human AI could remember your life better than you could & also copy elements of your personality by 2125 -- 75%
3) An adult reading LW in 2025 will live to be 200 years old -- <5%
4) A hypothetical LW nation could by genetic means increase the 95th percentile of human life expectancy to ≥110 years [N4] -- >95%
5) Earth spawns extraterrestrial civilizations which are substantial (eg combined compute > the average of earth's compute in 2025, 2075 & 2125) in the next 10^6 years -- 25%
6) “16 doublings of American billionaire wealth as expressed in purchasing-power-for-human-needs—a roughly 70,000x increase” [N5] or a similarly large increase in the global economy by 2125 -- <5%
N3. I have my issues with this, but that is a different topic.
N4. Assuming such a policy was supported by >50% of the population over an extended period of time. Currently the 95th percentile of life expectancy in the US is 96 years but this drops to <0.01% by 110 years.
N5. From www.lesswrong.com/posts/KFFaKu27FNugCHFmh/by-default-capital-will-matter-more-than-ever-after-agi accessed 2025-01-04. That post does not actually argue in favor of these positions. The 100 year time frame for the prediction is from me.
Things that seem physically very daunting are in a very very different category from some thing that is obviously easily [N6] physically possible. That the latter would achieve the former is a want based belief.
So humans likely (IMO probability > 50%) need to coexist with {super human AI}/{vastly super human AI} for an extended period of time during which payoff to humans from the other technologies is underwhelming. This means that generations of humans are intellectually devalued but without the upsides of living 1000s of years, living on Mars & being able to metamorph into god like AI. I think it's worthwhile to consider both bad & good scenarios of how this may play out.
For such a coexistence to be happy & harmonious the required cultural & political changes are likely substantial. The prevailing obsession is with out of control AI & X risk. Economic, political & social discord may not be an X risk but it can be hugely consequential (WWI, USSR, Great Depression etc) (& likely also increase X risk indirectly). A citizen's income is a significant tangible benefit that could ameliorate the situation. If things progress in a pleasant direction, this could be an opportunity for humans to have more free time with AI doing most (but not all) of the work. This could also be a time for humans to consider things like genetic approaches to life extension from a comfortable position of economic security. In the absence of necessary work such discussions could become a meaningful activity for humans.
N6. Ie with meager physical inputs.
TBD
Hzn