This is a special post for quick takes by Kaarel. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
Positives of a future with human capability improvement over an AI future.
Beren Millidge has an essay arguing for the claim that a future in which humanity proceeds with biological capability increasing is scarier than a future in which we develop AGI. Here are some concluding sentences from his essay:
Ultimately, human intelligence amplification and the resulting biosingularity has a deeper and more intractable alignment problem than AI alignment, at least if we don’t just assume it away by asserting that humans and our transhuman creations just have some intrinsic and ineffable access to ‘human values’ that potential AIs lack.
The only potential positive as regards alignment of the biosingularity is that it will happen much later, most likely in the closing decades of the 21st century and around the end of the natural lifespans of my personal cohort. This gives significantly more time to prepare than AGI, which is likely coming much sooner, but the problem is much harder and requires huge advances in neuroscience and understanding of brain algorithms to even reach the level of control we have over today’s AI systems (which is likely far from sufficient).
I disagree with his thesis — I think that instead of creating AIs smarter than humans, it would be much better to proceed with increasing the capabilities of humans (for at least the next 100 years). [1] [2] Millidge's claim that the only potential positive of a biofoom is that it starts later seems clearly false. In this note, I will list other imo important (pro tanto) positives of a human foom. I agree with Millidge that there are also positives of the AGI path; [3] these won't be discussed in the present note. To assess which path is better overall, one could want to compare the positives of the human path to the positives of the AGI path, [4] but I will not do that here. The rest of this note is the list of positives. [5]
more similar entity => more similar values
an argument in favor of the human path:
- assumption 1. it makes sense to speak of the values of an entity, and the values of an entity are some sort of kinda-smooth function of the entity's structure/constitution
- assumption 2. humanity (taken as an entity) currently has pretty good values
- conclusion. somewhat modified humanity will still have pretty good values. in particular, a somewhat bioenhanced humanity will still have pretty good values
- in contrast: an AGI future will involve the-process-happening-on-earth changing a lot more from what it is now, certainly by each point in time, but also by the time each higher capability level is reached
we can also give a similar argument for single humans:
- assumption 1. it makes sense to speak of the values of an entity, and the values of an entity are some sort of kinda-smooth function of the entity's structure/constitution
- assumption 2'. individual humans growing up in current human societies have pretty good values
- conclusion'. somewhat modified individual humans growing up in somewhat modified societies will still have pretty good values
- in contrast: an AGI future will involve AGIs that are much more different than these future humans growing up in contexts that are much more different from the contexts in which current humans grow up. this is certainly true by each time (or each time after the beginning of each "foom proper"), but also by the point each higher capability level is reached
said another way:
- the target of right values is drawn largely around where the arrow(s) determining the values that humans have landed. shooting more similar arrows in a similar way is a decent strategy for hitting that target again.
[humans are]/[humanity is] just cool
- on the biofoom path, it will still be humanity, made of humans. humanity is cool. humans are cool. humans with increased capabilities and humanity with increased capabilities would be cool as well. like, of course it's possible for us to become even much cooler, but we're already occupying a very specific rare cool region in mindspace
- the smarter humans you make will be slotting into existing human institutions/organizations/communities. existing human institutions/organizations/communities will still be useful to the smarter humans. this is a reason for existing human institutions/organizations/communities to persist/thrive/develop. [6] and existing human institutions are carriers/supporters/implementers of human values and also cool
- the more capable humans will be continuing existing human (social, political, artistic, technological, scientific, mathematical, philosophical) projects and traditions. human thought will continue to develop. this is cool
- like, even if the humans with improved capabilities were to (let's say) form their own state and build a massive biodome around it and cause the outside to eventually become uninhabitable by polluting it or militarily leveling it to make room for automated factories or whatever, killing all other humans, that would be a very bad thing for them to do, but it would still be a human future, and that's pretty important
an analogy:
- suppose you are at intelligence level
- the agent could just be you after having taken a linear algebra course or after inventing some new methods in numerical linear algebra
- the agent could just be you after getting gene therapy which changes a single genetic variant to one that better supports learning (even in adulthood)
- the agent could be some sort of novel mind you create from scratch by some training procedure (ok maybe involving some culture which was also involved when you grew up)
- now, there is at least some range of your mind-making precision/understanding parameter from
- to spell out the analogy: with humanity as the agent, the human path is like becoming smarter via gene therapy and learning linear algebra, whereas the AI path is like creating a novel thing happening on earth largely in novel ways from scratch [9]
reasons why human thought would be guiding development more/better in a biofoom
- we have a lot of experience with and understanding about humans and specifically how to raise humans
- for example, voters, politicians, and researchers will all have much higher-quality starting intuitions about what millions of human einsteins would be like than about what a bunch of AIs would be like
- biofooming will be happening later, so we will have more time to prepare before it starts happening [10]
- but also biofooming will be happening much slower, so humans will have more time to think about each improvement
- like, the speed of development will be slower compared to the speed of human thought, so more human thought can go into each step of development. each unit step of development will be more human-thought-fully chosen
- this is true at the level of individual humans thinking about what should be done, and also true at the level of institutions and governments (like, a government running at human speed will be better able to legislate a slow biofoom)
reasons a biofooming world would be diffusely human-friendly (sociopolitical factors, anti-[gradual disempowerment] stuff)
- since humans will still be expensive to create and will continue to run on [at least order
- it will be somewhat difficult/weird for more capable humans to render the environment unlivable to less capable humans, because all humans have basically the same environmental requirements (for now) [11]
- ditto for good governance, laws, norms. e.g. it'd be quite natural for a law that makes it harder to manipulate the parents of an IVF einstein out of their property to also make it harder to manipulate the amish out of their property
- ditto for organizations and economic structures. e.g. it would be very natural for AIs to have a research process that operates in neuralese on a server, with translation costs being high enough that even if a human could do some useful task, doing the task yourself is cheaper than "translating" the task and context to a human; this effect is still present in human-human interactions but the degree to which it is present on the human path is smaller than the degree to which it is present on the AI path
- empathy is easier/"more natural" toward beings that are more like you. when agent B is more similar to agent A, it is more likely that [golden rule]/[categorical imperative] style reasoning leads A to treat B well [12]
- power concentration stuff is less bad in the biofoom case. it's not like there would be some entity controlling these more capable humans. these humans will be growing up in various families, involved in various communities/cultures/nations, going to various schools. in the biofoom case, RSI will not be localized to a single lab [13] . monopolies are much more natural in the AI case than in the biofoom case
- the AGI path involves power shifting away from people (and to AIs or companies) a lot more
- fooming happening slower means that the change in technological/social/cultural/environmental conditions during any
- fooming happening slower compared to the pace of life means that more life can be lived by existing living beings before being at the frontier of development start to find them boring and useless (or like, i don't want to say that this happens necessarily, but there is a force pushing in this direction)
- in a biofoom, there will be a graph of caring connecting most humans to most other humans with not that many steps; it will only have such-and-such clustering coefficients
- human institutions, organizations, systems are generally more likely to survive for longer. in particular, the following specific human institutions/organizations/systems supporting the ability of each human to live a good long life are more likely to survive for longer: states, laws, systems enforcing laws, social safety nets, democracy, cryonics organizations
- technologies which benefit humans with higher capabilities will also be somewhat likely to benefit current-100-iq humans. eg cures to diseases and other medical treatments, better educational methods, new words/concepts, most consumer technologies. if humans remain central to [doing stuff]/[the economy], then a large fraction of economically rewarded innovation will be making humans more capable, and so technological progress will generally be pointed in a more humane direction
- in general: various mad local forces at play in the world (greed, status-seeking, etc) will stay pointed in a more humane direction
- also, governments will be much better able to understand, track, and deal with messy world problems in the biofoom case
we know humans [can be and often are] kind/nice to other humans
about humans, we know the following:
- most humans consider it very bad to personally kill other humans, and would not kill other humans in normal circumstances. most humans consider the killing of humans highly reprehensible in general [14]
- most humans consider it bad to steal from humans. most humans think human property rights should basically be respected. most humans are not in favor of taking property from people. most people would consider it very bad to take property from a person such that this person would then be unable to live an alright life
- probably there are some humans who have a deep enough commitment to humanity that they'd remain nice to all existing humans even after individually fooming a lot (though note this won't be happening in the human foom)
also:
- there are specific steps/niches in human evolutionary history which made us nicer to unrelated strangers
other factors
- there are factors much more tightly constraining capabilities in the biofoom case than in the AI foom case:
- brain volume is tough to increase by idk more than
- roughly, there is some not-THAT-large finite number of intelligence-increasing "genetic ideas" in the current gene pool, and for at least some initial period these set a cap on how far you can go biologically. it will be very hard to genetically write novel more capable human learning algorithms. the genome interface to mind reprogramming is kinda cursed
- it will be expensive and potentially immoral and illegal to "run experiments"
- brain volume is tough to increase by idk more than
- humanity's correct values are somewhat well-tracked by reflection / self-endorsed development, and the biofoom path will be like that
- it makes sense to speak of what sort of reflection and development should happen, distinguishing this from development that is likely to happen. it is very much not true that anything goes! if it were likely/natural that our society would get replaced by a molecular squiggle maximizer, that wouldn't mean that our society's true values are to make lots of those molecular squiggles, and that wouldn't mean this was what should happen all along. whereas if we reflect carefully and understand more stuff and become better versions of ourselves and conclude that we should make lots of some molecular squiggles, then it's plausible that this was what should happen all along.
- it is (at least prima facie) extremely scary/reckless to have a step in your development where you hand things over to some novel mind created largely from scratch! this is done much more on the AI path than on the human path
- i say some more on good development and also the general topic of this note here: https://www.lesswrong.com/posts/iemgJhjNLa5eyevWR/kh-s-shortform?commentId=PHm2ZkagfyrhT2Wvz
a concluding remark
self-improvement generally has many good properties over creating a new agent/mind from scratch [15]
i think we should ban AGI ↩︎
It is also a possibility that both options are bad. My view is that we should push forward with increasing human capabilities biologically and culturally/educationally. But I think this question deserves serious analysis, and there are certainly specific things here that one should be very careful about and regulate. However, this note will not be analyzing this question. ↩︎
That said, I think the analysis of the positives in his essay gets very many things wrong. ↩︎
But one also doesn't have to do that, to compare the two paths. One can also just "directly" think about what would happen along each path. ↩︎
They are not listed in order of importance. ↩︎
clearly this is a reason for these to be around for longer in wall clock time, but also it's a reason for them to be around until higher capability levels ↩︎
in fact it seems plausible that, at least if the mind design has to be done from your own bounded perspective, it will keep being better to self-improve forever. i think this is plausible on this individual future life coolness axis and also all-things-considered. ↩︎
ok, if you (imo incorrectly) believe in some sort of soul theory of personal identity then maybe to make this a fair example you would need to imagine the soul getting detached in all three examples, but then maybe you will think all of these are suicides... so maybe this isn't a good analogy for you... but hmm i guess maybe you should also in the same sense believe in there being a soul attached to each society though, and then it would be a good analogy actually ↩︎
ok, there's a meaningful amount of shared culture. even if you thought human minds and AI minds are "mostly cultural" and that subbing out representation/learning/etc algorithms/structures and radically changing learning contexts doesn't make it legit to say AGI will be a novel thing created largely in novel ways from scratch, it is still at least a really big step; it's still creating some totally new guy ↩︎
as Millidge says ↩︎
yes, there are sort of examples like climate change. but it is still much easier to imagine entities that are just programs that can be run on arbitrary computers being totally fine with or even preferring a very different environment. there is a large difference in degree here ↩︎
and we at least know humans have a propensity to carry out and be moved by this style of reasoning ↩︎
or three labs or whatever. In reality, I think only a single lab will matter, absent strict capability regulation. ↩︎
there is a major issue around diffuse effects — like, people and institutions currently not tracking [decreasing the lifespan of
this is also important to track when thinking about AIs making more capable AIs ↩︎
3