I acknowledge that I am skipping ahead a bit here however, I hope you will grant me this in order to gain some useful perspective.
AGI is in reach. AGI represents a second data point, after homo sapiens, in the development of general intelligence. Given that we have demonstrated this is possible what insights can we draw?
Firstly, that general intelligence is an emergent property that is not confined to biological systems and is better thought to be a natural phenomenon dependent upon a threshold of complexity within an appropriate architecture. It may therefore be seen as a universal property and a natural progression of any system sufficiently stable enough to allow the development of complex architectures.
Secondly, it is a second data point. In a similar way to finding a second data point on the existence of life elsewhere in the universe, it is a key milestone to reflect on where we should view our position and trajectory as a precursor (creator of) general intelligence.
Thirdly, alignment of general intelligences. In the case of homo sapiens our intelligence has been aligned to our environment through a gradual process of evolution by natural selection and later through the more rapid process of cultural evolution. It could be argued that we do see breakdown in the alignment of homo sapiens which is manifested as psychopathy. The fundamental problem with psychopathy within AGI is that it will have properties of superintelligence whilst simultaneously lacking the property of alignment. The concerns of which have been put far more eloquently by multiple others within this sphere.
What conclusions can we draw from the above? The natural go-to intuition pump is the Drake equation however, in the light of these points I would modify the variable L of the Drake equation from 'The average lifetime of a communicating civilization in years' to 'The average lifetime of a communicating civilization in years before the creation of an unaligned AGI'. If you extrapolate from our index case of homo sapiens this would dramatically reduce the probability of hearing from extraterrestrial life in keeping with what we find experimentally.
A second data point therefore gives cause for concern.
I acknowledge that I am skipping ahead a bit here however, I hope you will grant me this in order to gain some useful perspective.
AGI is in reach. AGI represents a second data point, after homo sapiens, in the development of general intelligence. Given that we have demonstrated this is possible what insights can we draw?
Firstly, that general intelligence is an emergent property that is not confined to biological systems and is better thought to be a natural phenomenon dependent upon a threshold of complexity within an appropriate architecture. It may therefore be seen as a universal property and a natural progression of any system sufficiently stable enough to allow the development of complex architectures.
Secondly, it is a second data point. In a similar way to finding a second data point on the existence of life elsewhere in the universe, it is a key milestone to reflect on where we should view our position and trajectory as a precursor (creator of) general intelligence.
Thirdly, alignment of general intelligences. In the case of homo sapiens our intelligence has been aligned to our environment through a gradual process of evolution by natural selection and later through the more rapid process of cultural evolution. It could be argued that we do see breakdown in the alignment of homo sapiens which is manifested as psychopathy. The fundamental problem with psychopathy within AGI is that it will have properties of superintelligence whilst simultaneously lacking the property of alignment. The concerns of which have been put far more eloquently by multiple others within this sphere.
What conclusions can we draw from the above? The natural go-to intuition pump is the Drake equation however, in the light of these points I would modify the variable L of the Drake equation from 'The average lifetime of a communicating civilization in years' to 'The average lifetime of a communicating civilization in years before the creation of an unaligned AGI'. If you extrapolate from our index case of homo sapiens this would dramatically reduce the probability of hearing from extraterrestrial life in keeping with what we find experimentally.
A second data point therefore gives cause for concern.