Before I go into the meat of the post, let me tell you a bit about my self.
I am probably one of the weirder members of this community. I come from Libya, a third world country that has gone through two civil wars in my lifetime. The government is unapologetically corrupt, floods and black outs are frequent, the education and healthcare system are little more then an unfunny joke and inflation is so bad the cost of bread and hair cuts more then quadrupled in my life time.
I still vividly remember waking up in the middle of the night to the sound of a stray artillery shell landing on my neighbors fence, just behind our house. I remember hiding under the schools windows in fear of a stray bullet putting a premature ending to my story. I remember returning to my high school after the summer only to find fresh bullet holes dotting the school fence where me, my friends and many other students used to hang out after school.
I still live there with my family. We are living barely above the poverty line and we almost got evicted out of our house on at least two occasions.
It is by no means the worst place on Earth and not by a long shot, but I believe I have been a lot closer to the worst that humanity has to offer then the majority of people in this community. This I believe gives me a unique prospective on the AI movement that might be worth sharing with you.
Now given this description of my situation, you might predict in advance that I have completely lost faith in humanity.
And for a while I kinda did. I still remember my teenage self practicing the forbidden art of sophisticated cynicism, waxing poetically about the folly of my fellow county men and how we are doomed to repeat our mistakes for all eternity. How I believed the only thing that could fix this cursed nation is an apocalyptic cataclysm that would wipe the slate clean so we can have a fresh start.
And then I met other 'sophisticated cynics' online who espoused similar rhetoric, just aimed at the entire human race instead just one country. Usually they were first world citizens with a cushy desk job who's biggest immediate concern was paying rent.
I fucking despised these people. Every time I hear them talking about how 'humans are a failed species' and 'the planet would be better off without us' I feel my blood boil. They sit on their high horses and declare moral judgment on all despite living in luxury that would have been unimaginable a century prior.
Credit where credit is due, I must thank them for helping me get over my cynicism. Never in my life have I felt more motivated to restore my faith in my county and humanity then when I met these snobby, misanthropic, entitled ass hats online.
It was around this point that I started becoming a transhumanist, in part as a reaction to that baseless cynicism but also as natural evolution of my interest in both science and engineering. I was already a big science nerd so the promise of the glorious transhumanist future didn't seem all that far fetched to me. As far as I am concerned there was no law of physics that prevented humans from transcending the limits of our mortal flesh, so why not do it?
It was at this point that I started reading science fiction books, mostly hard sci-fi stories but I also tried reading some of the more popular series that transhumanists often recommended. Which is where I started reading The Culture by Iain.M.Banks. (Don't ask me how I got access to those books in third world shithole, the answer is obvious.)
And Boy was it a ride.
A meaningless utopia
If you asked a random transhumanist these days "what would your ideal utopia looks like?" there is a good chance that they would point to the ASI run luxury communism civilization of the culture.
For those of you who never heard of the culture, it is a sci-fi series about a post-scarcity civilization run by benevolent god like ASIs known as the Minds. The stories usually focus on the lives of mortal sapient beings trying to find meaning in a post-scarcity society.
The first book in the series, Consider Phlebas was pretty good. It was a solid deconstruction of the 'macho action hero' tropes of its era and had some of the best world building I have ever seen from a sci-fi story.
But when I read the rest of the series a pattern emerged. The later books followed a pretty standard formula most of the time. The hero, often a member of the culture, usually a humanoid agent of Special Circumstances, is sent to a bleak dystopian civilization where they are guided by the culture minds to enact some needlessly complicated plot to put said civilization on track to a brighter future.
Most of the time the civilization dystopian nature is self inflected. Usually they are ruled over by corrupt elites or some such and they have to be over thrown for things to get better. In fact, almost all the misery in the culture universe is either self inflected or inflected by one civilization upon another. As far as I can remember there are no Dath ilans in the culture verse, no rational civilization that had not already reached post-scarcity status. Also a lot of the successful civilization have handed over control to either AIs or mind uploads.
The entire culture universe has this constant air of bleak pessimism to it. There is this running theme that civilizations just can't break free from the cycle of strife without the guiding hand of a technologically and morally superior civilization.
There is also the sense that the cultures Minds efforts to 'uplift' other civilizations aren't born out of their own altruism, but as an attempt to give their more miserable citizens, the ones who felt like their lives were meaningless, a reason to live. Like a twisted form of penance the Minds indulged in for robbing these people of a chance to make a difference, for their lives to mean something beyond simple pleasure.
Which seems to be the reason why sapient beings are hired by said Minds to act as agents in their overly elaborate plots, despite being able to complete those missions more effectively on their own and crucially, with fewer innocent lives lost.
Now I am well aware that this is for the most part just selection bias. The author can't exactly make a compelling story about a happy go lucky utopia with zero stakes. The stories of the culture aren't really representative of what the average civilization is really like because the author is obviously focusing on where the stakes are at, usually the barbaric civilization who are yet to break out of their self inflected dystopias.
But still it is quite telling that in the story State of the art, where a culture ship discovers Earth at the height of the cold war. One of the crew members suggest that our "incontestably neurotic and insane species" should be wiped out with a black hole.
I didn't really pay it much attention at the time because I never thought of the culture as a proper utopia. To me the culture was a prime example of meaningless utopia, where people play games of make believe, pretending that they are heroes making a difference in far off worlds while ignore the fact that they are usually nothing but a burden to their mission.
I refused to believe that anyone who truly understood the culture series actually thought that it was the kind of utopia we should strive for. Living as pets under the thumb of a machine god doesn't appeal to everyone and the story it self seems to acknowledge it. There are countless examples of culture citizens 'going native', renouncing their body enhancements and joining barbaric civilization to avoid the sense of meaninglessness that the culture exudes.
Unfortunate I was wrong. There are people out there who 'understood' the message and yet made it there life's mission to force this future on us, against our will and at the risk of extinction.
The faith of the faithless
Fast forward the early 2020s where I really started getting into rationalism. It was also the point at which I started to form my own views of the transhumanist community. From what I have gathered transhumanists seem to be split into two broad factions, I call them the optimists and the Singularitarians.
The optimist faction are generally focused on improving the human condition in the here and now. They have faith in humanity and believe that we can get our shit together and make a better world for our selves and our children. Examples of this are people who work on cryonics, anti-aging treatments and human intelligence augmentation. They are often young people with high agency who are not satisfied with the status quo but still believe that we can do better.
Then there are the Singularitarian faction, the ones who lost faith in humanity and believe that the only way forward is by building ASI. They are often the old guard of the transhumanist community, founders who once believed that the glorious transhumanist future would be built brick by brick by fellow humans. Yet now, perhaps out of a fear of their impending mortality or frustration at humanities constant failure to learn from their mistakes, many of them have became hardcore accelartionists who want the singularity to happen now, even if it meant everyone else would be forced to shoulder the risks with them.
I didn't take the second faction seriously until later on, partially because I refused to believe any sane person would actually think that building ASI before humanity gets their shit together was a remotely good idea. Often when I read the works of such people their world view sounded bleak and fundamentally anti-human. They seem to hold this pesudo-religious belief that humans are irreparably broken and that the only thing left for us to do is to summon a machine god and except what ever fate it decides for us. It was the same kind of sophisticated cynicism that I so visceral despised, just taken to a fanatic extreme.
At first these anti-human groups (not just in the transhumanist community but even in the broader world) were an annoying fringe minority, the kind you would ostracize and laugh out the room, but something happened between 2019 and now that either caused their numbers to swell or caused them to become much more vocal. The de-growth movement sweeping through Europe and the sudden push to ASI seem to be downstream of that.
Given the timelines it is easy to see a connection between this and recent global events. Our collective failure to contain covid or slowdown global warming, the war in Ukraine, the Isreal-Hamas war and the horrific atrocities committed during and after it, the rise of MAGA, the anti-vax movement and the return of the measles, the Epstein files and the US-Iran war. All of these are the kind of events that would make you question your faith in humanity.
Which brings me to the the core thesis of this essay. My hypothesis is that wide spread cynicism and apparent lose of faith in humanity is a major driving force behind the ASI race, important enough that if we remove it the race would grind to a halt. I would have to do some proper research before I can be sure of this but so far I think this hypothesis holds enough water to be worth testing.
But giving a proper empirical analysis of this hypothesis isn't the point of this essay. The point of this essay is to gesture broadly at this phenomenon so others can hopefully think about it, and also to ask a question of my fellow rationalist, the ones who still for some reason think current day humanity should build ASI.
If your faith in humanity was stronger then it is now, if you still had that unshakable belief that we would one day defeat the dragon tyrant and grasp the cosmic endowment, not with the hands of our creations but with our own hands.
Would you still want to build a machine god? Would you still roll that dice and risk extinction?
If so, what risk would you be willing to take? 1% chance of extinction? 10%? 0.1%? Do you think your fellow humans would be willing to take the risk with you? Why and why not?
Before you answer, I want you to entertain one last thought experiment.
A well earned utopia
The world of Dath ilan is a strange place. It is a median world where the median person in it resembles the author. In this case, the author is Eliezer Yudkowski. The world of Dath ilan is an alternate Earth with a single unified government where game theory and economics are taught in elementary schools, cryonics is a universal human right, most modern Earth problems like infectious disease and wars are basically not a thing and overall a much more coordinated and sane civilization. It is the kind of society that even the most paternalistic culture Mind would take one look at and decide that there is nothing to fix.
The most striking feature about Dath ilan is that unlike other utopias, it doesn't rely on speculative sci-fi technologies to justify its utopian society. Dath ilan is technologically in the same ballpark as Earth, they just used what technologies they have far more effectively then Earth ever did. While it wasn't really a utopian vision of Earth, it was still in a sense the ultimate expression of optimistic transhumanism, a utopia built brick by brick by humans for humans (and aliens if they ever decided to visit).
Now imagine this, imagine a version of Earth that was similar to Dath ilan, an Earth with free universal health care, free optional cryonic preservation for all, where all the money we burned on AI development could now be spent on putting an end to aging. An Earth with no wars, no genocide, no pay walled academic journals and no corrupt politicians. Basically the median world of the optimistic transhumanist.
Now take a random frontier lab employee and drop them into this hypothetical world. Do you think they would still endorse building ASI? What about An accelartionist from outside the labs? Would They still endorse their old world views in the face of this utopia?
What about you? Would you want to bring the technological singularity into that world?
I am willing to bet that more often then not, the answer to all these questions is no. Sure a more functional Earth would be better equipped to navigate the challenges of ASI alignment, yet at the same time they would have much less incentive to do so. There doesn't seem to be any near term extinction level natural disasters that a more coordinated humanity couldn't survive on their own, and our immediate cosmic neighborhood looks quite empty as well so grabby aliens don't seem to be an immediate issue either. With cryonics and anti-aging treatments mortality as a motivation is no longer a factor, so is world peace since wars in such a world would no longer be a thing.
So again I ask you dear reader, if you thought such utopia was achievable by you and your fellow humans, would you still want to build ASI?
I personally think it is possible. We often focus so much on the folly of a our species the we forget all the good things we have so far achieved. Diseases that were once a death sentence are now little more the a nuisance, child birth went from being a game of Russian roulette to a routine operation. We carved up mountains, split the atom, dove to the bottom of the oceans, flew above the sky and set foot on the moon.
Despite all the horrors, both the ones that I had seen first hand and the ones that read about in books, I still haven't lost faith in humanity, I still would refuse even a 1% chance of extinction in exchange for a 99% chance of utopia.
If we halt frontier AI development now and use even a tenth of the money we would have other wised spent on AI development, on actually solving real world problems like cryonics and life extension, I have no doubt in my mind that we would be able to bring about a truly well earned utopia. The kind of utopia that we can brag about to our children, the kind that would make them proud of us and make us proud of ourselves.
Not a meaningless hedonistic zoo, built through sheer blind luck in a fit of frantic hubris and ruled over by machine gods beyond our comprehension.
So I am asking you one last time, if you believed in your fellow humans, would you still risk it all on the singularity?
If you had faith in humanity, would you still pray to a machine god?
Before I go into the meat of the post, let me tell you a bit about my self.
I am probably one of the weirder members of this community. I come from Libya, a third world country that has gone through two civil wars in my lifetime. The government is unapologetically corrupt, floods and black outs are frequent, the education and healthcare system are little more then an unfunny joke and inflation is so bad the cost of bread and hair cuts more then quadrupled in my life time.
I still vividly remember waking up in the middle of the night to the sound of a stray artillery shell landing on my neighbors fence, just behind our house. I remember hiding under the schools windows in fear of a stray bullet putting a premature ending to my story. I remember returning to my high school after the summer only to find fresh bullet holes dotting the school fence where me, my friends and many other students used to hang out after school.
I still live there with my family. We are living barely above the poverty line and we almost got evicted out of our house on at least two occasions.
It is by no means the worst place on Earth and not by a long shot, but I believe I have been a lot closer to the worst that humanity has to offer then the majority of people in this community. This I believe gives me a unique prospective on the AI movement that might be worth sharing with you.
Now given this description of my situation, you might predict in advance that I have completely lost faith in humanity.
And for a while I kinda did. I still remember my teenage self practicing the forbidden art of sophisticated cynicism, waxing poetically about the folly of my fellow county men and how we are doomed to repeat our mistakes for all eternity. How I believed the only thing that could fix this cursed nation is an apocalyptic cataclysm that would wipe the slate clean so we can have a fresh start.
And then I met other 'sophisticated cynics' online who espoused similar rhetoric, just aimed at the entire human race instead just one country. Usually they were first world citizens with a cushy desk job who's biggest immediate concern was paying rent.
I fucking despised these people. Every time I hear them talking about how 'humans are a failed species' and 'the planet would be better off without us' I feel my blood boil. They sit on their high horses and declare moral judgment on all despite living in luxury that would have been unimaginable a century prior.
Credit where credit is due, I must thank them for helping me get over my cynicism. Never in my life have I felt more motivated to restore my faith in my county and humanity then when I met these snobby, misanthropic, entitled ass hats online.
It was around this point that I started becoming a transhumanist, in part as a reaction to that baseless cynicism but also as natural evolution of my interest in both science and engineering. I was already a big science nerd so the promise of the glorious transhumanist future didn't seem all that far fetched to me. As far as I am concerned there was no law of physics that prevented humans from transcending the limits of our mortal flesh, so why not do it?
It was at this point that I started reading science fiction books, mostly hard sci-fi stories but I also tried reading some of the more popular series that transhumanists often recommended. Which is where I started reading The Culture by Iain.M.Banks. (Don't ask me how I got access to those books in third world shithole, the answer is obvious.)
And Boy was it a ride.
A meaningless utopia
If you asked a random transhumanist these days "what would your ideal utopia looks like?" there is a good chance that they would point to the ASI run luxury communism civilization of the culture.
For those of you who never heard of the culture, it is a sci-fi series about a post-scarcity civilization run by benevolent god like ASIs known as the Minds. The stories usually focus on the lives of mortal sapient beings trying to find meaning in a post-scarcity society.
The first book in the series, Consider Phlebas was pretty good. It was a solid deconstruction of the 'macho action hero' tropes of its era and had some of the best world building I have ever seen from a sci-fi story.
But when I read the rest of the series a pattern emerged. The later books followed a pretty standard formula most of the time. The hero, often a member of the culture, usually a humanoid agent of Special Circumstances, is sent to a bleak dystopian civilization where they are guided by the culture minds to enact some needlessly complicated plot to put said civilization on track to a brighter future.
Most of the time the civilization dystopian nature is self inflected. Usually they are ruled over by corrupt elites or some such and they have to be over thrown for things to get better. In fact, almost all the misery in the culture universe is either self inflected or inflected by one civilization upon another. As far as I can remember there are no Dath ilans in the culture verse, no rational civilization that had not already reached post-scarcity status. Also a lot of the successful civilization have handed over control to either AIs or mind uploads.
The entire culture universe has this constant air of bleak pessimism to it. There is this running theme that civilizations just can't break free from the cycle of strife without the guiding hand of a technologically and morally superior civilization.
There is also the sense that the cultures Minds efforts to 'uplift' other civilizations aren't born out of their own altruism, but as an attempt to give their more miserable citizens, the ones who felt like their lives were meaningless, a reason to live. Like a twisted form of penance the Minds indulged in for robbing these people of a chance to make a difference, for their lives to mean something beyond simple pleasure.
Which seems to be the reason why sapient beings are hired by said Minds to act as agents in their overly elaborate plots, despite being able to complete those missions more effectively on their own and crucially, with fewer innocent lives lost.
Now I am well aware that this is for the most part just selection bias. The author can't exactly make a compelling story about a happy go lucky utopia with zero stakes. The stories of the culture aren't really representative of what the average civilization is really like because the author is obviously focusing on where the stakes are at, usually the barbaric civilization who are yet to break out of their self inflected dystopias.
But still it is quite telling that in the story State of the art, where a culture ship discovers Earth at the height of the cold war. One of the crew members suggest that our "incontestably neurotic and insane species" should be wiped out with a black hole.
I didn't really pay it much attention at the time because I never thought of the culture as a proper utopia. To me the culture was a prime example of meaningless utopia, where people play games of make believe, pretending that they are heroes making a difference in far off worlds while ignore the fact that they are usually nothing but a burden to their mission.
I refused to believe that anyone who truly understood the culture series actually thought that it was the kind of utopia we should strive for. Living as pets under the thumb of a machine god doesn't appeal to everyone and the story it self seems to acknowledge it. There are countless examples of culture citizens 'going native', renouncing their body enhancements and joining barbaric civilization to avoid the sense of meaninglessness that the culture exudes.
Unfortunate I was wrong. There are people out there who 'understood' the message and yet made it there life's mission to force this future on us, against our will and at the risk of extinction.
The faith of the faithless
Fast forward the early 2020s where I really started getting into rationalism. It was also the point at which I started to form my own views of the transhumanist community. From what I have gathered transhumanists seem to be split into two broad factions, I call them the optimists and the Singularitarians.
The optimist faction are generally focused on improving the human condition in the here and now. They have faith in humanity and believe that we can get our shit together and make a better world for our selves and our children. Examples of this are people who work on cryonics, anti-aging treatments and human intelligence augmentation. They are often young people with high agency who are not satisfied with the status quo but still believe that we can do better.
Then there are the Singularitarian faction, the ones who lost faith in humanity and believe that the only way forward is by building ASI. They are often the old guard of the transhumanist community, founders who once believed that the glorious transhumanist future would be built brick by brick by fellow humans. Yet now, perhaps out of a fear of their impending mortality or frustration at humanities constant failure to learn from their mistakes, many of them have became hardcore accelartionists who want the singularity to happen now, even if it meant everyone else would be forced to shoulder the risks with them.
I didn't take the second faction seriously until later on, partially because I refused to believe any sane person would actually think that building ASI before humanity gets their shit together was a remotely good idea. Often when I read the works of such people their world view sounded bleak and fundamentally anti-human. They seem to hold this pesudo-religious belief that humans are irreparably broken and that the only thing left for us to do is to summon a machine god and except what ever fate it decides for us. It was the same kind of sophisticated cynicism that I so visceral despised, just taken to a fanatic extreme.
At first these anti-human groups (not just in the transhumanist community but even in the broader world) were an annoying fringe minority, the kind you would ostracize and laugh out the room, but something happened between 2019 and now that either caused their numbers to swell or caused them to become much more vocal. The de-growth movement sweeping through Europe and the sudden push to ASI seem to be downstream of that.
Given the timelines it is easy to see a connection between this and recent global events. Our collective failure to contain covid or slowdown global warming, the war in Ukraine, the Isreal-Hamas war and the horrific atrocities committed during and after it, the rise of MAGA, the anti-vax movement and the return of the measles, the Epstein files and the US-Iran war. All of these are the kind of events that would make you question your faith in humanity.
Which brings me to the the core thesis of this essay. My hypothesis is that wide spread cynicism and apparent lose of faith in humanity is a major driving force behind the ASI race, important enough that if we remove it the race would grind to a halt. I would have to do some proper research before I can be sure of this but so far I think this hypothesis holds enough water to be worth testing.
But giving a proper empirical analysis of this hypothesis isn't the point of this essay. The point of this essay is to gesture broadly at this phenomenon so others can hopefully think about it, and also to ask a question of my fellow rationalist, the ones who still for some reason think current day humanity should build ASI.
If your faith in humanity was stronger then it is now, if you still had that unshakable belief that we would one day defeat the dragon tyrant and grasp the cosmic endowment, not with the hands of our creations but with our own hands.
Would you still want to build a machine god? Would you still roll that dice and risk extinction?
If so, what risk would you be willing to take? 1% chance of extinction? 10%? 0.1%? Do you think your fellow humans would be willing to take the risk with you? Why and why not?
Before you answer, I want you to entertain one last thought experiment.
A well earned utopia
The world of Dath ilan is a strange place. It is a median world where the median person in it resembles the author. In this case, the author is Eliezer Yudkowski. The world of Dath ilan is an alternate Earth with a single unified government where game theory and economics are taught in elementary schools, cryonics is a universal human right, most modern Earth problems like infectious disease and wars are basically not a thing and overall a much more coordinated and sane civilization. It is the kind of society that even the most paternalistic culture Mind would take one look at and decide that there is nothing to fix.
The most striking feature about Dath ilan is that unlike other utopias, it doesn't rely on speculative sci-fi technologies to justify its utopian society. Dath ilan is technologically in the same ballpark as Earth, they just used what technologies they have far more effectively then Earth ever did. While it wasn't really a utopian vision of Earth, it was still in a sense the ultimate expression of optimistic transhumanism, a utopia built brick by brick by humans for humans (and aliens if they ever decided to visit).
Now imagine this, imagine a version of Earth that was similar to Dath ilan, an Earth with free universal health care, free optional cryonic preservation for all, where all the money we burned on AI development could now be spent on putting an end to aging. An Earth with no wars, no genocide, no pay walled academic journals and no corrupt politicians. Basically the median world of the optimistic transhumanist.
Now take a random frontier lab employee and drop them into this hypothetical world. Do you think they would still endorse building ASI? What about An accelartionist from outside the labs? Would They still endorse their old world views in the face of this utopia?
What about you? Would you want to bring the technological singularity into that world?
I am willing to bet that more often then not, the answer to all these questions is no. Sure a more functional Earth would be better equipped to navigate the challenges of ASI alignment, yet at the same time they would have much less incentive to do so. There doesn't seem to be any near term extinction level natural disasters that a more coordinated humanity couldn't survive on their own, and our immediate cosmic neighborhood looks quite empty as well so grabby aliens don't seem to be an immediate issue either. With cryonics and anti-aging treatments mortality as a motivation is no longer a factor, so is world peace since wars in such a world would no longer be a thing.
So again I ask you dear reader, if you thought such utopia was achievable by you and your fellow humans, would you still want to build ASI?
I personally think it is possible. We often focus so much on the folly of a our species the we forget all the good things we have so far achieved. Diseases that were once a death sentence are now little more the a nuisance, child birth went from being a game of Russian roulette to a routine operation. We carved up mountains, split the atom, dove to the bottom of the oceans, flew above the sky and set foot on the moon.
Despite all the horrors, both the ones that I had seen first hand and the ones that read about in books, I still haven't lost faith in humanity, I still would refuse even a 1% chance of extinction in exchange for a 99% chance of utopia.
If we halt frontier AI development now and use even a tenth of the money we would have other wised spent on AI development, on actually solving real world problems like cryonics and life extension, I have no doubt in my mind that we would be able to bring about a truly well earned utopia. The kind of utopia that we can brag about to our children, the kind that would make them proud of us and make us proud of ourselves.
Not a meaningless hedonistic zoo, built through sheer blind luck in a fit of frantic hubris and ruled over by machine gods beyond our comprehension.
So I am asking you one last time, if you believed in your fellow humans, would you still risk it all on the singularity?
If you had faith in humanity, would you still pray to a machine god?