I had a very long writeup on this but I had a similar journey from identifying as a transhumanist to deeply despising AI, so I appreciate seeing this. I'll quote part of mine and perhaps identify:

"I worked actively in frontier since at least 2012 including several stints in "disruptive technology" companies where I became very familiar with the technology cult perspective and to a significant extent, identified. One should note that there is a definitely healthy aspect to it, though even the most healthiest aspect is, as one could argue, colonialist - the idea of destructive change in order to "make a better world."

Until 2023..

And yet I also had a deep and abiding love of art, heavily influenced by "Art as Prayer" by Andrei Tarkovsky, and this actually fit into a deep and abiding spiritual, even religion viewpoint that integrated my love of technology and love of art.

In my spare time and nontrivially, I spent a lot of time writing, drawing here and there, as well as writing poetry - all which had a deep and significant meaning to me.

---

In the Hermetic system, Man is the creation of clay and spirit imbued with the divine spark to create; this is why science and technology can be seen as a form of worship, because to learn of Nature and to advance the "occult" into the known is part of the Divine Science to ultimately know God and summon the spirits via the Machine; the computer is the modern occultist's version of the pentagram and circle. But simultaneous with this, is the understanding that the human is the one who ultimately is the creator, who "draws imagination from the Empyrean realms, transforms them into the material substance, and sublimates it into art" as glimpses of the magical. Another version of this is in the Platonic concept of the Forms, which are then conveyed into actual items via the labor of the artist or the craftsman.

As such, the deep love of technology and the deep love for art was not in contrast in the least, because one helped the other and in both, the ultimately human spirit was very much glorified. Perhaps there is a lot of the cyborg in it, but the human is never extinguished.

---

The change came in last year with the widespread release of Chatgpt. I actually was an early user of Midjourney and played around with it as an ideation device, which I would have never faulted a user of AI for. But it was when the realization that the augmentation of that which is human was exchanged for the replacement of human that I knew that something had gone deeply, terribly wrong with the philosophy of the place.

Perhaps that was not the only reason: another transformative experience was reading Other Minds: The Octopus, the Sea, and the Deep Origins of Consciousness which changed a lot of my awareness of life from something of "information"(which is still the default thinking of techbros, which is why mind-uploading is such a popular meme) into the understanding of the holobiont that life, even in an "organism" is in fact a kind of enormous ecosystem of many autonomous or semi-autonomous actors as small as the cellular level(or even below!). A human is in fact almost as much "not human" as human - now while the research as been revised slightly, it still comes out that an average human is perhaps at most 66% human, and many of our human cells show incredible independent initiative. Life is a "whorl of meaning" and miraculous on so many fundamental levels.

And now, with the encroachment of technology, was the idea of not only replacing humanity with the creation of art, one of the most fundamentally soulful things that we do, but the idea of extinguishing life for some sort of "more perfection." Others have already talked about this particular death cult of the tech-bros, but this article covers the omnicidal attitude well, which goes beyond just the murder of all of humanity but also the death of all organic life. Such attitudes are estimated to be among AI researchers between 5 to 10%, the people who are in fact, actively leading us down this path who literally want you and everything you love to die.

I responded to the widespread AI the same way with a lot of other researchers in the area - initially with a mental breakdown(which I am still recovering from) and then with the realization that with the monster that had been created, we had to make the world as aware as possible and to try to stop it.

Art as Transcendence

Art is beauty, art is fundamental to human communication and I believe that while humans exist, we will never stop creating art. But beyond that, is the realization of art as transcendent - it speaks of the urgency of love, the precariousness of life, the marvel that is heroism, the gentleness of caring, and all of the central meaning of our existence, things which should be eternal and things I truly realize now are part and parcel of our biology. One doesn't have to relate to a supernatural soul to know that these things should be infused into our very being and that they are, by nature, not part of any mechanical substrate. A jellyfish alien that breathes through its limbs would be more true to us than a computer with a human mind upload in that sense; for anything that is organic is almost certainly that same kind of ecosystem, of life built upon life, of a whorl of meaning and ultimately of beauty, while the machine is a kind of inert death, and undeath in its simulacrum and counterfeit of the human being.

And in that, I realized that there was a point where the human must be protected from the demons that we are trying to raise, because no longer are we trying to summon aid to us, but so many of the technologists have become besotted and have fallen in love with their own monsters, and not with the very life that has created us and given us all this wonderous existence to feel, to breathe and to be. "

I think you are incorrect on dangerous use case, though I am open to your thoughts. The most obvious dangerous case right now, for example, is AI algorithmic polarization via social media. As a society we are reacting, but it doesn't seem like it is in an particularly effectual way.

Another way to see this current destruction of the commons is via automated spam and search engine quality decline which is already happening, and this reduces utility to humans. This is only in the "bit" universe but it certainly affects us in the atoms universe and as AI has "atom" universe effects, I can see similar pollution being very negative for us.

Banning seems hard, even for obviously bad use cases like deepfakes, though reality might prove me wrong(happily!) there.

Its not a myth, but an oversimplification which makes the original thesis much less useful. The mind, as we are care about, is a product and phenomenon of the entire environment it is in, as well as the values we can expect it to espouse.

It would indeed be akin to taking an engine, putting it in another environment like the ocean and expecting the similar phenomenon of torque to rise from it.

Lifelong quadriplegics are perfectly capable of love, right?

As a living being in need of emotional comfort and who would die quite easily, it would be extremely useful to express love to motivate care and indeed excessively so. A digital construct of the same brain would have immediately different concerns, e.g. less need for love and caring, more to switch to a different body, etc.

Substrate matters massively. More on this below.

Again, an perfect ideal whole-brain-emulation is a particularly straightforward case. A perfect emulation of my brain would have the same values as me, right?

Nope! This is a very common and yet widespread error, which I suppose comes from the idea that the mind comes from the brain. But even casually, we can tell that this isn't true: would a copy of you, for example, still be recognizably you if put on a steady drip of cocaine? Or would it still be you if you were permanently ingesting alcohol? Both would result in a variation of you that is significantly different, despite otherwise identical brain. Your values would likely have shifted then, too. Your brain is identical - only the inputs to it have changed.

In essence, the mind is the entire body, e.g.

https://www.psychologytoday.com/us/blog/body-sense/202205/the-fiction-mind-body-separation

There is evidence that even organ transplants affect memory and mood.

https://www.sciencedirect.com/science/article/abs/pii/S0306987719307145#:~:text=Neuroplasticity is one of the,at the time of transplantation.

The key here is that the self is always a dynamic construct of the environment and a multiplicity of factors. The "you" in a culture of cannibalism will likely have different values than a "you" in a culture of Shakers, to add to it.

The values of someone who is a digital construct who doesn't die and doesn't need to reproduce very much will be very different from a biological creature that needs emotional comfort, values trust in an enviromment of social deception, holds heroism in high regard due to the fragility of life, and needs to cooperate with other like minds.

Is it theoretically possible? If you replicate all biological conditions to a digital construct, perhaps but its fundamentally not intrinsic to the substrate, where digital substrate entails perfect copying via mechanical processes, while biology entails dynamic agentic cells in coordination and much more variability in process. Its like trying to use a hammer to be a screwdriver.

The concept of the holobiont goes much deeper into this and is a significant reason why I think any discussion of digital copying is the equivalent of a shadowy undead mockery than anything else, since it fails to account for the fundamental co-evolutions that build up an "organism."

https://en.m.wikipedia.org/wiki/Holobiont

In life, holobionts do change and alter, but its much more like evolutional extensions and molding by degree. Mechanism just tromps over it by fiat.

But you do pass on your consciousness in a significant way to your children through education, communication and relationships and there is an entire set of admirable behaviors selected around that.

I generally am less opposed to any biological strategy, though the dissolution of the self into copies would definitely bring up issues. But I do think that anything biological has significant advantages in that ultimate relatedness to being, and moreover in the promotion of life: biology is made up of trillions of individual cells, all arguably agentic, which coordinate marvelously into a holobioant and through which endless deaths and waste all transform into more life through nutrient recycling.

I am in Vision 3 and 4, and indeed am a member of Pause.ai and have worked to inform technocrats, etc to help increase regulations on it.

My primary concern here is that biology remains substantial as the most important cruxes of value to me such as love, caring and family all are part and parcel of the biological body.

Transhumans who are still substantially biological, while they may drift in values substantially, will still likely hold those values as important. Digital constructions, having completely different evolutionary pressures and influences, will not.

I think I am among the majority of the planet here, though as you noted, likely an ignored majority.

I don't mind it: but not in a way that wipes out my descendants, which is pretty likely with AGI.

I would much rather die than to have a world without life and love, and as noted before, I think a lot of our mores and values as a species comes from reproduction. Immortality will decrease the value of replacement and thus, those values.

I generally feel that biological intelligence augmentation, or a biosingularity is by far the best option and one can hope such enhanced individuals realize to forestall AI for all realistic futures.

With biology, there is life and love. Without biology, there is nothing.

Its not merely the rejection of God, its a story of "progress" to reject also reverence of nature and eventually, even life and reality itself, presumably so we can accept mass extinction for morally superior machines.