I don't think the assumption is highly dubious. You don't need to believe in the possibility of mind uploading or biological immortality to assume radically transformative changes in the human condition due to advanced AI. The "Neuroscience and Mind" section of Dario Amodei's (who has a formal background in biophysics) essay attempts to clearly speculate what'll happen in these areas with the helped of advanced AI (even setting aside that mind uploading is probably "possible in principle").
Even if some goals are unattainable, AGI could still likely (as Dario speculates) drive radical advancements in areas like health, longevity, and cognitive enhancement. The point isn't to guarantee specific outcomes, but to recognise that AGI will likely push the boundaries of what we currently believe is possible and transform the world unrecognisably. And we should be mentally reflecting and preparing for that.
It’s widely accepted that powerful general AI, and soon after, superintelligence, may eventually be created.[1] There’s no fundamental law keeping humanity at the top of the intelligence hierarchy. While there are physical limits to intelligence, we can only speculate about where they lie. It’s reasonable to assume that even if we hit an S-curve in progress, that plateau will be far beyond anything even 15 John von Neumann clones could imagine.
Gwern was one of the first to recognise the "scaling hypothesis"; others followed later. While debate continues over whether scaling alone will lead to AI systems capable of self-improvement, it seems likely that scaling, combined with algorithmic progress and... (read 1302 more words →)
I agree that there are significant uncertainties on the specific consequences of AI accelerating bio/medicine R&D, but I think even without buying into Amodei's specific speculations on life extension, you would still get wildly transformative breakthroughs and unforeseen consequences. I do agree it seems to make sense to be wary of just extrapolating past increases in life expectancy.
Time will tell!