I saw that Katja Grace has said something similar here; I'm just putting my own spin on the idea.
The relevance of the evolutionary analogy for inner alignment has been long discussed in this community, but one observation that seems to not be mentioned is that humans are still... pretty good at inclusive genetic fitness? Even in way-out-of-distribution environments like modern society, we still have strong desires to eat food, stay alive, find mates and reproduce (although the last one has relatively decreased recently; IGF hasn't totally generalized). We don't monomanically optimize for IGF, but we (and probably future NN-based AIs) don't monomanically optimize for anything, and our motivational circuits still do a pretty good job at keeping our species alive. So... why should we expect future AIs to catastrophically fail (i.e. be completely non-inner aligned with what we wanted it to do) at doing the actions we rewarded in RL training, which should be a much stronger outer optimizer than evolution?
Some possible objections:
We (or at least a majority of humans) do still have inner desires to have kids, though; they just get balanced out by other considerations, mostly creature comforts/not wanting to deal with the hassle of kids. But yeah, evolution did not foresee birth control, so that's a substantial misgeneralization.
We are still a very successful species overall according to IGF, but birth rates continue to decline, which is why I made my last point about inner alignment possibly drifting farther and farther away the stronger the inner optimizer (e.g. human culture) becomes.