Why did you pick caring about each other as a thing culture + evolution was trying to do?

You're not alone in making what I think is the same mistake. I think it's actually quite common to feel like it's amazing that evolution managed to come up with us humans who like beauty and friendship and the sanctity of human life and so on - evolution and culture must have been doing something right, to come up with such great ideas.

But in the end, no; evolution is impressive but not in that way. You picked caring about each other as the target because humans value it - a straightforward case of painting the target around the bullet-hole.

Evolution actually solved alignment (approximately) on two levels: aligning brains with evolution's goal of inclusive genetic fitness - to be fruitful and multiply - (even though the easy/natural goal attractor for intelligence is empowerment), and aligning brains with other brains because the unit of selection (genes) is spread across individual bodies (the disposable soma). There are many obvious failure examples, but that's hardly surprising because evolution proceeds by breaking things.

The first type of alignment (towards genetic fitness) gives rise to lust, sexual attraction, parenting desires, etc. The second form of alignment (to other humans) - more relevant for AGI - manifests as empathy, love, and altruism.

No, and here's why:

1. Evolution's goal, to the extent that it even has a goal, is so easy to satisfy that literally any life/self-replicating tech could do this. It cares about reproductive fitness, and that goal requires almost no capabilities. It cares nothing for specifics, but our goals require much, much more precision than that.

2. Evolution gives almost zero optimization power to capabilities, while far far more optimization power is dedicated to capabilities by the likes of Deepmind/OpenAI. To put it lightly, there's a 30-60% chance that we get much, much stronger capabilities than evolution this century.

3. The better example is how humans treat animals, and here humans are very misaligned to animals, with the exception of pets, despite not agreeing with environmentalism/nature as a good thing. So no, I disagree with the premise of your question.

Humans obtain value from other humans and depend on them for their existence. It is hypothesized that AGIs will not depend on humans for their existence.  Thus, humans who would not push the button to kill all other humans may choose not to do so for reasons of utility that don't apply to AGI.  Your hypothetical assumes this difference away, but our observations of humans don't.  

As you not, human morality and values were shaped by evolutionary and cultural pressure in favor of cooperation with other humans.  The way this presumably worked is that humans who were less able or willing to cooperate tended to die more frequently. And cultures that were less able or willing to do so were conquered and destroyed.   It is unclear how we would be able to replicate this or how well it translates.

It is unclear how many humans would actually choose to press this button.   Your guess is that between 5% and 50% of humans would choose to do so.  

That doesn't suggest humans are very aligned; rather the opposite.  It means that if we have between 2 and 20 AGIs (and those numbers don't seem unreasonable) between 1 and 10 would choose to destroy humanity.   Of course, extinction is the extreme version; having an AGI could also result in other negative consequences.

For sexually reproducing species, evolution will disfavour genes that lead to individuals slaughtering others of their species for little or no reason. Slaughtering members of the opposite sex obviously reduces mating opportunities. Slaughtering members of the same sex reduces the mating opportunities of their opposite-sex children.

This obviously hasn't completely prevented humans from slaughtering both particular individuals and entire populations of humans of a different tribe.  But this isn't what always happens.  It could be a lot worse.

Maybe it is worse for asexual species (mitigated by slower evolution without sex), perhaps explaining why at least for vertebrates asexuality is rare, and asexual species seem to not last long.