Many influential AI techniques either explicitly draw inspiration from or are similar to mechanisms found in biology. 

• Biological neurons and artificial neural networks
• The visual cortex and convolutional neural networks
• Attention and attention mechanisms like Transformers
• Short-term memory and LSTMs
• Episodic memory and experience replay
• Curiosity and exploration techniques
• lots of others, probably

Some basic reasons why this might be:

1. Because those answers are convergent. Any two good solutions to a problem are going to get the same answers because there is just one answer. 
2. Because we're stealing. Evolution did a whole lot of work for us and it's easier to steal than figure it out on our own. If we did try to do it on our own without stealing, we might find totally different answers, but that would take a whole lot more time and effort. 
3. Because we're romantics. We like looking for solutions that look to us like ourselves, that are "in our own image", as it were. We didn't gain much technical advantage from doing it this way and could've just as well done it a different way, but it was more appealing to us to do it this way. 
4. It's just marketing. These techniques aren't really all that similar to the biological mechanisms, but it gets more hype if you can draw a connection to something the brain is doing. 

All of these play some role, and in differing degrees depending on the particular technique. The extent to which any of these explanations is generally the case may have strategically important implications for AI safety. 

Insofar as 2 is more the case, then progress in neuroscience may be something of a limiting factor in further AI progress. It may also point to progress in techniques which more closely imitate brains (such as SNNs) being things to watch. 

To the extent 1 or 2 are strongly the case, searching for a paradigm very different from deep learning which might be more interpretable may be hopeless (even more than it already is, let us say). 

I'd be interested in hearing what people think about to what extent these four stories (or a different one I didn't think of) apply either in the general case or to specific techniques.