I looked into this a little a long time ago while in cog psych grad school, and concluded that each savant whose story I looked at actually practiced their special skill a ridiculous amount, and was bad at lots of other things because they hadn't spent time on those. I haven't looked at every story but I strongly suspect that different weird obsessions and time-on-task means they're specialized but not superhuman.
Savant syndrom indentifies people with general intellectual impairment who, in one specific field, reach ordinary or even exceptional performance.
In The Psychological Unity of Humankind, Eliezer argues that
The argument behind this makes formal sense, but it's applicability strongly depends on how well we can judge what does and doesn't require complex adaptation. Reports of savants provide an interesting test of this; some of them seem like they are not merely an exceptional level of human skill, but not reachable by ordinary people. For example, in a recent post here that reminded me of this, the author claims:
Other things I remember hearing are someone seeing at a glance that there are 163 peas on a plate, or remembering every word he ever heard. If these kinds of abilities can develop as a consequence of individual genetic quirks or possibly even brain injuries, then clearly we just don't have a good intuition about what's "close" in brain design space.
Now that I've made clear what kind of ability I'm talking about, has anyone done the relevant digging?