Part of what people may call a superstar effect in science is called the "Matthew effect" by sociologists.
“I'd argue that, just by knowing that a play is by Shakespeare, we assume that it's deep and meaningful, and read in deeper interpretations and symbolism than we would otherwise.”
I largely agree, but there are some Shakespeare plays that are just not that deep or meaningful—Romeo and Juliet, for example. I think Shakespeare knew he was writing about some silly teenagers, but contemporary readers invest their relationship with more profundity than the play warrants. And, just to be a bit more nuanced, I read his plays for their superior wordplay and less for the deep interpretations that literary critics lay on top of them.
In his post on science slowing down, Scott said:
I'd argue that there are way more than a hundred Shakespeares around today, and there were several in Shakespeare's time. By Shakespeares, I mean authors who could have produced works of comparable quality to Shakespeare, by some reasonable measure of quality.
This seems surprising; there do not seem to be hundred living authors that are almost universally agreed to be must-reads in the same way that Shakespeare was.
But this lack hints at a resolution of the paradox: we just don't have space for a hundred authors with the same fervour as we make space for Shakespeare. Neither as individuals nor as cultures can we fit these in. Shakespeare was a literary superstar. And superstars are rare, due to network effects and the power law of fame.
So my thesis would be that:
Part of this is the halo effect: superstars just get judged as better than anyone else.
Also, just by being famous, the interpretation of their work is altered. Bits of Shakespeare have permeated popular culture, and many articles and theories have been created about him. When we watch a Shakespeare play, we don't just see the words; we see the layers of cultural meaning and interpretation that have accumulated on it.
I'd argue that, just by knowing that a play is by Shakespeare, we assume that it's deep and meaningful, and read in deeper interpretations and symbolism than we would otherwise. If we rediscovered two old plays, and they were word for word identical, but one was believed to be by Shakespeare and the other by some forgotten minor playwright, I'd expect that the first one would be a better play, just by what the audience would bring to it.
Apart from those effects, superstars have the unique ability to focus more on their own vision. They have great self-confidence, and they can afford to trust that their audiences will have the patience to follow them where they want to go - rather than expecting immediate literary gratification. This would tend to result in works that are better than the average work of someone of equivalent skill, and more likely to be "deep", "insightful", or "timeless". This effect might be even more obvious with bloggers than with authors.
So, though the number of superstars is severely limited, the number of potential superstars of equivalent skill can and most likely does increase with population.
I'd argue that there's also a superstar effect in science. But here it combines with Scott's explanation 3: low hanging fruit. Newton did not come up with general relativity; Einstein didn't find quantum field theory; Tesla didn't invent the laser. You can't develop an idea until certain pre-requisites are met.
And, unlike those solitary geniuses, most of science and technology is collaborative. Superstars get to be part of the best teams, interact with the best other scientists, and are more free to focus on the biggest, sexiest problems. I expect that there are many non-superstars who would have developed a certain part of theory, if a superstar hadn't got there first. It seems plausible to me that a single scientific superstar could have done the equivalent of derailing a hundred promising careers, just by getting to the key insight faster - without necessarily being much smarter (if at all) than the ones they preempted.
Then, as discoveries pour in from superstars, and the far less productive non-superstars, the domain of science changes, and new avenues of discovery open up. And these new avenues are going to be claimed by the next generation of superstars, who will get there first. I expect that if we removed every single superstar of science in the last two hundred years, that we'd get roughly comparable scientific progress, with alternate superstars rising to the fore.