One of the most interesting media experiments I know of is the Yahoo Media experiments:
"Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market", Salganik et al 2006:
We investigated this paradox experimentally, by creating an artificial ‘‘music market’’ in which 14,341 participants downloaded previously unknown songs either with or without knowledge of previous participants’ choices. Increasing the strength of social influence increased both inequality and unpredictability of success. Success was also only partly determined by quality: The best songs rarely did poorly, and the worst rarely did well, but any other result was possible.
"Web-Based Experiments for the Study of Collective Social Dynamics in Cultural Markets", Salganik & Watts 2009:
Using a ‘‘multiple-worlds’’ experimental design, we are able to isolate the causal effect of an individual-level mechanism on collective social outcomes. We employ this design in a Web-based experiment in which 2,930 participants listened to, rated, and downloaded 48 songs by up-and-coming bands. Surprisingly, despite relatively large differences in the demographics, behavior, and preferences of participants, the experimental results at both the individual and collective levels were similar to those found in Salganik, Dodds, and Watts (2006)...A comparison between Experiments 1 and 2 reveals a different pattern. In these experiments, there was little change at the song level; the correlation between average market rank in the social influence worlds of Experiments 1 and 2 was 0.93.
This is analogous to test-retest error: if you run a media market with the same authors, and same creative works, how often do you get the same results? Forget completely any question about how much popularity correlates with 'quality' - does popularity even correlate with itself consistently? If you ran the world several times, how much would the same songs float to the top?
The most relevant rank correlation they seem to report is rho=0.93*. That may seem high, but the more datapoints there are, the higher the necessary correlation soars to give the results you want.
A rho=0.93 implies that if you had a million songs competing in a popularity contest, the #1 popular song in our world would probably be closer to only the #35,000th most popular song in a parallel world's contest as it regresses to the mean (1000000 - (500000 + (500000 * 0.93))). (As I noted the other day, even in very small samples you need extremely high correlations to guarantee double-maxes or similar properties, once you move beyond means; our intuitions don't realize just what an extreme demand we make when we assume that, say, J.K. Rowling must be a very popular successful writer in most worlds simply because she's a billionaire in this world, despite how many millions of people are writing fiction and competing with her. Realistically, she would be a minor but respected author who might or might not've finished out her HP series as sales flagged for multi-volume series; sort of like her crime novels published pseudonymously.)
Then toss in the undoubtedly <<1 correlation between popularity and any 'quality'... It is indeed no surprise that, out of the millions and millions of chefs over time, the best chefs in the world are not the most popular YouTube chefs. Another example of 'the tails comes apart' at the extremes and why order statistics is counterintuitive.
* They also report a rho=0.52 from some other experiments, which are arguably now more relevant than the 0.93 estimate. Obviously, if you use 0.52 instead, my point gets much much stronger: then, out of a million, you regress from #1 to #240,000!
Increasing the strength of social influence increased both inequality and unpredictability of success.
Cool. I was going to top-level comment a possible explanation: the more info about other people's judgements is shared (e.g. via the internet), the stronger misinformation cascades will be, because there are more judgements you observe that are ambiguously either actual data vs. just copying other people's judgements. With more misinformation cascades, you have more things that are "popular because they're popular", and "unpopular because they're unpopular... (read more)
There are a variety of YouTube channels I've come across that seem like they don't get nearly enough views.
The best example that comes to my mind right now is probably Massimo Bottura. Wikipedia does a good job of describing who he is:
So... arguably the best chef in the entire world. On YouTube. Showing you how he cooks. How he thinks.
Yet his videos only get 10-20k views. That might sound like a lot on the surface, but pay attention to view counts on YouTube and you'll find random housewives consistently getting hundreds of thousands of views for their cooking videos.
I think you have to consider that some sort of failure. Market inefficiency, false negative, non-meritocratic, whatever.
Another good example is David Heinemeier Hansson. Another wildly successful and influential person. DHH is the creator of Ruby on Rails, which is perhaps the most popular web framework out there. He also is a best selling author, and founder of an incredibly successful company, Basecamp.
I feel like I didn't do him justice. He's one of the people I think of when I think about who's had an impact on me. Maybe I should have let Wikipedia introduce him like I did for Massimo.
Anyway, DHH has a YouTube channel where he lets you inside his brain and shows you how he thinks about writing software. How many views do you think those videos get? It's in the same ballpark as Massimo: 10-60k. And also similar to Massimo, you can find people who are way less qualified consistently getting hundreds of thousands of views on similar videos.
I want to focus on my core point here. DH6, not DH5. You might be able to make points about how being brilliant doesn't necessarily make you good on camera, or about how marketing and promotion matters a lot, or about releasing videos on a consistent schedule, or about how maybe they're not producing the types of content YouTube viewers want to watch. Some of those might be fair points, but do you think that they refute my core point? My core point is that there are people like DHH, like Massimo, who should be getting way more views on their videos, given the quality of the content they're producing and what they've done to promote that content. Do you disagree with that?
Here are some examples of very high quality content coming from what I'd consider to be diamonds in the rough, rather than being from famous people like DHH and Massimo:
Maybe you're thinking to yourself: "Ok? So, what's your point?" My point is just that it's not meritocratic. That quite often, the best don't rise to the top.
Again, maybe you're still thinking to yourself: "Yeah, of course the best don't always rise to the top. Duh. Is that it?"
Yeah, pretty much. If this point is already obvious to you, then this post probably won't be very interesting. I'm just saying things you already know. But to me, it's not obvious. I still have an instinct that the best stuff would rise to the top way more often than it actually does.
I should be upfront about something: I have a pretty big personal attachment to this question. I spent over five years working on two separate startups that both failed pretty miserably. And in both cases, I felt like the market wasn't very fair to me. I felt like I had pretty good quality products, and that my level of success was not even close to proportionate to the quality I produced. Well, not just the quality of the product, I'm trying to talk about product + marketing + everything else. My logic was, roughly:
That didn't happen. And if you talk to other startup founders, I'm sure they'll tell you that I'm not alone. Paul Graham likes to say that you have to be 10x better than the competition to overcome inertia. That is so true.
I think the same thing is true in other fields too. Cooking is one that comes to mind. I know that over time, I've walked into random little restaurants that seemed to be pretty low on business, but had fantastic food! And it's not like the location or service were terrible or something. They did really well on whatever all encompassing metric of quality you want to use. But despite that, they had very little success.
I've been watching the show Chefs Table on Netflix recently. They basically do a little profile/biography on different world class chefs, and I've been noticing a similar trend where early in their careers they had serious problems with lack of success. Financial issues, depression, coming close to closing down, in some cases acutally going out of business and then re-opening. And then, for whatever reason, in comes a rush of success.
But the quality didn't really change! Maybe it changed a little bit, but not enough to account for the amount of change in success. It's not like quality went from 3/10 to 10/10, more like 9.5/10 to 10/10, perhaps. Before the rush of success, they were basically the same world class chefs producing the same incredible food. For whatever reason, the success just didn't come. They hadn't risen to the top yet. They were still diamonds in the rough.
I get the sense that lack of meritocracy is an issue in most fields, not just startups and cooking. I recall an anecdote about writing and the arts from a blog post I read today:
Again, I don't really have any insightful takeaways here. I've just been noticing myself slowly realizing that the best don't always rise to the top, and these are the examples that have been sticking out to me.