The research community is very far from being efficient.

One of my own fields of research is Markov chain Monte Carlo methods, and their applications in computations for Bayesian models.  Markov chain Monte Carlo (MCMC) was invented in the early 1950s, for use in statistical physics.  It was not used by Bayesian statisticians until around 1990. There was no reason that it could not have been used before then - the methods of the 1950s could have been directly applied to many Bayesian inference problems.

In 1970, a paper generalizing the most common MCMC algorithm (the "Metropolis" method) was published in Biometrika, one of the top statistics journals.  This didn't prompt anyone to start using it for Bayesian inference.

In the early 1980s, MCMC was used by some engineers and computer scientists (eg, by Geoffrey Hinton for maximum likelihood inference for log-linear models with latent variables, also known as "Boltzmann machines").  This also didn't prompt anyone to start using it for Bayesian inference.  

After a form of MCMC starting being used by Bayeian statisticians around 1990, it took many years for the literature on MCMC methods used by physicists to actually be used by statisticians.  This despite the fact that I wrote a review paper describing just about all these methods in terms readily accessible to statisticians in 1993.

In 1992, I started using the Hamiltonian Monte Carlo method (aka, hybrid Monte Carlo, or HMC)  for Bayesian inference for neural network models.   This method was invented by physicists in 1987.  (It could have been invented in the 1950s, but just wasn't.)  I demonstrated that HMC was often hundreds or thousands of times faster than simpler methods, gave talks on this at conferences, and wrote my thesis (later  book) on Bayesian learning in which this was a major theme.  It wasn't much used by other statisticians until after I wrote another review paper in 2010, which for some reason led to it catching on.  It is now widely used in packages such as Stan.

Another of my research areas is error-correcting codes.  In 1948, Claude Shannon proved his noisy coding theorem, establishing the theoretical (but not practical) limits of error correction. In 1963, Robert Gallager invented Low Density Parity Check (LDPC) codes.  For many years after this, standard texbooks stated that the theoretical limit proved to be possible by Shannon was unlikely to ever be closely approached by codes with practical encoding and decoding algorithms.  In 1996, David MacKay and I showed that a slight variation on Gallager's LDPC codes comes very close to achieving the Shannon limit on performance.  (A few years before then, "Turbo codes" had achieved similar performance.)  These and related codes are now very widely used. 

These are examples of good ideas that took far longer to be widely used than one would expect in an efficient research community.  There are also many bad ideas that persist for far longer than they should.

I think both problems are at least partially the result of perverse incentives of researchers. 

Lots of research is very incremental - what you describe as "...there was instantly an explosion of activity as researchers raced to apply it to all the important NLP problems and be the first to publish".  Sometimes, of course, this explosion of activity is useful.  But often it is not - the idea isn't actually very good, it's just the sort of idea on which it is easy to write more and more papers, often precisely because it isn't very good.  And sometimes this explosion of activity doesn't happen when it would have been useful, because the activity required is not the sort that leads to easy papers - eg, the needed activity is to apply the idea to practical problems, but that isn't the "novel" research that leads to tenure, or the idea requires learning some new tools and that's too much trouble, or the way forward is by messy empirical work that doesn't look as impressive as proving theorems (even if the theorems are actually pointless), or extending an idea that someone else came up with doesn't seem like as good a career move as developing your own ideas (even when your ideas aren't as good).

The easy rewards from incremental research may mean that researchers don't spend much, or any, time on thinking about actual original ideas.  Getting such ideas may require reading extensively in diverse fields, and getting one's hands dirty with the low-level work that is necessary to develop real intuition about how things work, and what is important. Academic researchers can't easily find time for this, and may be forced into either doing incremental research, or becoming research managers rather than actual researchers.  

In my case, the best research environment was when I was a PhD student (with Geoffrey Hinton).  But I'm not sure things are still as good for PhD students.  The level of competition for short-term rewards may be higher than back in the 1990s.

There is a close parallel in academic research. Here, the "market" is generally efficient: most problems that are solvable are already solved. 

That's false. For most problem that are solvable nobody cares about solving them as it's not valuable to solve them.

Larry McEnerney's LEADERSHIP LAB: The Craft of Writing Effectively has a huge impact on how I think about reasearch. The research that's done depends highly on what's valued in certain research communities. 

Two look at two of my recent research questions:

After reading about how Fibrin fixates fascia because it's in fascial fluid and accumlates when the flow of fascial fluid is blocked, I wanted to know the protein composition of fascial fluid. I don't think there's any available paper that gives me the answer. That's not because it would be very hard to do the relevant research but because few people care about the protein composition of fascial fluid. This is not research I can do, but someone who had a lab could easily do it.

My other recent research idea came after reading Core Pathways of Aging. I had a feeling like "Why did I believe the bullshit they told my in MicroBiology 101 (might also have been 102) about transposons?" Then I started running an evolutionary simulation of human hunter gatherers for transposons to understand better what's going on. While I searched around for papers on other simulations on hunter gatherers and found that nobody really does this and that people instead use some differential equations from the 60's with fancy math from a time before the compute to do good simulations was available. 

While I'm still in my project and the outcome isn't yet final, it's very interesting how my model produces right now the 51% male birth rate that's the literature value and not the 50% I would have naively expected. It seems to me like the research topic is underexplored because it didn't fall into what any researcher in the field considered interesting.