There are a few posts in Lesswrong about the Hamming Question
What is the most important problem in your field?
I'd like to ask a modified question:
What is the most important problem in any field for which you specifically have a comparative advantage in solving?
This post is a collection of strategies for finding the answer to the first half of this question. Determining your own comparative advantage to solving these problems is left as an exercise to the reader.
Readers of Lesswrong may come to a consensus on the most important existential threats to the human race. Advocates of Effective Altruism may come to a consensus on the most effective ways to save lives. However, there can be no consensus on the modified Hamming question.
If you're a Lawyer, Pharmacists, CEO, or Biologist, your vocation probably has not honed specific skills that give you a comparative advantage in solving the AI alignment problem. The best you might be able to contribute on this problem would be strategic donation of money, however, even in this activity, you possess no comparative advantage. Donation of your money to this problem may just be more effective than the donation of your mental or physical efforts.
However, if you can find an appropriate problem, perhaps one in which no one else with your particular talents and knowledge is currently working to solve, you might be able to contribute considerably more to the welfare of the human race than by the donation of your money alone.
The answer to the modified Hamming question will inevitably be specific to each person (or at least, specific to groups with a large overlap in your skills and knowledge), and since in every field, we see diminished marginal utility from each additional researcher, the answer will also inevitably be one which is not already identified by those in your field.
With the question and motivation outlined above, below are some problem search strategies.
You can systematically search all categories for inspiration.
Wordnet provides a convenient way to peruse semantic relationships. You can also access wordnet programmatically for more complex searches.
Similarly, you can peruse Wikidata both manually and programmatically.
One way to estimate the impact of a solution is to consider to total economic activity that it might optimize. For instance, would you save more labor by further optimizing farmers or by optimizing restaurant waiters? To answer this, you could look at breakdowns of GDP by sector or Value added by industry.
You can also look at average hours spent on activities. This may tell you for instance that optimizing commute is more important than optimizing house chores (though not necessarily easier).
Ideally, we would have a list of factors which caused the greatest reduction in life quality in the World population. However, I am not aware of a detail-level list which is based on any evidence. One method to achieve this might be a thought experiment: visualize yourself or others performing daily activities and consider which factors cause the most pain, discomfort, etc.
This is pretty straight-forward. Look through lists of unsolved problems.
There have been many improvements throughout history which did not solve a specific problem, but were more generalized, and offered small improvements across a wide variety of problems. While I can't offer a theory uniting all methodologies of discovery and invention, you may be inspired by reading through historical lists of scientific discoveries or major inventions.
P.S. If anyone comes up with a general theory of innovation that's more insightful than existing systems, please let me know.
The low-hanging fruit of existing fields may have already been picked. However, new fields blossom at the juncture of mature fields and often grow to answer new problems or opportunities of modern life. These new fields may inspire you to tackle more interesting problems.
You can keep a pulse on emerging technologies. Unfortunately, I'm not aware of a way to track completely new scientific fields other than stumbling upon them while skimming through a publication site such as arxiv.