The main function of managers of all stripes (according to me) is to solve coordination problems. However, there are also “autonomous” systems for solving some specific kinds of coordination problems - markets, in particular, are a major example. But these are limited to problems which fit their interface well - i.e. a high degree of standardization, in the transactions if not the products themselves. A requirement for large amounts of context, or difficulty recognizing success even in hindsight creates problems for markets.
I could imagine systems which solve (some) more context-heavy or alignment-limited coordination problems, with the scalability and decentralized “autonomy” of markets. Management/governance via prediction markets is one class of ideas in this vein, but it seems like modern information technology should open up many other possibilities. The way I picture it, it's not so much "build an AI to run a company" as "organize the company so its constituent people together implement the same algorithm the AI would". Just like competitive markets perform a variant of backprop and gradient descent, other AI algorithms should have equivalent structures in human organization.
In recent years, availability of information/data has grown much faster than our ability to index it. Matching particular people with the highest-value information for them - i.e. indexing the information - is largely a social technology problem, though one which benefits a lot from computer science.
Suppose I'm researching mutations of mitochondrial DNA. Perhaps the topic has been studied by a particular group separate from the main field, with different aims and entirely different jargon. A text search for the normal jargon won't turn up their work, no matter how helpful and relevant it might be. In order to find something like that, we need some kind of jargon-independent indexing. For instance, if we had a detailed world-model, then we could link the information directly to structures in the world-model (i.e. modelled mitochondria). If we had extensive augmented reality, we use object recognition to recognize mitochondria visually, and then index information to that - so that even research using a different word for "mitochondria" would pop up when looking at a picture of a mitochondrion.
More generally, it would be useful to be able to index information based on the real-world patterns or mathematical structures they involve. This seems like something which could, in principle, be implemented purely socially, if we knew a good way to do it. (I'm sure people have tried things like this before, but usually projects like these try to use data structures which are convenient for programming, rather than data structures which represent the world well.)
More Efficient Notation
People joke that Einstein's greatest invention was tensor index notation. Good mathematical notation makes a surprisingly large difference, as anyone who's used both assembly language and python can attest.
It seems to me that the main trend in mathematical notation from ~1850 to ~1950 was to introduce really bad notation for things which previously had no formal notation at all. (This is part of the more general trend of introducing not-very-good foundations for things which previously had no foundations. Set theory and Turing machines and the like were better than nothing, but they are more like assembly than Lisp.) I expect in the future, progress will come more from improving notation, as a side effect of the general project of refactoring mathematical foundations.
Continuing improvement in programming is, of course, a similar story. In that department, I think progress will come from better understanding the structure of the world around us, and incorporating those insights into programming languages (abstraction is one potential example).
Model-Based Financial Markets
Similar to Abram's suggestions for model-based prediction markets, as automated trading gains share of the financial markets, there's potential for financial systems which take advantage of the legibility of automated models.
In particular: modern markets direct capital to existing projects. Traders probably have some views about what projects would be really good investments, but the market does not have an easy mechanism to share those views with business founders/managers during the planning phase. Instead, it's guess and check: a founder/manager tries a project, and then finds out how much the market likes that project. Part of the problem here is that project-space is very high dimensional, so it's not easy to compress all the information from the traders into a low-dimensional price signal.
But with transparent automated traders, we could in principle use a backprop-like mechanism to identify projects which the traders think are promising, without the slow guess-and-check process.