When I knew nothing of economics, unemployment wasn't mysterious. People wanted a job, and couldn't get one - well, people often want stuff they can't get. Nothing strange there, just one of those things.
Then I learnt some simple economics, and it became more mysterious. The employment market is a market, with the salary being the price. Why doesn't this market clear? Why doesn't the price (salary) simply adjust, and then everyone gets a job? It seemed profoundly mysterious that this didn't happen.
I've been gradually introducing myself to more economics (mostly indirectly) and I've encountered a lot of explanations for this perpetual market failure. Thus the mystery of unemployment is, if not resolved, at least somewhat explained. Since I would really have enjoyed reading a collection of unemployment explanations when I was initially puzzled (almost any explanation of unemployment you read in the press is worthless) I thought I'd do this for others. So here is my (entirely personal and idiosyncratic) summary of the main explanations I've encountered.
Classical explanations: not a problem
The "revealed preferences" idea holds that we can't establish what people want from what they say, but only from observing their actions and their choices. So if people are not getting jobs, maybe they don't really want them? Or maybe they don't want them, given government unemployment insurance?
I should note that whatever you think of this explanation, it argues that unemployment is not an economic problem. It views the unemployed as happily and rationally choosing not to work, which thus suits their preferences better and thus is of higher economic value (neglecting, of course, the cost of providing unemployment insurance).
Classical explanations: market interference
If a market fails to clear, the first step is to look at outside interventions that may be preventing it from doing so. Trade unions and governments are typical candidates for this. Trade unions engage in collective bargaining, requiring that employers pay their workers a certain minimum level (in terms of salary, safety, or working conditions). This raises the price of employees above the market clearing level, and thus causes a shortage of jobs.
Similarly, governments can set minimum wages, regulate who is allowed to work in certain professions, and tax employers or employees (the difference is not relevant economically) for the job. They can impose other restrictions, such as making it harder to fire employees, or creating red tape that must be dealt with (thus raising the cost of employing anyone).
Frictions and "Irrationalities"
The market equilibrium is an attractor state the market is supposed to move towards. Frictions are things that oppose this movement or slow it down. "Irrationalities" are ways in which humans differ from the perfectly informed selfish expected utility maximisers of classical economics. They may not be irrationalities in the standard sense, just deviations from the model (note that unlike the stereotype, economists are often very aware of the limitations of their models).
There are costs involved in changing most jobs. The employee may need to change their social circles, their habits, their schedules, and may need to relocate to another area. The employer may need to arrange training for a new worker, integrate them into a team, and so on. Changing a job is often a very stressful event, implying that employees, at least, feel these costs are significant. In consequence, people don't relocate for a slight improvement of conditions, and employers don't fire people to replace them with slightly cheaper workers. The market is thus less likely to adjust.
Another friction is the stickiness of nominal wages. People seem very unwilling to accept a nominal pay cut, taking this as an attack on their status. This prevents companies from rapidly adjusting to new economic conditions (in both directions: they are less likely to raise wages, if they can't claw that raise back later) with their current employees. Some jobs have important non-monetary aspects to them (eg working for government and charities because one wants to make a difference, working for a market leader, working for a company with a certain corporate culture, etc...); these aspects (that form part of the whole compensation package) cannot be rapidly adjusted up or down. Managers and workers often develop relationships with each other, reducing the likelihood that bosses would rapidly fire workers or cut wages if economic conditions demanded it.
Human behaviour and biases can provide a general explanation why agents follow behaviours that make little economic sense (issues of status, of habit, prejudices, etc...). Into this category go all the "relative status" explanations: it's not important to have a good standard of living, but a comparatively good standard of living.
Another interesting friction is changes in the economy produced by new technologies, new innovations, new companies, and people entering or leaving certain industries. The economic landscape is always changing, and all economic agents have difficulty adjusting rapidly (because of the frictions above, and because of lack of information about the new setup, see next section). This explanation is particularly interesting, because they provide a possible reasons that frictions wouldn't die out, but could continue for a long time. But if the equilibrium state is constantly changing, then frictions can maintain themselves indefinitely, perpetually slowing down any move toward labour market equilibrium.
Information and agency
Information economics is a fascinating field, full of weird insights, including several that can contribute to unemployment. At the most basic level, employees don't know about the jobs on offer (they don't know all job descriptions, and even those job descriptions only give a very partial impression of what the job entails) and employers similarly don't know about all potential employees. There is a cost to finding this information.
Information markets are unlike any other market, though. Even the smallest of information errors can create new equilibriums - stable equilibriums where the market doesn't clear. What this means is that, lack of information can explain stable rates of involuntary unemployment even in the absence of any other frictions.
The principal agent problem is another huge factor here. If the "principal" requires an "agent" to do work for them, they cannot be sure the agent will do so in the way they expect (this problem can be seen as an information issue: the principal doesn't know something - the agent's performance - and the agent cannot fully demonstrate it to them). Hiring managers may not fully trust their candidates, and they might not be fully trusted by their bosses, who are not fully trusted by the shareholders, and so on.
This means that worker X may be willing to do a job for salary S (including implicit remuneration), and firm Z may be willing to offer that, but neither can trust the other to uphold the deal. Obviously this effect is stronger when the output can't be measured formally, or when implicit aspects of the job (working conditions, office atmosphere) are important to the employee.
This is one of the reasons that companies often pay "efficiency wages", ie pay more than the market rate for jobs where output can't be measure formally. Employees who value being part of the company, have good relations with their bosses, and fear losing their good position, will work harder to demonstrate their competence, and bring more value to their employer (intuitive explanation for this: the marginal worker, paid at the market rate, only prefers having a job to quitting it by an infinitesimal amount, and will quite if anything goes slightly worse - is this the kind of employee you'd like to have on staff?). Thus companies that pay efficiency wages often profit from doing so.
But paying above the market rate, even for good reasons, still means that the market won't clear. Note that efficiency wages alone are also enough to explain involuntary unemployment.
Macroeconomics (by which I mean mainly the Keynesian/Monetarist schools) is built on the seeming trivial observation that the salaries of workers are the means by which they buy products, and that the sale of these products are the income of the firms that employ people. Therefore, if the job market doesn't "clear" it's not as simple as just waiting for salaries (the price of that market) to adjust. If salaries fall, this feeds into less cash for consumers, which generally implies less consumption, which feeds into less revenue for firms, lowering their demand for workers, etc...
There are some theorems that imply that the job market should still reach equilibrium... if there are no frictions. In the presence of frictions, there is no reason to assume that the job market would clear: persistent involuntary unemployment can be a permanent feature of the economy. In classical economics, wages cannot rise in a sector while a single person remains involuntarily unemployed (because they could undercut someone and get their job or a fraction of their job); in practice, wages do start rising long before involuntary unemployment goes to zero (a "strong job market for programmers" does not require every aspiring programmer to have a job). This raises the cost of hiring someone, preventing unemployment from disappearing.
Furthermore, the economy need not settle down to a stable equilibrium either: it can go through cycles of expansion and contraction, thus explaining the business cycle. This adds yet another explanation for unemployment: the economy being in recession.
The other claims of macro are that there are several different states that the economy could be in, for the same given inputs, and that fiscal and monetary policy (government taxing/spending and interest rate changes) can move it from one state to another, affecting the unemployment rate. The main practical differences between Keynesian and Monetarists revolve around the role of government and what to do when interest rates are at zero, but that is not relevant for this analysis.
This is where I get to share my own unvarnished and unwashed opinions as to the validity of these arguments. Take this section with many caveats and doubts. Then add a few more.
First of all, classical explanations. Most of them seem off, for the same two reasons: being unemployed is very painful for most of the population, and most people care more about their relative position (are they as well off as their peers?) than about their absolute position. This means that if some intervention gradually reduces everyone's salaries by 50%, then most people will still be willing to work. Thus there seems to be little explanatory power in classical explanations that assume that people won't work because the salary is too low. This category also includes employment taxes, since, whoever nominally pays them, the actual burden falls on those with the least flexibility, ie the employees (you can start talking about elasticity here, if you want).
But firms are not individuals, and function very differently. Thus classical explanations that explain why firms won't hire people (rather than why people won't be hired) seem much stronger. This includes minimum wages and labour market rigidities (eg firms can't easily fire people once hired). In practice, minimum wages seem to have little impact (there are a bunch of contradictory studies here), but labour market rigidities seem very important. Cross country comparisons ("Southern Europe" unemployment rates versus "Northern Europe"/Anglo Saxon ones) seem to bear this out.
I feel that labour market rigidities along with the various frictions and information issues (to a greater or lesser extent) seem a good explanation for the background rate of unemployment of a country (averaged over the business cycle).
Conversely, the Keynesian/Monetarist model seem good at explaining the business cycle. Some of the classical theory is also of value in explaining recessions caused by external shocks (eg the oil shocks of the 1970s).
Notice what is absent from these explanations: general education level, free trade (or its absence), infrastructure, technology, company X opening (or closing) a factory/research centre/office in location Y. It seems the stories that are always reported in the media may affect who gets a job (and at what wage), and may affect the overall growth rate, but they don't directly affect the unemployment rate at all (indirect effects may exist, but they have to be analysed carefully and are often hard to predict).
So. Now you (kinda) know.