"For mature and well-understood economics such as that of the United States, consensus forecasts are not notably biased or inefficient. In cases where they miss the mark, this can usually be attributed to issues of insufficient information or shocks to the economy."
Maybe it's the allure of alarmism, but aren't we mostly concerned with predicting catastrophe? This is kind of like saying you can predict the weather except for typhoons and floods.
Maybe it's the allure of alarmism, but aren't we mostly concerned with predicting catastrophe? This is kind of like saying you can predict the weather except for typhoons and floods.
I think the analogy goes the other way. A weather forecast that didn't cover catastrophes would still be useful. I like knowing if it's going to be rainy or sunny, wet or dry.
Similarly, I find it useful to know in a general sense which way short-term interest rates are going, how much inflation to expect over the next few years, and whether the job market is getting better or worse from quarter to quarter.
Yes, sometimes there are external shocks or surprising internal developments, but an imperfect prediction is still better than none.
Except that the shocks usually have a disproportionate effect on the economy. The forecasting is useful, but any strategy contingent upon the forecasting will have to take into the account that when your forecasts fail, it won't just be a little, it will be massive.
Macroeconomic indicators such as total GDP, GDP per capita, inflation, unemployment, etc. are reported through direct measurement every so often
Actually, such numbers are usually determined through sampling. They are also subject to definitions and methodologies change (see e.g. inflation).
consensus forecasts are not notably biased or inefficient.
What does this mean? Specifically, what are your definitions and criteria of being "biased" and "inefficient" in this context?
In cases where they miss the mark, this can usually be attributed to issues of insufficient information or shocks to the economy.
Sounds like No True Scotsman :-/
Yes, you're right that it's not possible to measure everything, so sampling is often used in lieu of direct measurement. I had mentioned sampling in my earlier post.
consensus forecasts are not notably biased or inefficient.
I'm using the same definitions as used in the literature. The "bias" concept is discussed in the cited papers, plus in my earlier post http://lesswrong.com/lw/k2a/the_usefulness_of_forecasts_and_the_rationality/
The "efficiency" criterion is more difficult to define, but here it means roughly "makes use of all the available information" -- sort of synonymous with rationality.
The meanings of the terms are of course up for debate, and the different papers don't quite agree on the right meaning.
In cases where they miss the mark, this can usually be attributed to issues of insufficient information or shocks to the economy.
It's certainly a flaw that they can't predict shocks, but to the extent that a few shocks explain most forecasting error, that would have different implications than if the forecasts were wrong in all sorts of small ways.
The "insufficient information" refers to the quality of existing data they have access to. In some cases, people made wrong forecasts because the data about current indicator values that they were working with had errors, or was incomplete (e.g., they didn't have information on a particular indicator value for a particular month).
The "efficiency" criterion is more difficult to define, but here it means roughly "makes use of all the available information"
How do you know? Or, more explicitly, on the basis of which evidence are you willing to make the claim that consensus macro forecasts "make use of all the available information"?
Besides, just having information is necessary but not sufficient. You also need models which will take this information as inputs and will output the forecasts. These models can easily be wrong. Is the correctness of models used included in your definition of efficiency?
It is difficult to conclusively demonstrate efficiency, but it is easy to rule out specific ways that forecasts could be inefficient. That's what the papers do.
I'm interested in forecasting, and one of the areas where plenty of forecasting has been done is macroeconomic indicators. This post looks at what's known about macroeconomic forecasting.
Macroeconomic indicators such as total GDP, GDP per capita, inflation, unemployment, etc. are reported through direct measurement every so often (on a yearly, quarterly, or monthly basis). A number of organizations publish forecasts of these values, and the forecasts can eventually be compared against the actual values. Some of these forecasts are consensus forecasts: they involve polling a number of experts on the subject and aggregating the responses (for instance, by taking an arithmetic mean or geometric mean or appropriate weighted variant of either). We can therefore try to measure the usefulness of the forecasts and the rationality of the forecasters.
Why might we want to measure this usefulness and rationality? There could be two main motivations:
My interest in the subject stems largely via (2) rather than (1): I'm trying to understand just how valuable forecasting is. However, the research I cite has motivations that involve some mix of (1) and (2).
Within (2), our interest might be in studying:
The macroeconomic forecasting discussed here generally falls in the near but not very near future category in the framework I outlined in a recent post.
Here is a list of regularly published macroeconomic consensus forecasts. The table is taken from Wikipedia (I added the table to Wikipedia).
| Organization name | Forecast name | Number of individuals surveyed | Number of countries covered | List of countries/regions covered | Frequency | How far ahead the forecasts are made for | Start date |
|---|---|---|---|---|---|---|---|
| Consensus Economics[2][3] | Consensus ForecastsTM | More than 700[2][3] | 85[2][3] | Member countries of the G-7 industralized nations, Asia Pacific, Eastern Europe, and Latin America.[2][3] | Monthly[2][3] | 24 months | October 1989[4] |
| FocusEconomics[5] | FocusEconomics Consensus Forecast[6] | Several hundred[6] | More than 70[6] | Asia, Eastern Europe, Euro Area, Latin America, Nordic economies[6] | Monthly[6] | ? | 1998[7] |
| Blue Chip Publications division of Aspen Publishers[8] | Blue Chip Economic Indicators[8] | 50+[8] | 1 | United States | Monthly[8] | ? | 1976[8] |
| Federal Reserve Bank of Philadelphia | Survey of Professional Forecasters[9][10] | a few hundred | 1 | United States | Quarterly[9] | 6 quarters, plus a few more long-range forecasts | 1968[9][10] |
| European Central Bank | ECB Survey of Professional Forecasters[11][12] | ? | ? | Europe | Quarterly[11] | Two quarters and six quarters from now, plus the current and next two years | 1999[11][12] |
| Federal Reserve Bank of Philadelphia | Livingston Survey[13] | ? | 1 | United States[13] | Bi-annually (June and December every year)[13] | Two bi-annual periods (6 months and 12 months from now), plus some forecasts for two years | 1946[13] |
Strengths and weaknesses of the different surveys
The history of research based on consensus forecast sources
There has been a gradual shift in what consensus forecasts are used in research studying forecasts:
There has also been a gradual shift in views about forecast accuracy:
Tabulated bibliography (not comprehensive, but intended to cover a reasonably representative sample)
| Paper | Forecast used | Conclusion about efficiency and bias of individual and consensus forecast |
|---|---|---|
| McNees (1978) | Own data (3 people, 4 quarterly forecasts) |
Some forecasts are biased, and forecasters are not rational |
| Figlewski and Wachtel (1981) | Livingston Survey | Inflationary expectations are more consistent with the adaptive expectations hypothesis than the rational expectations hypothesis. The paper was critiqued by Dietrich and Joines (1983), and the authors responded in Figlewski and Wachtel (1983). |
| Keane and Runkle (1990) | Survey of Professional Forecasters (called the ASA-NBER survey at the time) | Individual forecasters appear rational, although rationality is not established conclusively. Methodological problems are noted with past literature arguing for irrationality and bias in individual forecasts. |
| Swidler and Ketchler (February 1990) | Blue Chip Economic Indicators | Consensus forecasts are unbiased and efficient. Does not appear to look at individual forecasts. |
| Batchelor and Dua (November 1991) | Blue Chip Economic Indicators | Consensus forecasts are unbiased, but some individual forecasts are biased. |
| Ehrbeck and Waldmann (1996) | North-Holland Economic Forecasts | The abstract: "Professional forecasters may not simply aim to minimize expected squared forecast errors. In models with repeated forecasts the pattern of forecasts reveals valuable information about the forecasters even before the outcome is realized. Rational forecasters will compromise between minimizing errors and mimicking prediction patterns typical of able forecasters. Simple models based on this argument imply that forecasts are biased in the direction of forecasts typical of able forecasters. Our models of strategic bias are rejected empirically as forecasts are biased in directions typical of forecasters with large mean squared forecast errors. This observation is consistent with behavioral explanations of forecast bias." |
| Stark (1997) | Survey of Professional Forecasters | Attempts to replicate, for the Survey of Professional Forecasters, the results of Lamont (1995) for the Business Week survey that forecasters get more radical as they gain experience. Finds that the results do not replicate, and posits an explanation for this. |
| Laster, Bennett, and Geoum (1999) | Blue Chip Economic Indicators | Individual forecasters are biased. The paper describes a theory for how such bias might be rational given the incentives facing forecasters. The empirical data is a sanity check rather than the focus of the paper. |
| Batchelor (2001) (ungated early draft here) | Consensus ForecastsTM | Does not discuss bias in Consensus ForecastsTM per se, but notes that it is better than the IMF and OECD forecasts and that incorporating information from those forecasts does not improve upon Consensus ForecastsTM. |
| Ottaviani and Sorensen (2006) | (none, discusses general theoretical model) | Abstract: "We develop and compare two theories of professional forecasters’ strategic behavior. The first theory, reputational cheap talk, posits that forecasters endeavor to convince the market that they are well informed. The market evaluates their forecasting talent on the basis of the forecasts and the realized state. If the market expects forecasters to report their posterior expectations honestly, then forecasts are shaded toward the prior mean. With correct market expectations, equilibrium forecasts are imprecise but not shaded. The second theory posits that forecasters compete in a forecasting contest with pre-specified rules. In a winner-take-all contest, equilibrium forecasts are excessively differentiated." |
| Batchelor (2007) | Consensus ForecastsTM | Consensus forecasts are unbiased, some individual forecasts are biased. But the persistent optimism and pessimism of some forecasters seems inconsistent with existing models of rational bias. |
| Ager, Kappler, and Osterloh (2009) (ungated version) | Consensus ForecastsTM | There are consistently biased forecasts for some countries, but not for all. A lack of information efficiency is more severe for GDP forecasts than for inflation forecasts. |
The following overall conclusions seem to emerge from the literature:
Some addenda