Summary
In the first half of this essay, I recount two anecdotes. (The impatient reader can skip these.) First, Gerald Murnane tries, around Melbourne in the 1950s, to find a system that'll make him money betting on horse races. Then, Bill Benter, in Hong Kong in the 1990s, comes up with a system for picking horses that makes him nearly $1B.
In the second half of this essay, I discuss alpha:
- In finance, alpha is (a) excess returns earned on an investment above the market when adjusted for risk, (b) an investor's ability to beat the market, or (sometimes) (c) a strategy or resource that consistently generates excess returns. Outside finance, alpha refers simply to someone's ability to beat their competitors, the edge they have over their rivals.
- Often what appears to be alpha is just noise. There are domains, like stock exchanges, where alpha is scarce, and other domains where it is abundant. Generally, the more valuable the object, the more crowded the field, and the harder it is to find alpha.
- Six very tentative ways of finding alpha are (1) seeking decision-relevant yet generally inaccessible knowledge, (2) making deep analyses, (3) extrapolating stable-seeming, high-entropy trends or lines of reasoning, (4) finding systematic ways that others act suboptimally, (5) seeking out places where one has relatively greater ability than everyone else and/or (6) making bets that few others can afford to make.
In the long-ago 1960s, I knew a man who claimed to have been helped through a troubled period by his faith in psychoanalysis. When I myself was going through such a period, he urged me to read a certain huge book on the subject. I've forgotten the title but I recall the author's name, which was in gilt letters on the dark-green spine: Otto Fenichel. I read several chapters but I recall today only two short passages. One passage described the symptoms of a man with obsessive-compulsive disorder. He could never walk more than a few paces forward without obeying an urge to look behind him for any beetle that was lying helpless on its back and needing to be set upright. The other passage was the opening sentence of a section on gambling. According to the learned author, the gambler gambles in order to learn whether or not God has forgiven him for his masturbation.
Those are the opening sentences of the chapter, not on insect welfare, but on betting systems in Gerald Murnane's fairly Australian autobiography Something for the Pain (Murnane 2015, 89--100). Murnane goes on to write that, had the psychoanalysts "ever learned how much time and effort I've put into my search for a reliable and profitable betting system, they could only have concluded that I was either the all-time champion Onanist or, at least, the one of all the practitioners of the ancient art who felt the most guilty about it".
Most punters probably have more in common with the beetle sympathiser than with the uneasy sinner: they're driven more by compulsion than scrupulosity. And yes, Murnane's motivation was (he reports), in his later years, simply the pleasure of discovery, and in his early years, more akin to that of the FIRE crowd -- to be able to "rent a comfortable flat in Dandenong Road, Armadale; to own a small car; to join a middle-level golf club; and to put together a library of a few hundred volumes of fiction and poetry, along with a select collection of long-playing records".
During those early years -- this was in the late 1950s -- Murnane began to notice advertisements for a betting system called Form-Plan. Murnane had begun his life-long pursuit of profitable racing systems in 1952 when, at the age of 13, he began to search the Sporting Globe for patterns in the past form of winners; these efforts naturally worked perfectly for historical races, but far from perfectly on new races (this was a decade or two before the term overfitting was coined). Form-Plan claimed to select 50% winners and 75% placings; it was lavishly marketed and -- this was unusual for an advertised betting system -- its ads prominently featured its creator's name, address and photograph likeness. It wasn't the only betting system to be advertised in the Sporting Globe at that time, but it was the only one that lasted longer than a year.
Of course Murnane wondered (he reports) why someone who'd discovered the secrets of racing would advertise their discovery in the Sporting Globe. He worried not only that Form-Plan might be fraudulent, but also that, even if it worked initially, its effectiveness would be short-lived now that it had been publicised across the state of Victoria. Another reason to worry about Form-Plan was its seeming to select very few horses, meaning punters would need to bet large sums to earn a living from it (if it even worked).
Still, Form-Plan caused a stir in Murnane's racing circles. After over a year of seeing it advertised, and together with a colleague, Murnane paid the 10 pounds (the equivalent of about 6-7 good-quality hardcover books, he offers) for a copy. The system provided a set of rules for selecting horses, for example choosing only so-called top weights (handicappers would assign weights to horses in accordance with their abilities so as to even out the field). It focused exclusively on two-year-old horses; this was one reason why it selected so few horses.
The first Form-Plan bet Murnane placed was 5 pounds (3-4 hardcover books) on a filly named Snowflower ("Pale blue, tartan sash"), who ended up losing narrowly at Caulfield Racecourse. Ten days later, he bet another 5 pounds on the filly that had beaten Snowflower, Faithful City ("Green, gold Maltese cross, striped sleeves and cap"), at four-to-one odds:
The favourite was Ritmar (White, purple stripes), a filly from Sydney ridden by Neville Sellwood. I wish I could remember who rode Faithful City. Ritmar could gallop only at top speed, and the rider of Faithful City, having seen this, held his mount together for a last run at the favourite. Ritmar led by two lengths around the turn. The straight at Moonee Valley is short but uphill. Ritmar began to tire and Faithful City to gain ground. [My colleague] and I were in the old South Hill reserve, almost head-on to the finish. We had no idea which filly had won. The judge studied the print of the photo finish for three or four minutes and then declared Faithful City the winner.
I have always maintained that a writer achieves nothing by trying to describe feelings; that feelings can only be suggested. I will therefore report only that my feelings, after Faithful City's narrow win, were mixed. I had in my possession the key to lifelong wealth, but so too had all those numerous unknown buyers of Mr Maclean's method. How long would it be before half the population of Melbourne heard about Form-Plan and flocked to the races to get their share of the easy money?
The next bet again was on Faithful City, who won at about two-to-one odds. Murnane bet 8 pounds (5-6 hardcover books) and profited the equivalent of ~4x his weekly allowance as student teacher. But soon thereafter the system began to make a loss, and after about a year of Form-Plan betting Murnane gave up on it:
I don't recall when the ads for the system disappeared from the Sporting Globe, but I recall my receiving a newsletter from Mr Maclean at some time in 1959. He had decided to change some of the rules of his method. We were no longer to back horses starting from wide barriers or on days when the track was heavy. Needless to say, these rules would have prevented me from backing several losers during the months past. The last message that I received from Mr Maclean was another newsletter. He had devised a completely new method of systematic betting. Results for the past few years were outstanding. Persons who had previously purchased Form-Plan could buy the new system at a discounted price. I decided that the man was incorrigible and a rogue, but that was a few years before I read Otto Fenichel's book. Now I incline to the belief that poor Mr Maclean was truly desperate to learn how he stood with God.
Murnane's Dream Was Benter's Reality
To the best of my knowledge, Gerald Murnane never found his winning system. But he was no Don Quixote. Others succeeded where he did not.
A decade and a half after Faithful City won the race at Moonee Valley, a horse racing columnist for the Washington Post named Andrew Beyer put into practice his own betting system:
In the early 1970s -- armed with sharpened pencils, a mini calculator, large sheets of paper, tall stacks of the Daily Racing Form and a bottle of Jack Daniel's -- Beyer performed his painstaking work. By carefully and thoroughly accounting for the biases of tracks and adjusting horses' clocked times accordingly, he collapsed the complications down into a single number: a speed figure. Speed figures allowed for reliable comparisons between California horses and Florida horses, between five furlongs and six, between a lowly claiming race and a big-money graded stakes. "Speed figures clarified mysteries, subtleties and apparent contradictions of the sport that I had always thought were beyond human understanding", Beyer wrote in 1975.
By then, Beyer was already a successful gambler, in large part because his private calculations gave him a healthy edge. His books' publication educated the market and eroded his advantage. Beyer accepted this trade-off because he wanted to be a published author.
Later on, in Hong Kong in the 1990s, Bill Benter, formerly an advantage player in Las Vegas, earned nearly $1B betting on horses. (Hong Kong had a vibrant, high-liquidity betting market for horse racing at the time.) The way he did it -- inspired by Brecher (1980) and Bolton and Chapman (1986) -- was by programming and continuously refining a computer model which at its most complex regressed >100 variables and comprised thousands of lines of code (Benter 2008). Benter's crowning glory was when he, on 3 November 2001, won the Triple Trio (worth ~$13M) by placing a bet correctly calling the top three horses (in any order) in three different races; the story goes (and here we should exercise scepticism) that he chose not to collect the winnings, instead letting the Hong Kong Jockey Club do with them what it always does with unclaimed winnings: give them to charity.
Like Murnane fearing Form-Plan's spread throughout Victoria, Benter too speculated that his edge would disappear as others adopt his methods (Benter 2008):
Even if the racing is honest, if the general public skill level is high, or if some well financed minority is skillful, then the relative advantage obtainable will be less. [...] [There is] no guarantee that the future will be as profitable as past simulations might indicate. The public may become more skillful, or the dishonesty of the races may increase, or another computer handicapper may start playing at the same time.
These days, there are entire professional gambling operations devoted to betting on horses: "Collectively known as computer-assisted wagerers, or CAWs, they are largely anonymous. These sophisticated bettors use horseracing data to the extreme, employing algorithms, research staff and sweetheart deals to enrich themselves. In recent years, they have increased their capital and their wagering to unprecedented heights, accounting for as much as one-third of the money bet nationwide." In 2021, a single CAW syndicate represented ~30% of bets placed on Californian tracks, but of course they'd make no money without a sizeable proportion of amateur punters.
Alpha, Seen Up Close
In finance, alpha refers to (a) excess returns earned on an investment above the market as a whole when adjusted for risk, (b) an investor's ability to beat the market and (sometimes) (c) a strategy or resource that repeatedly and consistently generates excess returns. The boundaries between these seem fuzzy; I've heard people use "alpha" to refer to information, models, software, talents and securities among other things.
Outside finance, alpha refers by metaphor to someone's ability to beat their competitors: in other words, an edge, a particularly successful tactic, being ahead of the curve, as the expression goes. It's what most sports dynasties had, most groundbreaking artists and most farseeing scientists. It's what Gerald Murnane sought. It's what Bill Benter found. It's not success, but it can be converted to it. It's the thing that Taleb (2007) describes:
Think about the "secret recipe" to making a killing in the restaurant business. If it were known and obvious, then someone next door would have already come up with the idea and it would have become generic. The next killing in the restaurant industry needs to be an idea that is not easily conceived of by the current population of restaurateurs. It has to be at some distance from expectations. The more unexpected the success of such a venture, the smaller the number of competitors, and the more successful the entrepreneur who implements the idea. The same applies to the shoe and the book businesses -- or any kind of entrepreneurship. The same applies to scientific theories -- nobody has interest in listening to trivialities. The payoff of a human venture is, in general, inversely proportional to what it is expected to be.
Aiming at the unexpectedly good usually involves some amount of risk. There's a widespread assumption, in finance at least, that in order to get expected returns greater than the broader market you must tolerate higher risk. On that view, the only way to get higher returns is to increase risk, and the only way to reduce risk is to accept lower returns. Alpha exists only where that assumption is false. Falkenstein (2009):
The key is that risk taking and risk tolerance is not like physical courage, the ability to withstand physical pain, but rather like intellectual courage, the ability to withstand ridicule. Physical courage is something most people agree on, and invites empathy, as when we cringe when watching someone crash his bike. In contrast, intellectual courage as it is sometimes practiced is usually seen as foolishness because risk is a deviation from what everyone else -- the wisdom of crowds, the vast majority of smart people -- is doing, and invites ridicule that is pointed and very hurtful.
Risk means exposure to possible negative consequences. When I "risk doing X", I expose myself to possible negative consequences of doing X. When I "put Y at risk", I expose Y to such consequences. As lsusr writes, "Alpha is a bet against your society's beliefs." And finding alpha does at least sometimes imply taking risks, though taking risks doesn't necessarily mean you find alpha: sometimes you pay a premium for risk, as illustrated by the gambler and the casino.
Richard Hamming, mathematician and computer scientist, pointed at courage as one of the virtues a researcher needs in order to attain greatness. But he seemed to think it was important less because pursuing hard problems was risky, and more because it required a certain amount of self-confidence, or failing that the strength of will to push through anyway. "Once you get your courage up and believe that you can do important problems, then you can. If you think you can't, almost surely you are not going to."
Alpha, Seen from a Distance
Economists say we can make inferences about people's beliefs by observing their behaviour. If so, plenty of people believe there's alpha in financial markets. But the efficient-market hypothesis, or at least the strong version of it, says markets perfectly and immediately reflect all available information, meaning investors can't consistently beat the market using public data. What gives?
Markets are made up of people, and efficiency is not binary. When asset prices change as new information comes to light, that is because people make trades based on that information. Some of those people will make the right trades, and profit. The question is whether they can do that consistently. Austin Vernon writes:
One of the strange corollaries to the Efficient Market Hypothesis is that you can't tell if a market is efficient. But there is reason to believe America's public markets are very efficient. Over long periods, close to zero active managers can beat the market.
Renaissance Technologies is one of the most famous hedge funds. They employ some of the best mathematicians and scientists in the world to do quantitative modeling and predict stock prices. They are known for their Medallion Fund, which earns gobsmacking returns. But the fund has no outside investors, and it returns capital instead of compounding it. The fund's strategy focuses on short-term and events-based trading, not long-term analysis.
Renaissance's public funds have returns that look like many other hedge funds. Unremarkable even with all the mad scientists.
Hundreds of the best math minds in the world can only find a few billion dollars a year of inefficiency in market data. All evidence points to opportunities that are small and fleeting.
Renaissance likely is able to consistently find alpha, but a lot of what looks like alpha is probably only happenstance or survivorship bias. What would you expect to see if, each year, every hedge fund had an equal chance of making a profit and a loss? If you simulate 1K hedge funds, each of which has an annual profit sampled from a normal distribution around 0% (90% CI: -40% to +40%), you get the results shown below. In particular, the plot shows the returns for these funds over time (compared with a risk-free return of +2% to +4%). Note the log scale. After 20 years, the probability that a fund outperforms the risk-free return is 16%; after 40, 7%; and after 60 years, 5%. Despite a pure coin-flipping strategy, 50 of the 1K funds will still beat the risk-free return six decades out.
How do you distinguish alpha from noise? When a young Jesse Livermore -- one of the most famous stock traders ever to live -- brought home $1K, his mother disapproved of his venture, dismissing it as "gambling". Livermore went on to make millions, declare bankruptcy, make millions again, declare bankruptcy a second time, then to net $100M with massive short positions during the 1929 Wall Street Crash, and in 1934 to declare bankruptcy a third time. Livermore had a strategy for finding alpha -- he'd focus on historical price and volume data to look for trends and reversals, an early form of what's now called technical analysis -- and still did not consistently win out.
Livermore (1940) wrote: "There are times when money can be made investing and speculating in stocks, but money cannot consistently be made trading every day or every week during the year. Only the foolhardy will try it. It just is not in the cards and cannot be done."
The stock market is still more efficient today, and alpha is rarer and more fleeting.
In other domains, alpha is everywhere. Take an ancient forest. The forest holds plants, fruit, berries, mushrooms, animals. It's also home to predators -- wolves, bears, lynxes, humans. Most predators struggle to survive. Sometimes an individual or species finds a winning advantage, but more often they're stuck in a population cycle (fun case studies for students of Lotka-Volterra equations). But human hunters can stroll in and, in a merry afternoon, find and kill enough game to feed themselves for days. Human culture is a constant, inexhaustible source of alpha when humans compete with the other animals.
There are domains -- not always the domains we're interested in -- where alpha is abundant, and domains -- often the more valuable ones -- where it is scarce. As a general rule, the more valuable the object, the harder it is to find alpha. There are exceptions: for example, goods in altruistic markets are often more valuable for society than you'd expect from the number of people who seek them, simply because people are a mite selfish and tend to seek goods that they value for themselves over goods that are valuable for society as a whole. These exceptions are important, as we shall see.
Tentative, Generalised Ways of Finding Alpha
The history of humanity is crowded with seekers of alpha. Financial schemes have been turned into frauds and scams but also engines of growth. Priceless secrets have passed from lip to lip in back alleys and courts. People have bartered, cajoled one another and, when inspired or desperate, invoked the favour of gods. They have sought to be wiser and better than their peers, or to find fruitful niches their peers haven't yet discovered; they've sought to control markets, trade routes and natural resources; they've specialised, advertised, organised, tyrannised; they've laboured, they've contemplated and above all they have innovated.
With all that in mind, what can we say about finding alpha? The first thing we can say is that one should view any guide to finding alpha with a measure of scepticism, at least if one intends to compete in a crowded field.
But keeping Hamming's advice in mind, at a high level, the sources of alpha are information and circumstances, and talent maybe, that allow one to consistently take actions that successfully achieve certain ends. (This notion of alpha applies not only to zero-sum competitions, but to altruistic markets, too.) Bill Benter's end was to pick winning horses (or maybe to make money, or more likely both), and the system he developed (the intricate regression model) and the context in which he deployed it (Hong Kong in the 1990s) allowed him to do that well and consistently.
I can think of a bunch of things that may plausibly help me find alpha (though to be clear I'm not confident in any of these, and even if some correlate with or even cause us to find alpha they may not constitute effective practical advice):
- First, I can think of ways of finding alpha by seeking knowledge about the situation you're in and the problem you're solving:
- You can seek decision-relevant yet generally inaccessible knowledge. For example, Nathan Rothschild, banker, businessman and financier, supposedly profited from the English victory at Waterloo by getting early warnings from his couriers and using it to speculate on the Stock Exchange.
- You can make deep analyses and carry out empirical research that allow you to make better predictions about the future. For example, Bill Benter analysed horse racing and built a predictive empirical model for forecasting winners.
- You can follow stable-seeming trends or lines of reasoning to their logical endpoints; if these are high-entropy, ambiguous or otherwise hard to interpret, others may not anticipate them. For example, Gordon Moore looked at a few years' history of chip transistor density and boldly drew a line reaching a decade out. Others have seen how the human circle of moral compassion has tended outwards from those who are closest to us, and as a result have projected it to distant strangers, other animals and those not yet born. (Thomas Robert Malthus, in founding the study of population dynamics, did this too, though he underestimated human ingenuity and failed to anticipate the demographic transistion.)
- You can try to find systematic ways that others act suboptimally (for example, due to quirks of human psychology, biases or flawed social structures). For example, the Brooklyn Dodgers found alpha in signing Jackie Robinson and other black players into a league where no one else dared or wanted to break the colour barrier; in the following decade, Robinson and others led the Dodgers to 6 pennants and 1 World Series victory. Similarly, John Maynard Keynes was a successful investor partly because he invested substantially in assets outside the United Kingdom, avoiding home bias.
- Second, I can think of ways of finding alpha by changing environment to one that's conducive to taking ends-achieving actions:
- You can seek out places where you have relatively greater ability than everyone else. For example, someone who's a good-but-not-great poker player may do well by picking the right table (a table with inferior players).
- You can make bets (with money, time, esteem or any other resource) that few others can afford to make. For example, companies sometimes seek to influence policy-makers to create costly regulatory hurdles, shutting out less well-resourced competitors. (NB: This may not always be ethical.)
- Third, I suppose it's possible to find alpha in getting better at a thing, but this is difficult, time-consuming and obvious.
A promising strategy or piece of information, or even a glimpse of one, arrives fused with the fear that someone else will discover it too. That fear was probably one part of Gerald Murnane's mixed feelings at Moonee Valley. But it's a useful fear, because it inspires you to act.
Thanks to Oliver Guest for giving feedback on a draft.
References
Benter, William. 2008. “Computer Based Horse Race Handicapping and Wagering Systems: A Report.” In Efficiency of Racetrack Betting Markets, 183--98. World Scientific.
Bolton, Ruth N, and Randall G Chapman. 1986. “Searching for Positive Returns at the Track: A Multinomial Logit Model for Handicapping Horse Races.” Management Science 32 (8): 1040--60.
Brecher, Steven L. 1980. Beating the Races with a Computer. Software Supply.
Falkenstein, Eric. 2009. Finding Alpha: The Search for Alpha When Risk and Return Break down. Vol. 511. John Wiley & Sons.
Livermore, Jesse L. 1940. How to Trade in Stocks: The Livermore Formula for Combining Time Element and Price. Laurus-Lexecon Kft.
Murnane, Gerald. 2015. Something for the Pain: A Memoir of the Turf. Text Publishing.
Taleb, Nassim Nicholas. 2007. The Black Swan: The Impact of the Highly Improbable. Vol. 2. Random House.
2
1
1