How to read the news before it happens
FOUNDING PARTNER, EGO VENTURES
AT THE turn of the last century, the New York bootlegger Irving Wexler seemingly achieved the impossible. He predicted the winner of every race he wagered on, reaping handsome rewards for his efforts and confounding peers and bookies in the process.
In an industry where everyone searches for an edge, Wexler appeared to have harnessed the ultimate one – omniscience. However, his God-like powers owed little to crystal balls or mathematical models, instead exploiting a simple quirk to his benefit.
Due to the illegal nature of horserace betting, bookmakers were forced to work away from the gaze of authorities – outside of racecourses, relying upon associates at the course to feed them results. This created a delay between the time a race finished and when the bookies were aware of the result – the more inefficient the messaging system, the greater the delay and opportunity for those with more efficient means of communication.
Wexler exploited this inefficiency, betting on races after the result was known but before bookies had been passed the information from the courses – giving the illusion of precognition. In fact, he was simply intercepting the bookmakers’ telegraph service. When they got wise to the scam, the bookmakers sent false information down the wire, bringing Wexler’s advantage, and streak, to an abrupt end.
Further refinements in telegraph technology soon closed the bookies’ information gap. Today, information is instant and ubiquitous, legislation prevents financial gain from inside knowledge, and any legal advantages in predicting the future with more timely and accurate information appear largely spent – forecasters now rely upon more esoteric and less accurate means of divining the future.
ENTER BIG DATA
And yet, a new paradigm is emerging, breaking the old rules about the limited value of publicly available information. Enter stage left: Big Data – the tech buzzword of the year, and apparently a new, legal way to predict the future.
Big Data differs from previously available “small” data sets in three distinct ways: greater volume, complexity and speed. A McKinsey report from May this year evangelises about the numerous applications and potential, claiming that a retailer using Big Data to the full could increase its operating margin by more than 60 percent.
But how does Big Data help predict the future? By exploiting three basic concepts: firstly, there is always someone out there with more information about the likelihood of a particular event than you. Secondly, sentiment often precedes outcome. And thirdly, the emergence of social technology has created an environment where shared thoughts and opinions have global reach and permanence.
To consider the application of Big Data in predicting the future, let’s take a question that is close to my heart: Which players will Tottenham Hotspur sign before the close of the transfer window? Before the advent of Big Data, one was forced to rely upon the scattergun approach of tabloid back pages, or some insider knowledge from the manager’s next-door neighbour’s second cousin. But with the emergence of Big Data we can now use the wisdom of crowds to more accurately predict an answer.
For example, one basic measure is the frequency with which a particular player’s name is linked to the club – the higher the frequency of mention, the more likely the deal.
The graph, below right, is taken from Google Insights and shows the number of times Adebayor’s name has been searched for with the word Tottenham in 2011. It should come as no surprise that in mid-August 2011, well after the upward trend had begun, Spurs’ manager Harry Redknapp confirmed the club was trying to buy the player.
SENTIMENTAL INSIGHTS
Another way in which Big Data is transforming forecasting methods is through sentiment analysis. The basis of this approach has been around for some time – economists have used leading indicators to make predictions for decades. However, with the emergence of Big Data, the complexity of data being used has radically shifted; whereas economists use structured, numerical data such as production or export numbers to predict macro-economic growth, Big Data has facilitated the use of subjective and unstructured data to see into the future.
Big Data is also fast – often available in near real-time. One application of this is to predict the success of a product. By mining social media sites such as Twitter and Facebook, marketers are able to build up a view of the collective sentiment on particular products just ahead of release. HP Labs published an academic paper last year, Predicting the Future with Social Media, that demonstrated an uncanny ability to foresee box office sales for blockbuster movies. The researchers found that the frequency of positive mentions of an upcoming movie on Twitter is positively correlated with its future performance, providing greater predictive accuracy than the Hollywood Stock Exchange, the industry gold standard.
FINANCIAL INNOVATORS
It should come as no surprise that businesses have begun to sit-up and take notice of Big Data, with industries where success is directly linked to predicting the future, such as investment management, banking and government leading the way. An article in the Bank of England’s Quarterly Review this June said “Internet search data have the potential to be useful for economic policy making,” finding that in predicting changes in house prices, search data outperformed some existing indicators. We already have the first Twitter hedge fund, Derwent Capital, whose strategy is based upon another 2010 academic paper. This paper, by Johan Bollen, Huina Mao and Xiao-Jun Zeng, showed the frequency with which companies and stocks are mentioned on Twitter is positively correlated to price direction, giving the ability to predict the Dow’s closing price with 87.6 per cent accuracy. According to reports, in its first month of trading this July Derwent Capital returned 1.85 per cent, versus an average hedge fund return of 0.76 per cent for the same month.
A cottage industry has emerged, with start-ups such as Recorded Future and Palantir Finance claiming to have developed complex lexical and mathematical algorithms to find trends and make useful predictions from Big Data. Palantir is backed by Peter Thiel, founder of PayPal and one of Silicon Valley’s top stars. Recorded Future is based in Boston and its temporal analytics engine claims to let you effectively search the future.
The days of profiting by predicting events after they have occurred are behind us – and long may that continue. However, the legacy of how this problem was solved – the implementation of technology with instant and ubiquitous information transfer – has given rise to a new predictive capability. Big Data’s potential is vast, and sceptics should be wary of betting against it.
Brijesh Malkan is a founding partner at EGO Ventures. Its mission is to disrupt and democratise the investment management industry.