The trouble with statistics: The same flaws afflict economic data as political opinion polls
Who will win the US presidency?
Opinion polls have got a bad name in Britain, at least. During the 2010 general election campaign, many suggested that Gordon Brown could still continue in power in a minority government or coalition. The polling record in the 2015 campaign was even worse. Most polls showed the two main parties neck and neck, and even the exit poll on the day did not predict the overall majority achieved by David Cameron.
Conventional, survey-based polling is encountering fundamental problems, and they’re not confined to the narrow area of trying to discover political opinions. They are much more general. Household surveys are the source of a wide range of statistics which guide policy, such as official rates of poverty, inflation, and, in the United States, unemployment rates and health insurance coverage. They are also a primary source of data for a lot of economic research.
But an article on their accuracy by Bruce Meyer of the University of Chicago and his colleagues in the Fall 2015 issue of the prestigious Journal of Economic Perspectives concludes starkly that “the quality of data from household surveys is in decline”. The title of the paper says it all: “Household surveys in crisis”. Households have become less likely to answer surveys at all, those that do respond are less likely to answer all the questions, and their answers are becoming less accurate.
In the US government’s consumer expenditure survey, for example, non-response rates have risen from 15 per cent in the mid-1980s to 35 per cent now. In surveys designed to measure poverty, 25 years ago between 10 and 20 per cent of respondents failed to answer all the questions, but this “item non-response” rate is now as high as 40 per cent.
Read more: Why the economic picture tends to be rosier than initial estimates suggest
In a lengthy discussion, the authors point out that it is difficult to pinpoint exactly why these things have happened, precisely because any survey designed to answer the question would itself be unreliable! But they suspect that an important reason is because “talking with interviewers, once a rare chance to tell someone about your life, is now crowded out”. People have so many opportunities to give feedback via email and social media that they just cannot be bothered with conventional surveys.
These trends create formidable problems for polling organisations in trying to correct for any biases which might exist in any particular survey or sample. Adjustments to a sample which might have worked 20 years ago to give an accurate picture of the population as a whole no longer have the same validity. This is exactly the problem that the political pollsters in the UK have encountered.
Against this background, what are we to make of the opinion polls in the run up to the American presidential election?
On the face of it, Hillary Clinton does seem to have a small lead over Donald Trump, but her margin is narrowing. Perhaps the most scientific prediction would be to just toss a coin.