Big Data: Don’t ignore the dark side
SOFTWARE rules the world. This isn’t just what we say out in Silicon Valley, or a belief that my sector matters most – it’s a simple statement of fact.
What’s caused this to happen? In today’s world, data is everywhere and what was once thrown away is now cheaper to store. We generate more electronic information every 20 minutes than is held by the Library of Congress, with 90 per cent of all the data in the world having been generated in the past two years alone.
Accessing reliable data is no longer just a concern for tech firms. More companies are using this information to better understand customer behaviour, predict trends and make more accurate analyses. A study by McKinsey estimated that data-driven decision-making strategies could generate up to $100bn annually across the US healthcare system alone, helping to optimise innovation, improve research, and build new tools for doctors.
But with these widening opportunities come greater risks. Some of the world’s largest firms and financial institutions have been left exposed by major deficiencies in their processing and storage of data, running up costs that have spiralled into the millions. Unless the right steps are taken, the catastrophic results of inefficient management will become ever more frequent as the world grows increasingly reliant on data.
In one of the more striking examples, trading on Nasdaq was brought to a halt for almost three hours by a communications failure last year, with its system unable to process the sheer volume of information used to analyse trades and quotes. A third fewer shares were traded in the US, on what should have been a day of routine trading. This disruption sent brokers into a panic and experts have estimated that, when a system like Nasdaq suffers an outage, losses can total $100m per minute.
Google has also been hit, recently suffering an unplanned four-minute blackout that cut global internet traffic by 40 per cent. Days later, Amazon’s US site went down for about 30 minutes, estimated to have cost nearly $2m. In both cases, the blackouts were caused by a collapse in the system designed to handle the large amounts of data being exchanged.
Outages are not always accidental, with around half planned to allow for system updates and maintenance. However, potential financial losses are so large that business leaders should be increasingly concerned about the threats posed by data blackouts, not to mention the risks involved with inefficient data management. Preventing communication failures of the kind seen by Google and Amazon should be a priority. Certainly, avoiding dirty data of the sort that brought down Nasdaq is now one of the most pressing challenges that any business faces.
The opportunities presented by reliable analyses of data cannot be underestimated. It will open up new avenues for growth and innovation across all sectors. Big data analytics will bring an end to discovery by chance, allowing for breakthroughs that wouldn’t have been possible a few years ago. But huge amounts of money stand to be lost if data is not managed properly, while companies will only be trusted if their services are 100 per cent reliable.
As the world embraces the big data revolution, we must take care not to underestimate the risks we face.
David Richards is co-founder and chief executive of WANdisco.