Big industry responses are needed to big data explosion
BIG data has been the big tech issue of the last year and this trend is set to continue as volumes of transactional data continue to grow.
So what do people mean by big data? The definition is largely subjective and refers to datasets of a size that is beyond the ability of typical database tools to capture, archive, and manage. Sources can include human and machine generated unstructured data such as logfiles, semi-structured such as documents and structured relational data. Rather than being defined in terms of “greater than x terabytes”, as storage and the capacity of database tools increases, so will the definition of what constitutes big data.
Information is now growing at a faster rate than our electronic systems improve (Moore’s Law states that that the number of transistors that can be placed inexpensively on an integrated circuit can be expected to double approximately every two years). According to IDC estimates, the average enterprise will need to manage 50 times more information by 2020 while increasing IT staff by only 1.5 per cent – a challenge that businesses will have to respond to in an efficient manner.
BIG DATA AND FINANCE
In the financial sector, the stipulations of ever mounting layers of regulation – such as the Markets in Financial Instruments Directive review (Mifid II) – as well as increased volumes of data are driving the requirements for exponentially increasing data storage and analysis. “On some days in 2011, data volumes were almost double that of the previous day,” says Simon Garland, chief strategist at Kx Solutions. “In addition, events such as the flash crash have brought wild, short-term increases in data volumes to the forefront of concerns, while problems in the Eurozone have resulted in market instability and therefore increased volatility.”
ABOVE AND BEYOND
Beyond simply achieving regulatory compliance for transactional reporting and best execution, companies should also be focusing on making big data analytics work for their business. Through data intelligence mining and analytics, behavioural modeling and inference-based decisions, big data can be used to provide a competitive advantage.
In the finance sector, financial risk modeling, near real-time financial analysis and cyber security and threat analysis can all be given the edge with the aid of big data sets and powerful analytics. “With big data, none of it can be discarded,” says Garland. “All of it must be analysed, tick by tick. The institution that does that will spot market opportunities which someone just working with aggregated data could miss.”