There is a very big mountain of data out there, and if you are going to climb over the obstacles that obscure it, you will need all the help and support you can get. So FSN writer Lesley Meall finds out why analysts think it’s worth the effort, and considers some of the software and systems that small and medium businesses will need to help them on their way.
‘The amount of data that organisations and individuals have available to them is exploding,’ says Michael Chui, an analyst with the McKinsey Global Institute, in something of an understatement. Converging technology trends, widespread mobile device adoption, and the seismic shift from analogue to digital, have resulted in a world where we create 2.5 quintillion bytes of data each day, where 30 billion pieces of content are shared each month just on Facebook, and where 90 per cent of the data out there has come into being during just the past two years. There have never before been quite so many devices and systems capable of generating, collecting and storing quite so much digital data, and we are not afraid to use them.
But the proliferation of barcodes, cell phone signals, digital images, transactional databases, personal location records, online searches, radio-frequency identification tags, social data, video clips, website visits, and ‘exhaust data’ from physical objects and all sorts of systems connected to internet, might as well be white noise. Most individuals and businesses are so busy trying to find the information they need right now, that they don’t have the time (or the inclination) to take a few steps back, broaden their vision, and explore the very many ways in which they could potentially exploit it. But this will change, because according to Chui: ‘Big data is going to transform the basis of competition.’
Just as importantly for small and medium businesses, this transformation is not going to take place only in organisations and business sectors that have traditionally been the biggest collectors and users of data. ‘The value isn’t just going to accrue to large organisations,’ says Chui, such as governments and multinationals. ‘It doesn’t just matter in sectors such as consumer-facing industries or financial services, where there is a lot of data, it matters in every sector,’ he says. ‘As the amount of data being generated is exploding, our ability to compute, our ability to analyse this data is also exploding,’ explains Chui, and because this is making big data more accessible and more available, ‘it has extraordinarily wide-ranging scope’.
According to research from Massachusetts Institute of Technology (MIT) businesses that make decisions on the basis of data and its analysis, rather than the basis of ‘intuition’ outperform their peers. ‘Being able to measure many more things can allow businesses to understand in much more detail what their customers, and employees and processes are doing,’ says Erik Brynjolfsson, director, centre for digital business at MIT, ‘and that tremendous improvement in measurement creates opportunities to do things differently.’ As Chui adds: ‘Users of big data report higher profits and growth,’ and although this is most marked in the large organisations and sectors (such as online retail, credit cards, grocery and insurance) that are currently the main users, the benefits are going to trickle down to smaller organisations too.
In its recent report Big data: the next frontier for innovation, competition and productivity, McKinsey outlines five main ways in which big data (and the way it enhances our ability to measure things) can be used to drive business benefits, by:
- Providing transparency and access to information.
- Exposing variations in data and enabling experimentation.
- Segmenting target populations (such as customers and employees).
- Replacing or supporting human decision-making with automated algorithms.
- Innovating new business models, products and services.
But if you want to try to exploit the potential advantage this could deliver, where do you begin? In organisations where most data is already held in a digital format, this may mean leaping into the void (that contains all of your internal data and all of the external data that relates to your business) and exploring it using tools such as Google MapReduce, Hadoop, or one of the systems now using these as the basis for cloud-based ‘big data’ analytics tools – such as IBM, which recently used the ‘big data’ processing capabilities of Hadoop to provide the online data analytics tool, IBM InfoSphere BigInsights, which is now being made available to users of IBM SmartCloud Enterprise.
Given the vast amount of potential data out there, most organisations (big or small) will either need or want to be quite selective, so tools for social media monitoring (see below) may provide a starting point. Though amongst small and medium businesses, many may need to start by making better use of the data they already collect and store. ‘In some sectors, most data is not yet held in a digital format,’ says Chui, with paper-based records the norm, and even in more automated environments, way too much time is still spent just trying to find information. ‘Among some knowledge worker groups, this can take up as much as 25 per cent of their time,’ he reports, before they can even begin to make use of it.
The more business information you capture in a digital format, the more you can potentially do with it – whether this is transactional data, such as who bought what and when, or non-transaction data on people, places and things. How easily you can exploit this depends on factors ranging from the reporting capabilities of your chosen software applications, to how fragmented those systems (and the associated data) are. It would be wonderfully convenient if you could use a tool such as Google desktop to search through all of the data on your disparate and disconnected systems, whether it is structured, unstructured, held in a database, as a Word file, PDF, zip file, or whatever. But you are constrained to searching on a single machine.
One alternative to this is to centralise your data, either offline or online. You can backup all (or selected) data into the cloud, and then use an online tool to search it. This isn’t ideal: you will have to pay for storage space, and some service providers will also charge you for the privilege of searching your own data. Even so, you may still want to check out services you can use to do this, such as Greplin, CloudMagic, CloudSight Search and Primadesk. The latter for example, has recently integrated with SugarSync, to provide its users with the ability to search all of their files, photos, music, movies and more and easily transfer files from other services into their SugarSync cloud.
As FSN has noted, many organisations would rather use traditional business intelligence (BI) or dashboarding application to make better use of the data at their disposal. But the value of corporate data can be greatly enhanced when it is combined with external data from sources such as social media, as Gary Simon highlights here.
Social intelligence is an absolutely vast area (before you even start to consider any of the other very many potential sources of ‘big data’ out there), and it is increasingly likely to have either a positive or negative impact on your business. This is why vendors such as SAP Business Objects are building the ability to handle (this and other types of) unstructured data into their BI tools, and dedicated social media monitoring solutions such as Visible Intelligence are proliferating. Their potential is explored in much more detail here, and Google can quickly reveal other similar tools such as Brandwatch and the MarketMe Suite.
But whilst all of this certainly involves large amounts of data, it’s just the most visible tip of the ever-increasing iceberg of ‘big data’ out there. As this can include personal location records, Facebook pages, online searches and website visits, video clips, ‘exhaust data’ from physical objects, and the myriad other types of structured or unstructured data you can potentially connect with or access via the internet. As FSN recently considered the exponential growth of video and the challenges for business (here) this seems like a good example with which to emphasise just how vast ‘big data’ is by comparison with the large amounts of data we more used to.
A quintillion bytes of data is rather difficult to envisage, let alone the 2.5 quintillion a day we are now creating, so it may help to compare the video that is currently proliferating with the text that previously proliferated – and take on board the fact that just one single second of high definition video requires 2000 times as many bytes to store as a single page of text. ‘If you look at all the data being generated and the amount of storage being manufactured, our ability to generate data is outstripping our ability to store it,’ reports Chui, adding: ‘If businesses are to capture some of the value of ‘big data’, understanding what to keep and what to throw away is going to be one of the biggest challenges.’