12/04/2015
Almost five years ago, Google’s Eric Schmidt announced that we had reached the point where more data was being created every two days, than in all of human history, up until 2003.
Since then, companies of all shapes and sizes have been getting to grips with new ways of handling the incredible volume of information that is becoming available to us every day.
For example, users of Facebook upload around one billion pieces of content to the social network site every day. In industry, machinery and vehicles are fitted with sensors and trackers that record their every move, and whenever we call a call centre, an audio recording of our conversation is made, and stored in a huge digital database.
In addition, whenever we go online (as most of us increasingly do for a number of reasons – shopping, socialising, making travel arrangements) we leave behind a digital footprint – a record of websites we visit, products viewed, even how long we leave the mouse cursor over certain areas of the screen, in some circumstances.
We collectively refer to both these huge datasets we are building, and the practice of interpreting, analysing and acting upon insights gleaned from this information, as “big data” – and it is changing the world we live in.
In healthcare, vaccine manufacturers are pooling data from clinical trials and using it to come up with better and cheaper medicine. Millions of financial transactions are logged and analysed every second, to crack down on fraud. And at CERN, the Large Hadron Collider is generating 300 gigabytes of data per second as it attempts to find the fundamental building blocks of our universe.
The online behemoths which have come to dominate business in the internet age – Google, Facebook, Amazon etc. – all base their business models on big data. It is by collecting and analysing huge amounts of information from us that they are able to determine precisely what we want – as well as selling advertising services which are capable of precisely targeting their client’s preferred demographics.
Why size is no issue
But Big Data is not just for the big boys, it matters to every company – no matter how small or traditional. To cater for this huge demand many companies have sprung up to offer services to other businesses, enabling them to launch big data initiatives of their own. In other words, to leverage the information they have available to improve effectiveness and efficiencies in their business, and ultimately increase profits.
A lot of the software and analytics tools needed to carry out big data analysis are built on open source principles – meaning they are essentially in the public domain and free for anyone to use for any purpose.
For example, Hadoop is a framework – a collection of software tools and applications – designed to allow organisations of any size to store and analyse huge amounts of information. It is designed to run on cheap, commonly-available hardware rather than expensive, specialist equipment that would previously have been necessary.
Companies including Amazon, Google, IBM, HP, as well as newer names such as Hortonworks, MapR and Cloudera offer big data solutions and support, as well as tailored versions of the free products, designed to work out-of-the-box and with less complex setup requirements. This also enables companies to minimise infrastructure investments or avoided completely by using cloud-based storage and analysis tools that can be rented when needed.
Big data initiatives, within businesses both large and small, are helping us to pinpoint precisely which efforts are causing which results – allowing processes from research and design, to manufacturing, marketing and after-sales support to be fine-tuned for optimal results.
Big data is big business. Worldwide, companies are expected to spend £80 billion on products and services to enable them to carry out big data analysis this year. But when it has been estimated that failing to effectively capitalise on their data costs business in the US alone close to £400 billion each year, it’s no surprise that respected market researchers Gartner recently found that 73% of businesses have already invested in a big data plan, or are planning to do so in the next few years.
The relevancy of The Internet of Things
Big Data is closely related to what is known as the “Internet of Things” – the concept that, increasingly, everyday devices are becoming connected and “smart”; able to communicate with each other wirelessly, and log information about the way they are used.
In the consumer sphere, watches, bathroom scales, power sockets, vacuum cleaners and even lightbulbs– are all becoming “smart” – connectable to a wireless network and capable of recording and uploading information about their use. And American industrial giant GE is investing heavily in what it calls the “Industrial Internet”, where sensors relay information that allow faults and breakdowns to be detected and sometimes even predicted before they happen, minimising downtime. By 2020, it is estimate that 30 billion devices around the world will all be connected to each other.
To me it is very clear that we are well past the time when businesses should be wondering if they have to worry about big data. The real question is, can they afford not to? Big data is already revolutionising just about every industry and area of life and if you aren’t on board, you’re risking getting left behind.
Bernard Marr’s latest book “Big Data: Using SMART Big Data, Analytics and Metrics To Make Better Decisions and Improve Performance” discusses how any business can create a big data strategy.
Guest bloggers may post on this site. The views, opinions and positions expressed within these guest posts are those of the author alone and do not represent those of Hiscox or its employees. The accuracy, completeness and validity of any statements made within these guest blogs are not guaranteed and we accept no liability for any errors, omissions or representations or any liability regarding infringement of intellectual property rights. Our social media house rules which also include details on how to contact us about any concerns you have regarding our social media channels, can be found here.
