Although Big Data and Analytics is not technically a new market, it has been garnering a lot of attention recently and for good reason. The confluence of several factors (Moore’s law, storage capacity price and capacity, network capacity and the impending growth of machine to machine) will cause an exponential growth in the volume of data and the ability to gather intelligence from the data for business and social benefit. Given the importance and disruptive potential, suppliers that touch the data are vying for a prominent market position.
Analytics will become a strong and disruptive force in nearly all vertical and horizontal segments. Why? It is inevitable but the trajectory of its growth will be lumpier. My outlook is based on experience with big data and from looking at other technology transitions for lessons and patterns that can help provide additional insights. This blog will look mostly at the softer issues.
The availability of skilled data analysts is one key issue that will slow down adoption. Having big data sets and state of the art tools and are not enough: they do not give you the answers. Talented, curious people that understand how to operate the tools and build models do. While some in the industry realize this, it will take more time to get more trained and capable data analysts than it will take to develop and acquire these new tools. A corollary to this point: companies will need to change their culture and processes to be more data driven. Taking a realist’s viewpoint, the organizational changes are often the hardest.
There is the law of diminishing returns for data. Our mindset has been more data equals better insights and therefore equals higher value or profit, and the cost of adding more data and more processing is less than the value added. However, at some point this relationship will start to break down and the insights may not get any more valuable and/or the incremental cost of storing and processing the data will be more than incremental value. We all realize that the cost of this processing and storage is becoming very low, but it is still not free. Another thought on value to consider: if all competitors are using approximately the same tools for competitive advantage, is it a sustainable advantage?
There are regulatory, privacy and customer acceptance issues that will need to be addressed. While not a showstopper, companies pursuing big data efforts will need to tread lightly. Consumers seem to be much more concerned about some companies than others in terms of collecting and analyzing data. Telcos, cable companies and utilities are often skewered for their data collection efforts (smart meters); Google seems to be able to collect much more data without too much push back.
Given these issues, my advice to the companies vying for the pole position in this market: 1) don’t just push the speeds and feeds of the technology; 2) help develop the business cases to make sure big data efforts are profitable; 3) take a leadership position in training and development; and 4) work with privacy groups to ensure that privacy is protected.