Some followers of Ventana Research may recall my work here several years ago. Here and elsewhere I have spent most of my career in the data and analytics markets matching user requirements with technologies to meet those needs. I’m happy to be returning to Ventana Research to resume investigating ways in which organizations can make the most of their data to improve their business processes; for a first look, please see our 2016 research agenda on Big Data and Information Optimization. I relish the opportunity to conduct primary market research in the form of Ventana’s well-known benchmark research and to help end users and vendors apply the information collected in those studies.

Much has happened since I was previously part of the Ventana Research team. One major change is the explosive growth in the use and acceptance of big data. For example, when I conducted the first benchmark research on big data, only 22 percent of participants were using Hadoop in production. Our more recent research shows more than 50 percent growth in the number of respondents using Hadoop in production (which is now 37%).

In the area of predictive analytics, another research study from my prior tenure identified a skills shortage. This shortage was identified in several ways. More than four out of five (83%) participants indicated that users did not have enough skills training, and more than half (58%) said they didn’t understand the mathematics needed to produce their own analyses. In the interim the numerous university programs have begun to help address this shortage. A Google search shows that New York University, Columbia, Indiana University, Wesleyan, University of Washington, University of Michigan, University of Rochester and University of Texas San Antonio have created data science programs – and this is just page one of the search results. I anticipate that these will be increasingly popular programs as the rise of big data will continue to drive demand for these skills. For the time being, our more recent study on predictive analytics suggests that these skill shortages still exist with very similar responses of 79 percent and 66 percent, respectively. I’ll be continuing to watch these and other analytics issues noted in our 2016 Business Analytics Research Agenda.

We have also seen a sea change in the acceptance of open source software in enterprises. I think it is fair to say that open source helped drive the growth in big data with various projects including Cassandra, Hadoop, MongoDB and Spark enabling organizations to experiment with large volumes of data before making significant license purchases to put those systems into production. The open source momentum is further evidenced by some large vendors taking formerly proprietary, “closed source” technologies and making them open source. Perhaps the biggest example is Microsoft making its .NET technology open source. My former employer Pivotal also converted its data management products, in which it had invested more than 10 years of proprietary development, to open source versions.

Another notable change is the growth of interest in the Internet of Things (IoT). Many years ago I considered a position with a vendor that helped organizations manage RFID data. Adoption was slow at the time, in part because of the cost of RFID tags but also because of the cost and challenges of collecting and analyzing very large volumes of data. As big data technologies have grown, so too has interest in IoT. Technologies exist today to make processing such large amounts of data possible in the time frames and at costs that make it practical to consider how instrumentation of devices can be used to enhance business performance. We’ll be undertaking specific research on this topic in 2016: See our Big Data and Information Optimization Research Agenda.

If he were alive today, Charles Darwin might have noted the emergence of a new species: the Unicorn, which Wikipedia defines as a startup company, often software-based, whose valuation exceeds US$1 billion. You might wonder how this financial trend impacts our research and the advice we provide. The answer is that such valuations have the potential to alter the behavior of the markets we cover. They provide these startup vendors access to funding great enough to change the competitive landscape. Such investments can put pressure on existing vendors to step up their game. In some cases it can also cause consolidation in the market or even cause certain vendors to exit markets, such as Intel did when it decided to get out of the Hadoop distribution market. At Ventana Research we are ready to help end-user organizations evaluate whether the unicorns are ready for prime time and how their existence might impact their existing software vendors. One way in which we help in this process is with our Ventana Research Value Indexes, which provide fact-based assessments of various software products within a variety of market segments.

So I hope you’ll pardon the interruption in our conversation. It’s good to be back, and I am looking forward to working with the entire Ventana Research team to provide research and insights that will help guide your use of technology to improve your business decisions and processes.

Regards,

David Menninger

SVP & Research Director