David Menninger's Analyst Perspectives

Research Agenda Will Track Big Data and Information Management in 2017

Posted by David Menninger on Feb 15, 2017 9:09:44 AM

Big data has become an integral part of information management. Nearly all organizations have some need to access big data sources and produce actionable information for decision-makers. Recognizing this connection, we merged these two topics when we put together our recently published research agendas for 2017. As we plan our research, we focus on current technologies and how they can be used to improve an organization’s performance. We then share those results with our readers.

Our research agenda for big data and information management begins with data preparation. We recently launched a benchmark research study on the topic and invite you to participate in it. End users today expect to be able to access data sources directly without support from IT. Data preparation attempts to solve that problem by making data more accessible throughout the information life cycle, in the end adding value to business and IT processes. In our research, we’ll explore how organizations establish data preparation to provide information responsiveness while also supporting enterprise requirements for data quality, governance and security.

The larger context of data preparation is the overall process of data integration. Data sources are larger, more diverse and more distributed, found in cloud-based systems, unstructured data and connected devices generating data throughout the enterprise. Data lakes and data virtualization are two ways in which organizations can integrate and provide access to these diverse data sources. We’ll look how organizations utilize data lakes, including whether they are using data virtualization, for comprehensive data sets that support both analytics and data governance.

In a related area, organizations face challenges when new data types and sources have been adopted without sufficient data governance. The open source ecosystem surrounding big data, which I have written about, necessitates the creation of new governance capabilities, which has spawned new open source projects and has attracted independent software vendors (ISVs) to fill in gaps with new governance tools. Our research will identify best practices for developing and implementing a comprehensive data governance strategy for all types of data, both new and old.

The Internet of Things, which includes an increasingvr_IoT_and_OI_15_latency_in_IoT_applications-1.png number of devices that are instrumented and connected, drives a new set of requirements in addition to governance. IoT data is generated constantly, increasing demands for streaming data technologies combined with data science to enable continuous analytics. Our recent IoT benchmark research shows that nearly half (46%) of organizations [IoT Q25] consider it essential to process IoT events in seconds or subseconds. We have further research planned to complement our IoT research by studying how organizations are dealing with streams of machine data and IoT data. We’ll also be diving deeper into the IoT research we have completed to share more of our findings and their implications for organizations of all sorts.

As organizations increase their arrays of big data sources, including streams of IoT data, the importance of data science also increases. Traditional analytical tools and techniques are not effective at identifying all the valuable relationships in large bodies of data, nor are people effective at real-time processing. Data science helps tackle these issues. However, as our Next-Generation Predictive Analytics research shows, most organizations suffer from a lack of skilled data science resources. More than three-quarters (79%) of participants reported that users don’t have enough skills to perform their own data science analyses.

Machine learning provides a potential solution to this skills shortage, at least in part. Through self-learning algorithms, the development and refinement of data science routines become more automated, requiring fewer resources to develop models that predict and optimize outcomes. Such automation is critical in the areas of big data and IoT. Manual processes simply cannot support the volumes of data and the continuous analysis required. We’ll take a close look at how machine learning impacts organizations as part of our 2017 agenda.

At the core of all these issues, information management requirements continue to grow. Both IT and lines of business need to be able to manage large volumes of structured and unstructured data, often streaming on a continuous basis. Big data technologies continue to proliferate to support streaming, at-rest and in-memory data. We will be studying how organizations deal with this proliferation of technologies to extract business value from the data that they gather.

For the full list of research planned, please download and review our Big Data and Information Management research agenda. I invite you to participate in this research as we conduct it during the year. I look forward to sharing the insights we discover and to helping your organization apply those insights to its business needs.


David Menninger
SVP & Research Director

Follow Me on Twitter @dmenningerVR and Connect with me on LinkedIn.

Topics: Big Data, Data Science, Analytics, Data Governance, Data Integration, Data Preparation, Information Management, Internet of Things, Machine Learning and Cognitive Computing, Machine Learning Digital Technology

David Menninger

Written by David Menninger

David is responsible for the overall research direction of data, information and analytics technologies at Ventana Research covering major areas including Analytics, Big Data, Business Intelligence and Information Management along with the additional specific research categories including Information Applications, IT Performance Management, Location Intelligence, Operational Intelligence and IoT, and Data Science. David is also responsible for examining the role of cloud computing, collaboration and mobile technologies as they affect these areas. David brings to Ventana Research over twenty-five years of experience, through which he has marketed and brought to market some of the leading edge technologies for helping organizations analyze data to support a range of action-taking and decision-making processes. Prior to joining Ventana Research, David was the Head of Business Development & Strategy at Pivotal a division of EMC, VP of Marketing and Product Management at Vertica Systems, VP of Marketing and Product Management at Oracle, Applix, InforSense and IRI Software. David earned his MS in Business from Bentley University and a BS in Economics from University of Pennsylvania.