Services for Organizations

Using our research, best practices and expertise, we help you understand how to optimize your business processes using applications, information and technology. We provide advisory, education, and assessment services to rapidly identify and prioritize areas for improvement and perform vendor selection

Consulting & Strategy Sessions

Ventana On Demand

    Services for Investment Firms

    We provide guidance using our market research and expertise to significantly improve your marketing, sales and product efforts. We offer a portfolio of advisory, research, thought leadership and digital education services to help optimize market strategy, planning and execution.

    Consulting & Strategy Sessions

    Ventana On Demand

      Services for Technology Vendors

      We provide guidance using our market research and expertise to significantly improve your marketing, sales and product efforts. We offer a portfolio of advisory, research, thought leadership and digital education services to help optimize market strategy, planning and execution.

      Analyst Relations

      Demand Generation

      Product Marketing

      Market Coverage

      Request a Briefing



        David Menninger's Analyst Perspectives

        << Back to Blog Index

        Is Hadoop Disappearing?


        There’s been some speculation in the market that Hadoop may be disappearing. Some of this speculation has been driven by vendors that have recently downplayed Hadoop in their marketing efforts. For example, the Strata+Hadoop World conference is now known as the Strata Data Conference. The Hadoop Summit is now known as the Dataworks Summit. In Cloudera’s S-1 filing with the SEC for its initial public offering, the term “Hadoop” appears only 14 times, while the term “machine learning” appears 83 times. So, if some of the vendors that created the market appear to be pivoting away from Hadoop, does your organization need to do something similar, or is there a role for Hadoop in your IT architecture?

        In light of this speculation, I recently compiled findings from the last six years of research into Hadoop adoption rates and the ways in which organizations store big data. Ventana Research conducted one of the first research studies that looked at Hadoop adoption rates in 2011. Our most recent benchmark research, on data preparation, was just completed and will be published soon. In the six years in between, every data and analytics benchmark research undertaking we performed included questions about which technologies organizations use to store big data. I’ve included here a chart that shows the results over time.

        vr_VR_Hadoop_Adoption_Rates-01.svgThe chart requires some explanation and some caveats. The orange line represents Hadoop deployment rates and the yellow line represents RDBMSs. For one thing, we didn’t always ask the question in the exact same way. For instance, in our Internet of Things benchmark research we asked how organizations store event data (not big data). The dip below 40 percent in RDBMS usage corresponds to one particular study, the IoT study, in which we asked how IoT event data, a form of big data, was stored. Still, the trend over time from nearly 90 percent RDBMS usage in 2011 to about 60 percent usage in 2017 represents a significant decline.

        Also remarkable was the overall consistency of organizations’ adoption rates for Hadoop, which have hovered around 20 to 25 percent. In two instances, higher adoption rates were reported, but the benchmark research in both cases was specifically about big data, so the audience may have been biased toward Hadoop.

        The chart also depicts a decline in Hadoop adoption rates by a few percentage points over time. Does that mean Hadoop has peaked and is now in decline? Perhaps. But I’d argue that the more significant trend depicted in the chart is the decline of RDBMS usage for big data. In our benchmark research we ask organizations to list all the technologies they use to store big data. It appears to me that Hadoop (and perhaps other technologies) are being used to manage data that used to be stored in RDBMSs.

        What does this mean for you? Based on the research data and my conversations in the marketplace, Hadoop is here to stay and some of the workloads that were previously handled using RDBMSs are shifting to Hadoop and NoSQL technologies. RDBMSs still predominate but the research suggests that workloads are going to continue to be split among different environments.

        You can help your organization by ensuring that end users have access to and can work with a variety of technologies. In addition, your ability to make Hadoop data easier to access and use will also make an impact, as our Big Data Analytics research shows that only one in six organizations have the Hadoop skills that they need.

        Hadoop can be made more approachable in several ways. The research shows that users prefer to access big data using SQL and there are many SQL-on-Hadoop alternatives. There are also vendors such as AtScaleKyvos and others that provide a semantic layer on top of Hadoop so your organization can access Hadoop with existing BI tools. Data virtualization vendors such as Denodo provide yet another approach that makes Hadoop data available via SQL and standard BI tools.

        We continue to study big data deployments and are currently conducting Dynamic Insights research on the topic of data lakes. Please participate in this research as it will help us create a more accurate picture of the market that we can share with you in our future publications. As I have written previously, data lakes can be a useful way to help your organization get more value from its big data deployments.

        The ultimate objective of your big data efforts should be to enable your organization to focus on the A’s of big data – analytics, awareness, anticipation, action. Big data is here to stay. The research suggests it will take many forms and by recognizing this issue you can adopt big data strategies that will encompass those many forms and yield improved operational results for your organization.

        Regards,

        David Menninger

        SVP & Research Director

        Follow Me on Twitter and Connect with me on LinkedIn.

        David Menninger
        Executive Director, Technology Research

        David Menninger leads technology software research and advisory for Ventana Research, now part of ISG. Building on over three decades of enterprise software leadership experience, he guides the team responsible for a wide range of technology-focused data and analytics topics, including AI for IT and AI-infused software.

        JOIN OUR COMMUNITY

        Analyst Perspective Policy

        • Ventana Research’s Analyst Perspectives are fact-based analysis and guidance on business, industry and technology vendor trends. Each Analyst Perspective presents the view of the analyst who is an established subject matter expert on new developments, business and technology trends, findings from our research, or best practice insights.

          Each is prepared and reviewed in accordance with Ventana Research’s strict standards for accuracy and objectivity and reviewed to ensure it delivers reliable and actionable insights. It is reviewed and edited by research management and is approved by the Chief Research Officer; no individual or organization outside of Ventana Research reviews any Analyst Perspective before it is published. If you have any issue with an Analyst Perspective, please email them to ChiefResearchOfficer@ventanaresearch.com

        View Policy

        Subscribe to Email Updates

        Posts by Month

        see all

        Posts by Topic

        see all


        Analyst Perspectives Archive

        See All