David Menninger's Analyst Perspectives

MicroStrategy is a long-standing business intelligence and analytics vendor that operates worldwide. Founded in 1989, this publicly traded company with hundreds of millions of dollars in revenue recently held its first in-person conference since prior to the pandemic. Similar to previous in-person events, the event was well attended by about 2,000 attendees and exhibitors. The theme, “MicroStrategy One,” is a way to explain the breadth of capabilities the company offers. The breadth of the product offering is one of the company’s greatest strengths, but also one of its biggest challenges.

Artificial intelligence (AI) has evolved from a highly specialized niche technology to a worldwide phenomenon. Nearly 9 in 10 organizations use or plan to adopt AI technology. Several factors have contributed to this evolution. First, the amount of data they can collect and store has increased dramatically while the cost of analyzing these large amounts of data has decreased dramatically. Data-driven organizations need to process data in real time which requires AI. In addition, analytics vendors have been augmenting business intelligence (BI) products with AI. And recently, ChatGPT has raised awareness of AI and instigated research and experimentation into new ways in which AI can be applied. This perspective, the second in a series on generative AI, introduces some of the concepts behind ChatGPT, including large language models and transformers. Understanding how these models work can help provide a better understanding of how they should be applied and what cautions are necessary.

Generative AI is a class of artificial intelligence used to generate new, seemingly real content. Broadly speaking, AI has traditionally been used to identify patterns in data and apply those patterns to categorize and predict behaviors. For instance, it can organize customers into groups (or clusters) with similar characteristics, or predict which customers are most likely to respond to certain offers.

In my perspective on decision intelligence, I lamented the fact that business intelligence technologies have left the rest of the exercise to the reader for too long. Making a decision is a process that involves many steps and many people. Decision-making is so complicated and divorced from day-to-day business processes that organizations have had to create entirely separate teams to focus on the analytics and data to support it. One aspect of the decision-making process that can be enhanced by technology is collaboration.

Organizations are becoming increasingly aware of the potential value that can be gained by processing big data. As data sources grow, it becomes important to have tools and methods to effectively process, analyze and visualize this information from disparate systems and warehouses.

Organizations are continuously searching for new business opportunities hidden in their data. They are using various technologies including artificial intelligence and machine learning (AI/ML) to uncover granular insights that can support decision-making. Existing tools and dashboards are effective for observing standard metrics; however, they do not address follow-up questions, such as why things are happening or how those events impact performance. Organizations also struggle to derive complete value from big data. Our Analytics and Data Benchmark Research shows that only 1 in 5 organizations are very confident in their ability to analyze large volumes of data.

Now more than ever, effective data management is crucial to enable decision-makers to better assess information and take calculated actions. It is also important to keep up with the latest trends and technologies to derive higher value from data and analytics and maintain a competitive edge in the market. However, every organization faces challenges with data management and analytics. And as organizations scale, the complexity only increases, creating a need for better data governance, data quality and streamlined and automated processes. DataOps can help solve many of the challenges organizations encounter when trying to unlock the power of data by expanding data use to various parts of an organization. Hitachi Vantara offers DataOps technology that enables organizations to improve data agility and automation. It provides cloud-ready infrastructure, advanced data management software and a broad range of support services.

Data analytics provide valuable insights and enable organizations to make better decisions, improve performance and gain a competitive advantage in the marketplace. Analytics can change frequently depending on the data being analyzed and the methods used to gather and process it. Factors such as new data, changes in the underlying systems or updates to algorithms can all contribute to differences in an analysis. AnalyticOps helps ensure data is accurate, up-to-date and consistent across different systems and teams, and that analytical models are robust, reliable and continuously improved.

For years various types of systems have produced log files to help with monitoring, debugging and performance management. Often, this information was used in forensic analyses of why interruptions in service or other problems occurred. In many cases, log files are still used this way. But systems have grown more complicated, and many more devices are instrumented. Systems have been decomposed into much finer-grained, interdependent services. Infrastructure is now distributed between on-premises and multiple cloud providers. In addition, expectations now include 24x7 operation and real-time responsiveness. All of these factors combine to create challenges with volume and velocity of data that is collected and analyzed.

I’ve previously written about the analytics continuum, which spans a range of capabilities including reporting, visualization, planning, real-time processes, natural language processing, artificial intelligence and machine learning. I’ve also written about the analysis that goes into making intelligent decisions with decision intelligence. In this perspective, I’d like to focus on one end of the analytics continuum, which I’ll label advanced analytics.

Markets have been more volatile than ever. It creates a need for decision makers to utilize technologies such as artificial intelligence and machine learning (AI/ML) to better understand the external factors that impact their business. By identifying these factors, organizations can better plan for changing market environments and seize market opportunities. However, manual modeling is a time-consuming process and results in a limited number of models and tests. Also, updating those models is slow and laborious. With the addition of market volatility, it creates multiple challenges for CFOs, managers and financial planning specialists. With limited exposure to external drivers of demand and delivery, the process becomes very costly. Developing accurate forecasts requires integrating exogenous data with the internal performance data, but it’s challenging to find quality external data and then get that raw data clean enough to input into any model. My colleague, Robert Kugel, recently shared his perspective on using external data for forecasting, budgeting and planning to enhance predictive capabilities.

Ventana Research recently announced its 2023 Market Agenda for Analytics, continuing the guidance we have offered for nearly two decades to help organizations derive optimal value from technology investments to improve business outcomes.

I’m proud to share Ventana Research’s 2023 Market Agenda for Digital Technology. Our focus in this agenda is to deliver expertise to help organizations prioritize technology investments that improve customer, partner and workforce experiences while also increasing organizational effectiveness and agility.

Organizations conduct data analysis in many ways. The process can include multiple spreadsheets, applications, desktop tools, disparate data systems, data warehouses and analytics solutions. This creates difficulties for management to provide and maintain updated information across multiple departments. Our Analytics and Data Benchmark Research shows that organizations face a variety of challenges with analytics and business intelligence. One-third of participants find it difficult to integrate analytics and BI with other business processes. Participants also find that not all software is flexible enough for the constantly changing business environment, and that it is hard to access all data sources.

For far too long, business intelligence technologies have left the rest of the exercise to the reader. Many of these tools do an excellent job providing information in an interactive way that lets organizations dive into the data and learn a lot about what has happened across all aspects of the business. More recently, many of these tools have added augmented intelligence capabilities that help explain why things happened. But rarely did any of these tools provide information about what to do or how to evaluate the alternative ways in which you might respond.

Analytics processes are all about how organizations use data to create metrics that help manage and improve operations. Yet, the discipline applied to analytics processes seems to be lacking compared to data processes. I’ve pointed out that the weak link in data governance is often analytics. Organizations can also do a better job tying AnalyticOps to DataOps and do more to define and manage metrics. Our research has shown that creating and managing metrics in a semantic model improves analytics processes.

In previous perspectives in this series, I’ve discussed some of the realities of cloud computing including costs, hybrid and multi-cloud configurations and business continuity. This perspective examines the realities of security and regulatory concerns associated with cloud computing. These issues are often cited by our research participants as reasons they are not embracing the cloud. To be fair, the majority of our research participants are embracing the cloud. However, among those that have not yet made the transition to the cloud, security and regulatory concerns are among the most common issues cited across the various studies we have conducted.

Recently, I suggested you need to “mind the gap” between data and analytics. This perspective addresses another gap — the gap in skills between business intelligence (BI) and artificial intelligence/machine learning (AI/ML).

Embedded business intelligence (BI) continues to transform the business landscape, enabling organizations to quickly interpret data and convert it into actionable insights. It allows organizations to extract information in real time and answer wide-ranging business questions. Embedding analytics helps tackle the issue of extracting information from data which is a time-consuming process. Our research shows organizations spend more time cleaning and optimizing data for analysis rather than creating insights. On top of that, they are adding more data sources and information systems which in turn introduces more complexity. Our Analytics and Data Benchmark Research shows that organizations face various challenges with analytics and BI. More than one-third of participants (35%) responded that they find it hard to integrate analytics and BI with business processes and connect to multiple data sources. By embedding analytics and BI into business processes and workflows, organizations can enable users to make critical decisions fast, enhancing overall business agility.

In today’s data-driven world, organizations need real-time access to up-to-date, high-quality data and analysis to keep pace with changing market dynamics and make better strategic decisions. By mining meaningful insights from enterprise data quickly, they gain a competitive advantage in the market. Yet, organizations face a multitude of challenges when transitioning into an analytics-driven enterprise. Our Analytics and Data Benchmark Research shows that more than one-quarter of organizations find it challenging to access data sources and integrate data and analytics in business processes. Vendors such as IBM offer a broad set of analytics tools with self-service capabilities that allows organizations to reduce IT dependencies and enables decision-makers to recognize performance gaps, market trends and new revenue opportunities. Its technology can simplify data access for self-service applications, enabling users to make business decisions informed by insights and take the guesswork out of decision-making.

In my previous perspectives on cloud computing, I addressed some of the realities of cloud costs as well as hybrid and multi-cloud architectures. In the midst of the pandemic, my colleague, Mark Smith, authored a series of perspectives on considerations for business continuity in general, beginning with this look at some of the investments organizations must make to mitigate the risk of business disruptions. In this perspective, I’d like to address some of the realities of business continuity and cloud computing and how they impact the digital technologies of an organization. The cloud can be both advantageous and disadvantageous when it comes to providing business continuity.

If you’ve ever been to London, you are probably familiar with the announcements on the London Underground to “mind the gap” between the trains and the platform. I suggest we also need to mind the gap between data and analytics. These worlds are often disconnected in organizations and, as a result, it limits their effectiveness and agility.

Artificial intelligence and machine learning are valuable to data and analytics activities. Our research shows that organizations using AI/ML report gaining competitive advantage, improving customer experiences, responding faster to opportunities and threats and improving the bottom line with increased sales and lower costs. No wonder nearly 9 in 10 (87%) research participants report using AI/ML or planning to do so.

As I recently pointed out, process mining has emerged as a pivotal technology for data-driven organizations to discover, monitor and improve processes through use of real-time event data, transactional data and log files. With recent advancements, process mining has become more efficient at discovering insights in complex processes using algorithms and visualizations. Organizations use it to better understand the current state of systems and business processes. It is also used to enable business process intelligence and improvement in any function or industry using events and activity models for data-driven decision-making. We assert that through 2024, 1 in 4 organizations will look to streamline their operations by exploring process mining to optimize workflow and business processes.

Process mining is defined as the analysis of application telemetry including log files, transaction data and other instrumentation to understand and improve operational processes. Log data provides an abundance of information about what operations are occurring, the sequences involved in the processes, how long the processes are taking and whether or not the processes are completed successfully. As computing power has increased and storage costs have decreased, the economics of collecting and analyzing large amounts of log data have become much more attractive.

In my first perspective on cloud computing realities, I covered some of the cost considerations associated with cloud computing and how the cloud costing model may be different enough from on-premises models that some organizations are taken by surprise. In this perspective. I’d like to focus on realities of hybrid and multi-cloud deployments.

Organizations are collecting data from multiple data sources and a variety of systems to enrich their analytics and business intelligence (BI). But collecting data is only half of the equation. As the data grows, it becomes challenging to find the right data at the right time. Many organizations can’t take full advantage of their data lakes because they don’t know what data actually exists. Also, there are more regulations and compliance requirements than ever before. It is critical for organizations to understand the kind of data they have, who is handling it, what it is being used for and how it needs to be protected. They also have to avoid putting too many layers and wrappers around the data as it can make the data difficult to access. These challenges create a need for more automated ways to discover, track, research and govern the data.

Business intelligence has evolved. It now includes a spectrum of analytics, one of the most promising of which has been described as augmented intelligence. Some organizations have used the term to describe the practical reality that artificial intelligence with machine learning is not replacing human intelligence, but augmenting it. The term also represents the application of AI/ML to make business intelligence and analytics tools more powerful and easier to use. It’s this latter usage that I prefer and I’d like to explore in this perspective.

The migration to cloud is obvious. Organizations are adopting cloud computing for all variety of applications and use cases. Managed cloud services, commonly referred to as software as a service (SaaS), offer many benefits to organizations including significantly reduced labor costs for system administration and maintenance, as many of these costs are shifted to the software vendor. SaaS also provides organizations with faster time to value as they adopt new technologies by eliminating the need to acquire and configure hardware, and it also eliminates the need to install software. In fact, we assert that by 2025, nine in 10 organizations will be using multiple cloud applications in order to minimize the costs of administration and maintenance. Yet, there are some challenges associated with cloud computing I’d like to address in a series of Analyst Perspectives:

Organizations are managing and analyzing large datasets every day, identifying patterns and generating insights to inform decisions. This can provide numerous benefits for an organization, such as improved operational efficiency, cost optimization, fraud detection, competitive advantage and enhanced business processes. By bringing the right, actionable data to the right user, organizations can potentially speed up processes and make more effective operational decisions.

When I looked at the state of analytics recently, it was clear that analytics are not as widely deployed within organizations as they should be. Only 23% of participants in our Analytics and Data Benchmark Research reported that more than one-half of their organization’s workforce are using analytics. There are many elements to becoming a data-driven organization, as my colleague Matt Aslett points out, but analytics are a necessary component. Our research shows that organizations recognize the importance of embedded analytics, ranking it the second most important digital technology in their analytics and data efforts behind big data and ahead of artificial intelligence and machine learning (AI/ML).

The analytics and business intelligence market landscape continues to grow as more organizations seek robust tools and capabilities to visualize and better understand data. BI systems are used to perform data analysis, identify market trends and opportunities and streamline business processes. They can collect and combine data from internal and external systems to present a holistic view.

I often use the term “analytics” to refer to a broad set of capabilities, deliberately broader than business intelligence. In this Perspective, I’d like to share what decision-makers should consider as they evaluate the range of analytics requirements for their organization.

Organizations are collecting vast amounts of data every day, utilizing business intelligence software and data visualization to gain insights and identify patterns and errors in the data. Making sense of these patterns can enable an organization to gain an edge in the marketplace and plan more strategically.

We’ve recently published our latest Benchmark Research on Data Governance and it’s fair to say, “you’ve come a long way, baby.” Many of you reading this weren’t around when that phrase was introduced in 1968 to promote Virginia Slims cigarettes, but you may have heard the phrase because it went on to become a part of popular culture. We’ve learned a lot about cigarettes since then, and we’ve learned a lot about data governance, too.

Organizations are continuously increasing the use of analytics and business intelligence to turn data into meaningful and actionable insights. Our Analytics and Data Benchmark Research shows some of the benefits of using analytics: Improved efficiency in business processes, improved communication and gaining a competitive edge in the market top the list. With a unified BI system, organizations can have a comprehensive view of all organizational data to better manage processes and identify opportunities.

I’ve never been a fan of talking about semantic models because most of the workforce probably doesn’t understand what they are, or doesn’t recognize them by name. But the findings in our recent Analytics and Data Benchmark Research have changed my mind. The research shows how important a semantic model can be to the success of data and analytics processes. Organizations that have successfully implemented a semantic model are more than twice as likely to report satisfaction with analytics (77%) compared with a 33% overall satisfaction rate. Therefore, I owe it to all of you to write about them.

Organizations are constantly trying to streamline and optimize business data to solve complex problems and identify opportunities to increase revenue and accelerate business growth. The data is usually stored in multiple systems with various data governance rules, which makes it complicated to democratize data and analytics within an organization. The most pressing concerns cited by participants in our Analytics and Data Benchmark Research include difficulty integrating with business processes, systems not flexible or adaptable to change and challenges accessing data sources.

Organizations are scaling business intelligence initiatives to gain a competitive advantage and increase revenue as more data is created. Lack of expertise, data governance and slow performance can impact these efforts. Our Analytics and Data Benchmark Research finds some of the most pressing complaints about analytics and BI include difficulty integrating with other business processes and flexibility issues. Kyvos is a BI acceleration platform that enables BI and analytics tools to analyze massive amounts of data. It offers support for online analytical processing-based multidimensional analytics, enabling workers to access large datasets with their analytics tools. It operates with major cloud platforms, including Google Cloud, Amazon Web Services and Microsoft Azure.

There is a fundamental flaw in information technology, or at least in the way it is most commonly delivered. Most technology systems are developed under the assumption that all people will use the system primarily in the same way. Sure, there are some options built in — perhaps the same action can be initiated by either clicking on a button, selecting a menu item or invoking a keyboard short-cut. The problem is that when every variation needs to be coded into the system, the prospect of providing personalized software programs to every individual is impractical.

The data governance landscape is growing rapidly. Organizations handling vast amounts of data face multiple challenges as more regulations are added to govern sensitive information. Adoption of multi-cloud strategies increases governance concerns with new data sources that are accessed in real time. Our Data Governance Benchmark Research shows that organizations face multiple challenges when deploying data governance. Three-quarters (73%) of organizations report disparate data sources as the biggest challenge, and half of the organizations report creating, modifying, managing and enforcing governance policies as the second biggest challenge.

Organizations have been using data virtualization to collect and integrate data from various sources, and in different formats, to create a single source of truth without redundancy or overlap, thus improving and accelerating decision-making giving them a competitive advantage in the market. Our research shows that data virtualization is popular in the big data world. One-quarter (27%) of participants in our Data Lake Dynamic Insights Research reported they were currently using data virtualization, and another two-quarters (46%) planned to include data virtualization in the future. Even more interesting, those who are using data virtualization reported higher rates of satisfaction (79%) with their data lake than those who are not (36%). Our Analytics and Data Benchmark Research shows more than one-third of organizations (37%) are using data virtualization in that context. Here, too, those using data virtualization reported higher levels of satisfaction (88%) than those that are not (66%).

I have written previously that the world of data and analytics will become more and more centered around real-time, streaming data. Data is created constantly and increasingly is being collected simultaneously. Technology advances now enable organizations to process and analyze information as it is being collected to respond in real time to opportunities and threats. Not all use cases require real-time analysis and response, but many do, including multiple use cases that can improve customer experiences. For example, best-in-class e-commerce interactions should provide real-time updates on inventory status to avoid stock-out or back-order situations. Customer service interactions should provide real-time recommendations that minimize the time to resolution. Location-based offers should be targeted at the customer’s current location, not their location several minutes ago. Another domain where real-time analyses are critical is internet of things (IoT) applications. Additionally, use cases like predictive maintenance require timely information to prevent equipment failures that help avoid additional costs and damage.

For years, maybe decades, we have heard about the struggles between IT and line-of-business functions. In this perspective, we will look at some of the data from our Analytics and Data Benchmark Research about the roles of IT and line-of-business teams in analytics and data processes. We will also look at some of the disconnects between these two groups. And, by looking at how organizations are operating today and the results they are achieving, we can discern some of the best practices for improving the outcomes of analytics and data processes.

Organizations face various challenges with analytics and business intelligence processes, including data curation and modeling across disparate sources and data warehouses, maintaining data quality and ensuring security and governance. Traditional processes are slow when transforming large and diverse datasets into something which is easily consumable in BI. And, it can take days or weeks to create reports and dashboards — maybe longer if processes change and new data sources are introduced. Our Analytics and Data Benchmark Research shows that the most time-consuming processes are preparing data, reviewing it for quality issues and preparing reports for presentation and distribution.

Today, organizations understand the importance of good external data that can be integrated with internal data to train machine learning models. Our Machine Learning Dynamic Insights research showed that external data adds a significant value in gaining competitive advantage, improving customer experience and increasing sales. But getting the right external data for a particular requirement is not always easy. Internal data is usually not enough to train different models because of its narrow scope of usage and lack of relevance. Manual data acquisition methods are resource-intensive and can take weeks or months to get the data ready to feed into models.

Natural language processing (NLP) is a field that combines artificial intelligence (AI), data science and linguistics that enables computers to understand, interpret and manipulate text or spoken words. NLP includes generating narratives based on a set of data values, using text or speech as inputs to access information, and analysing text or speech, for instance, to determine its sentiment. There are various techniques for interpreting human language, ranging from statistical and machine learning (ML) methods to rules-based and algorithmic approaches. In this perspective, we will focus on two aspects of NLP: natural language query (NLQ), which offers the ability to use natural language expressions to discover and understand data, and natural language generation (NLG), which uses AI to produce written or spoken narratives from a dataset. NLQ and NLG enable business personnel to communicate information needs with business intelligence (BI) systems more easily.

Despite all the advances organizations have made with respect to analytics, our most recent research shows the majority of the workforce in the majority of organizations are not using analytics and business intelligence (BI). Less than one-quarter (23%) report that one-half or more of their workforce is using analytics and BI. This is a problem. It means organizations are not enabling their workforce to perform at peak efficiency and effectiveness. It means the workforce in many organizations does not have access to the same information by which they are being measured. It means organizations must find other ways to communicate with, and manage, the workforce.

Many organizations invest in data governance out of concern over misuse of data or potential data breaches. These are important considerations and valid aspects of data governance programs. However, good data governance also has positive impacts on organizations. For example, I have previously written about the valuable connection between the use of data catalogs and satisfaction with an organization’s data lake. Our most recent Analytics and Data Benchmark Research demonstrates some of the beneficial links between data governance and analytics. In this Perspective, I’ll share some of the correlations identified in our research.

Organizations of all sizes are dealing with exponentially increasing data volume and data sources, which creates challenges such as siloed information, increased technical complexities across various systems and slow reporting of important business metrics. Migrating to the cloud does not solve the problems associated with performing analytics and business intelligence on data stored in disparate systems. Also, the computing power needed to process large volumes of data consists of clusters of servers with hundreds or thousands of nodes that can be difficult to administer. Our Analytics and Data Benchmark Research shows that organizations have concerns about current analytics and BI technology. Findings include difficulty integrating data with other business processes, systems that are not flexible enough to scale operations and trouble accessing data from various data sources.

Ventana Research recently announced its 2022 Market Agenda for Analytics, continuing the guidance we have offered for nearly two decades to help organizations derive optimal value from technology investments in order to improve business outcomes.

I’m proud to share Ventana Research’s 2022 Market Agenda for Digital Technology. Our focus in this agenda is to deliver expertise to help organizations prioritize technology investments that increase workforce effectiveness and organizational agility, ensuring ongoing operations during any type of disruption.

Organizations today have huge volumes of data across various cloud and on-premises systems which keep growing by the second. To derive value from this data, organizations must query the data regularly and share insights with relevant teams and departments. Automating this process using natural language processing (NLP) and artificial intelligence and machine learning (AI/ML) enables line-of-business personnel to query the data faster, generate reports themselves without depending on IT, and make quick decisions. Some organizations have started using NLP in self-service analytics to quickly identify patterns and simplify data visualization. Our Analytics and Data Benchmark Research finds that about 81% of organizations expect to use natural language search for analytics to make timely and informed decisions.

Organizations today are working with multiple applications and systems, including enterprise resource planning (ERP), customer relationship management (CRM), supply chain management (SCM) and other systems, where data can easily become fragmented and siloed. And as the organization increases its data sources and adds more systems and custom applications, it becomes challenging to manage the data consistently and keep data definitions up to date. This increases the need to use master data management (MDM) software that can provide a single source of truth to drive accurate analytics and business operations.

TIBCO is a large, independent cloud-computing and data analytics software company that offers integration, analytics, business intelligence and events processing software. It enables organizations to analyze streaming data in real time and provides the capability to automate analytics processes. It offers more than 200 connectors, more than 200 enterprise cloud computing and application adapters, and more than 30 non-relational structured query language databases, relational database management systems and data warehouses.

Organizations have become more agile and responsive, in part, as a result of being more agile with their information technology. Adopting a DevOps approach to application deployment has allowed organizations to deploy new and revised applications more quickly. DataOps is enabling organizations to be more agile in their data processes. As organizations are embracing artificial intelligence (AI) and machine learning (ML), they are recognizing the need to adopt MLOps. The same desire for agility suggests that organizations need to adopt AnalyticOps.

Talend is a data integration and management software company that offers applications for cloud computing, big data integration, application integration, data quality and master data management. The platform enables personnel to work with relational databases, Apache Hadoop, Spark and NoSQL databases for cloud or on-premises jobs. Talend data integration software offers an open and scalable architecture and can be integrated with multiple data warehouses, systems and applications to provide a unified view of all data. Its code generation architecture uses a visual interface to create Java or SQL code.

How does your organization define and display its metrics? I believe many organizations are not defining and displaying metrics in a way that benefits them most. If an organization goes through the trouble of measuring and reporting on a metric, the analysis ought to include all the information needed to evaluate that metric effectively. A number, by itself, does not provide any indication of whether the result is good or bad. Too often, the reader is expected to understand the difference, but why leave this evaluation to chance? Why not be more explicit about what results are expected?

Our research shows that nearly all financial service organizations (97%) consider it important to accelerate the flow of information and improve responsiveness. Even just a few years ago, capturing and evaluating this information quickly was much more challenging, but with the advent of streaming data technologies that capture and process large volumes of data in real time, financial service organizations can quickly turn events into valuable business outcomes in the form of new products and services or revenue.

Databricks is a data engineering and analytics cloud platform built on top of Apache Spark that processes and transforms huge volumes of data and offers data exploration capabilities through machine learning models. It can enable data engineers, data scientists, analysts and other workers to process big data and unify analytics through a single interface. The platform supports streaming data, SQL queries, graph processing and machine learning. It also offers a collaborative user interface — workspace — where workers can create data pipelines in multiple languages — including Python, R, Scala, and SQL — and train and prototype machine learning models.

Access to external data can provide a competitive advantage. Our research shows that more than three-quarters (77%) of participants consider external data to be an important part of their machine learning (ML) efforts. The most important external data source identified is social media, followed by demographic data from data brokers. Organizations also identified government data, market data, environmental data and location data as important external data sources. External data is not just part of ML analyses though. Our research shows that external data sources are also a routine part of data preparation processes, with 80% of organizations incorporating one or more external data sources. And a similar proportion of participants in our research (84%) include external data in their data lakes.

The technology industry throws around a lot of similar terms with different meanings as well as entirely different terms with similar meanings. In this post, I don’t want to debate the meanings and origins of different terms; rather, I’d like to highlight a technology weapon that you should have in your data management arsenal. We currently refer to this technology as data virtualization. Other similar terms you may have heard include data fabric, data mesh and [data] federation. I’ll briefly discuss these terms and how I see them being used, but ultimately, I’d like to share with you some research that shows why data virtualization can be valuable, regardless of what you call it.

Alteryx is a data analytics software company that offers data preparation and analytics tools to simplify and automate data wrangling, data cleaning and modeling processes, enabling line-of-business personnel to quickly access, manipulate, analyze and output data. The platform features tools to run a variety of analytic functions such as diagnostic, predictive, prescriptive and geospatial analytics in a unified platform, and can connect to various data warehouses, cloud applications, spreadsheets and other sources.

Data governance is a hot topic these days. In fact, we are conducting benchmark research on the subject here. With increasing regulations such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA), organizations face more external oversight of their data governance practices. The risk of significant fines associated with these and other regulations, coupled with organizations’ internal compliance requirements, has brought more attention to data governance practices. We anticipate through 2023, three-quarters of Chief Data Officers’ primary concerns will be governing the privacy and security of their organization’s data.

Collibra is a data governance software company that offers tools for metadata management and data cataloging. The software enables organizations to find data quickly, identify its source and assure its integrity. Line-of-business workers can use it to create, review and update the organization's policies on different data assets. Collibra’s software uses a microservice architecture and open application programming interfaces to connect to various data ecosystems. Its data intelligence cloud platform can automatically classify data from various sources such as online transaction processing databases, master repositories and Excel files without moving the data, so the information assets stay protected.

Sisu Data is an analytics platform for structured data that uses machine learning and statistical analysis to automatically monitor changes in data sets and surface explanations. It can prioritize facts based on their impact and provide a detailed, interpretable context to refine and support conclusions. The product features fact boards, annotations and the ability to share facts and analysis across teams. Data teams and analysts start by creating common definitions of key performance indicators, which Sisu then utilizes to automatically test thousands of hypotheses to identify differences between groups.

Rapidminer is a visual enterprise data science platform that includes data extraction, data mining, deep learning, artificial intelligence and machine learning (AI/ML) and predictive analytics. It can support AI/ML processes with data preparation, model validation, results visualization and model optimization. Rapidminer Studio is its visual workflow designer for the creation of predictive models. It offers more than 1,500 algorithms and functions in their library, along with templates, for common use cases including customer churn, predictive maintenance and fraud detection. It has a drag and drop visual interface and can connect to databases, enterprise data warehouses, data lakes, cloud storage, business applications and social media. The platform also supports push-down processing for data prep and ETL inside databases to minimize data movement and optimize performance.

Confluent Platform is a streaming platform built by the original creators of Apache Kafka. It enables organizations to organize and manage streaming data from various sources. Confluent launched its IPO in June this year and raised $828 million to further expand its business. Confluent Platform was brought to several public cloud vendor marketplaces last year as Confluent Cloud. The offering is currently available in Azure, AWS, and GCP marketplaces. Furthermore, the company strengthened its partnership with Microsoft at the beginning of this year, establishing Confluent Cloud as a fully managed Apache Kafka service directly available on Microsoft Azure. Azure customers can access the extensive library of pre-built connectors, a unified billing model with options to use Azure committed spend on Confluent Cloud, and deeper integrations with Azure services.

The annual Ventana Research Digital Innovation Awards showcase advances in the productivity and potential of business applications, as well as technology that contributes significantly to the improved processes and performance of an organization. Our goal is to recognize technology and vendors that have introduced noteworthy digital innovations to advance business and IT.

The annual Ventana Research Digital Innovation Awards showcase advances in the productivity and potential of business applications, as well as technology that contributes significantly to the improved processes and performance of an organization. Our goal is to recognize technology and vendors that have introduced noteworthy digital innovations to advance business and IT.

Event data can be used to enhance existing processes, but it can also be used to dramatically impact operations, revenue models and the bottom line for manufacturers. Our Benchmark Research shows 95% of manufacturers consider it important to speed the flow of information and improve responsiveness within business processes. In this perspective I’ll share how manufacturers are working with event data to transform their organizations.

I am happy to share insights gleaned from our latest Value Index research, an assessment of how well vendors’ offerings meet buyers’ requirements. The 2021 Ventana Research Value Index: Collaborative Analytics and Data is the distillation of a year of market and product research by Ventana Research. See our prior post for a description of our methodology and included vendors.

I am happy to share insights gleaned from our latest Value Index research, an assessment of how well vendors’ offerings meet buyers’ requirements. The 2021 Ventana Research Value Index: Embedded Analytics and Data is the distillation of a year of market and product research by Ventana Research. See our prior post for a description of our methodology and included vendors.

I am happy to share insights gleaned from our latest Value Index research, an assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Mobile Analytics and Data 2021 is the distillation of a year of market and product research by Ventana Research. See our prior post for a description of our methodology and which vendors are included.

We are happy to share some insights about Amazon QuickSight drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about Google Looker drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about ThoughtSpot drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about TIBCO Spotfire drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about Sisense drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about Infor Birst drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about Microsoft Power BI drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about Tableau drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about SAS drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

Teradata introduced some enhancements to its Vantage platform last year in which they expanded its analytics functions and language support, and strengthened tools to improve collaboration between data scientists, business analysts, data engineers and business personnel. Some of the key enhancements included expanding the native support for R and Python, extending the ability to execute a wide range of open-source analytics algorithms, and automatic generation of SQL from R and Python code. These updates are included to reduce data silos, enabling a wide range of data and analytics personas to collaboratively run complex analytics in a self-service manner.

We are happy to share some insights about SAP drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about Board International drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about Yellowfin drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about Domo drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about Oracle Analytics Cloud drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about Qlik drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about Information Builders’ WebFOCUS Business Intelligence and Analytics Platform drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about MicroStrategy drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

We are happy to share some insights about IBM drawn from our latest Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements.

Alation recently announced the release of its 2021.1 version, introducing new data governance capabilities, enhancements in search and discovery through data domains, and extended connector and query coverage for data sources. Alation’s new federated authentication enables users to query cloud services such as Amazon Web Services, Snowflake, Tableau and more, using a single sign-on. The release also includes a Search application programming interface that allows for the integration of Alation Search with third-party tools. And, with the addition of the Open Connector Framework software development kit in the 2021.1 update, Alation enables organizations to create connectors for data sources not already supported by Alation.

Everyone talks about data quality, as they should. Our research shows that improving the quality of information is the top benefit of data preparation activities. Data quality efforts are focused on clean data. Yes, clean data is important. but so is bad data. To be more accurate, the original data as recorded by an organization’s various devices and systems is important.

Machine learning is valuable for organizations, but it can be hard to deploy. Our Machine Learning Dynamic Insights research identifies that not having enough skilled resources and difficulty building and maintaining ML systems are pressing challenges organizations face in applying ML. Traditional ML model development is resource-intensive, requiring significant domain knowledge and time to produce and compare dozens of models. And as the number of ML models grow, their management becomes difficult. By bringing automation to ML, organizations can reduce the time it takes to create production-ready ML models. AutoML can also enable organizations to make data science initiatives more accessible across the organization.

The amount of data flowing into organizations is growing exponentially, creating a need to process more data more quickly than ever before. Our Data Preparation Benchmark Research shows that accessing and preparing data continues to be the most time-consuming part of making data available for analysis. This can potentially slow down the organizational functions which depend on the analysis results. Trying to get ahead of the backlog with incremental improvements to existing approaches and traditional technologies alone can be frustrating.

Organizations are becoming more and more data-driven and are looking for ways to accelerate the usage of artificial intelligence and machine learning (AI/ML). Developing and deploying AI/ML models can be complicated in many ways, often involving different tools and services to manage these solutions from end to end. Accessing and preparing data is the most common challenge organizations face in this process, and consequently, AI/ML vendors typically incorporate tools to address this part of the process. But there are many other steps in the process as well, such as coordinating the handoff between data scientists and IT or software engineers for deployment to production. This can potentially slow down the entire data-to-insights process. End-to-end platforms for AI offer the promise of simplifying these processes, allowing teams that work with data to improve organizational results.

Process-mining software isn’t exactly new, but it’s also not widely known in the software technology market. The discipline has been around for at least a decade, but is generating more interest these days with both specialist vendors and major enterprise software vendors offering process-mining products and services. We assert that through 2022, 1 in 4 organizations will look to streamline their operations by exploring process mining.

Organizations are accelerating their digital transformation and looking for innovative ways to engage with customers in this new digital era of data management. The goal is to understand how to manage the growing volume of data in real time, across all sources and platforms, and use it to inform, streamline and transform internal operations. Over the years, the adoption of cloud computing has gained momentum with more and more organizations trying to make use of applications, data, analytics and self-service business intelligence (BI) tools running on top of cloud-computing infrastructure in order to improve efficiency. However, cloud adoption means living with a mix of on-premises and multiple cloud-based systems in a hybrid computing environment. The challenge is to ensure that processes, applications and data can still be integrated across cloud and on-premises systems. Our research shows that organizations still have a significant requirement for on-premises data management but also have a growing requirement for cloud-based capabilities.

Having just completed the 2021 Ventana Research Value Index for Analytics and Data, I want to share some of my observations about how the market has advanced since our assessment two years ago. The analytics software market is quite mature and products from any of the vendors we assess can be used to effectively deliver information to help your organization improve its operations. However, it’s also interesting to see how much the market continues to advance and how much investment vendors continue to make.

I am happy to share insights gleaned from our latest Value Index research, an assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Data 2021 is the distillation of a year of market and product research by Ventana Research. Drawing on our Benchmark Research, we apply a structured methodology built on evaluation categories that reflect the real-world criteria incorporated in a request for proposal to analytics and data vendors supporting the spectrum of business intelligence. Using this methodology, we evaluated vendor submissions in seven categories: five relevant to the product experience ﹘ adaptability, capability, manageability, reliability and usability ﹘ and two related to the customer experience ﹘ TCO/ROI and vendor validation.

Organizations are increasingly using data as a strategic asset, which makes data services critical. Huge volumes of data need to be stored, managed, discovered and analyzed. Cloud computing and storage approaches provide enterprises with various capabilities to store and process their data in third-party data centers. The advent of data platforms previously discussed here are essential for organizations to effectively manage their data assets.

Ventana Research recently announced its 2021 Market Agenda for data, continuing the guidance we have offered for nearly two decades to help organizations derive optimal value and improve business outcomes.

Ventana Research recently announced its 2021 market agenda for Analytics, continuing the guidance we’ve offered for nearly two decades to help organizations derive optimal value from technology investments to improve business outcomes.

The industry is making huge strides with artificial intelligence (AI) and machine learning (ML). There is more data available to analyze. Analytics vendors have made it easier to build and deploy models, and AI/ML is being embedded into many types of applications. Organizations are realizing the value that AI/ML provides and there are now millions of professionals with AI or ML in their title or job description. AI/ML is even being used to make many aspects of itself easier. Organizations that want to build and deploy their own AI/ML models need to be realistic about the capabilities that are available today. As a practical matter, organizations should anticipate that a robust AI/ML deployment in the current environment requires a set of specialized skills and operational processes, including data operations (dataops) and ML operations (MLops). Collaboration across these disciplines and processes is also required.

Data is becoming more valuable and more important to organizations. At the same time, organizations have become more disciplined about the data on which they rely to ensure it is robust, accurate and governed properly. Without data integrity, organizations cannot trust the information produced by their data processes, and will be discouraged from using that data, resulting in inefficiencies and reduced effectiveness.

Organizations are dealing with exponentially increasing data that ranges broadly from customer-generated information, financial transactions, edge-generated data and even operational IT server logs. A combination of complex data lake and data warehouse capabilities are required to leverage this data. Our research shows that nearly three-quarters of organizations deploy both data lakes and data warehouses but are using a variety of approaches which can be cumbersome. A single platform that can provide both capabilities will help address organizations’ requirements.

Businesses are transforming their organizations, building a data culture and deploying sophisticated analytics more broadly than ever. However, the process of using data and analytics is not always easy. The necessary tools are often separate, but our research shows organizations prefer an integrated environment. In our Data Preparation Benchmark Research, we found that 41% of participants use Analytics and Business Intelligence tools for data preparation.

Traditional on-premises data processing solutions have led to a hugely complex and expensive set of data silos where IT spends more time managing the infrastructure than extracting value from the data. Big data architectures have attempted to solve the problem with large pools of cost-effective storage, but in doing so have often created on-premises management and administration challenges. These challenges of acquiring, installing and maintaining large clusters of computing resources gave rise to cloud-based implementations as an alternative. Public cloud is becoming the new center for data as organizations migrate from static on-premises IT architectures to global, dynamic and multi-cloud architectures.

Organizations are always looking to improve their ability to use data and AI to gain meaningful and actionable insights into their operations, services and customer needs. But unlocking value from data requires multiple analytics workloads, data science tools and machine learning algorithms to run against the same diverse data sets. Organizations still struggle with limited data visibility and insufficient insights, which are often caused by a multitude of reasons such as analytic workloads running independently, data spread across multiple data centers, data governance, etc. In our ongoing benchmark research project, we are researching the ways in which organizations work with big data and the challenges they face.

A data lake is a centralized repository designed to house big data in structured, semi-structured and unstructured form. I have been covering the data lake topic for several years and encourage you to check out an earlier perspective called Data Lakes: Safe Way to Swim in Big Data? for background. Our data lake research has uncovered some points to consider in your efforts, and I’d like to offer a deeper dive into our findings.

Every organization performing analytics with multiple employees needs to collaborate. They should be collaborating in the analytics process and in communicating the results of those analyses. As I continue my evaluation of analytics and data vendors, I have to admit some disappointment at the level of collaborative capabilities some analytics vendors provide. To be fair, the level of capabilities vary widely, but I expected collaborative capabilities to be more uniformly available as a standard feature in analytics technologies by now. I had anticipated that three-quarters of analytics vendors would include collaboration capabilities. More than half the vendors I have evaluated support some comments and discussion in their products, only a few have incorporated social recognition and wall posting as part of their collaborative capabilities. So, what impact does a lack of analytics collaboration have on organizations undergoing digital transformation?

For decades, data integration was a rigid process. Data was processed in batches once a month, once a week or once a day. Organizations needed to make sure those processes were completed successfully—and reliably—so they had the data necessary to make informed business decisions. The result was battle-tested integrations that could withstand the test of time.

Ventana Research has been evaluating analytics and business intelligence (BI) software for a long time—almost 20 years. Our methodology for these assessments is referred to as a Value Index. We use weightings derived from our benchmark research about how you, as buyers of these technologies, value and evaluate vendors. You can view our 2019 Value Index results here. I am in the process of completing the 2020 evaluation now.

In this analyst perspective, Dave Menninger takes a look at data lakes. He explains the term “data lake,” describes common use cases and shares his views on some of the latest market trends. He explores the relationship between data warehouses and data lakes and share some of Ventana Research’s findings on the subject. He also provides an assessment of the risks organizations face in working with data lakes and offers recommendations for maximizing the potential of data.

Effectively managing data privacy and security is a high-stakes matter. When an organization doesn’t get it right, it often becomes front-page news and occasionally becomes a subject of litigation. Yet organizations face an equally challenging imperative to ensure that business users have easy access to the data they need. Depending on how they are implemented, data governance policies can inhibit access to data, making it harder to find and utilize the data assets of an organization.

Artificial intelligence (AI) and machine learning (ML) are all the rage right now. Our Machine Learning Dynamic Insights research shows that organizations are using these techniques to achieve a competitive advantage and improve both customer experiences and their bottom line. One type of analysis an organization can perform using AI and ML is predictive analytics. Organizations also need to plan their operations to predict the amount of cash they will need, inventory levels and staffing requirements. Unfortunately, while planning begins with predictions, organizations can’t plan with AI and ML. Let me explain what I mean.

I was recently asked to identify key modern data architecture trends. Data architectures have changed significantly to accommodate larger volumes of data as well as new types of data such as streaming and unstructured data. Here are some of the trends I see continuing to impact data architectures.

MicroStrategy recently held its annual user conference, which focused on the theme of the “Intelligent Enterprise.” HyperIntelligence, an innovative product for delivering analytics throughout organizations that they introduced a year ago, was the star of the event. The company announced enhancements to HyperIntelligence and the latest version of its flagship platform, MicroStrategy 2020, as well as a new two-tiered education and certification program.

Ventana Research recently announced its 2020 research agenda for data, continuing the guidance we’ve offered for nearly two decades to help organizations derive optimal value and improve business outcomes. Data volumes continue to grow while data latency requirements continue to shrink. Meanwhile, virtually every organization is confronting a need for good data governance.

Ventana Research recently announced its 2020 research agenda for analytics, continuing the guidance we’ve offered for nearly two decades to help organizations derive optimal value from their technology investments and improve business outcomes.

It’s been exciting to follow the emergence of innovative capabilities in the analytics market, but for businesses it can be challenging to stay on top of all these changes. To help, we craft our research agenda using our firm’s knowledge of technology vendors and products and our experience with and expertise on business requirements.

The Oracle Analytics Summit 2019 was the inaugural user event for Oracle Analytics customers, and they also broadcast the video for thousands of others. You can watch the keynote at https://www.youtube.com/watch?v=eY0IPNqzsy4. Executives talked about some big organizational changes, including Bruno Aziza joining last year to lead the analytics organization. This event marked a transition and "a new beginning" for the Oracle Analytics portfolio, as the company announced three new analytics products.

Organizations’ use of data and information is evolving as the amount of data and the frequency with which that data is collected increase. Data now streams into organizations from myriad sources, among them social media feeds and internet-of-things devices. These seemingly ever-increasing volumes of devices and data streams offer both challenges and opportunities to capture information about a business and improve its operations.

Alteryx Inspire 2019, this year's user conference for Alteryx, drew around 4500 customers, partners, and prospects to Nashville’s Gaylord Opryland Resort & Convention Center in Tennessee last month. The strong attendance was a reflection of the strong growth Alteryx has experienced over the last year; roughly 50% growth year-over-year. This year's conference focused on Alteryx's evolution from data preparation to AI and machine learning, and both were front and center.

Summit 2019, Information Builders' annual user conference, drew about 1000 attendees this year, including customers, partners and prospects all working with Information Builders' technologies. Under new leadership, Summit 2019 showcased the direction Information Builders is moving in the next couple of years.

This year, I attended Informatica World 2019, Informatica's annual user conference. The main focus this year was on the cloud with a heavy does of AI. Under that focus, Informatica's conference emphasized capabilities across six areas (all strong areas for Informatica): data integration, data management, data quality & governance, Master Data Management (MDM), data cataloging, and data security. 

Qonnections 2019 is Qlik's annual user conference. Key news from this year's conference centered on acquisitions of Podium Data and Attunity, along with an expansion of certifications on Google Cloud Platform, AWS, and Azure, with the ability to support Red Hat OpenShift. Many of these announcements were centered on a key theme of a cloud and SaaS-first approach.

The emerging internet of things (IoT) is an extension of digital connectivity to devices and sensors in homes, businesses, vehicles and potentially almost anywhere. This innovation means that virtually any appropriately designed device can generate and transmit data about its operations, which can facilitate monitoring and a range of automatic functions. To do this IoT requires a set of event-centered information and analytic processes that enable people to use that event information to make optimal decisions and take act effectively.

Organizations now must store, process and use data of significantly greater volume and variety than in the past. These factors plus the velocity of data today — the unrelentingly rapid rate at which it is generated, both in enterprise systems and on the internet — add to the challenge of getting the data into a form that can be used for business tasks.

I am happy to share some insights gleaned from our latest Value Index research, which provides our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Collaborative Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. Drawing on our benchmark research and expertise, we apply a structured research methodology built on evaluation categories that are designed to reflect the real-world criteria incorporated in a request for proposal to vendors in analytics and business intelligence. Using this methodology, we evaluated vendor submissions in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). This research-based index is the first such evaluation to assess the full business value of collaborative analytics and business intelligence software. You can learn more about our Value Index as an effective vendor selection and RFI/RFP tool at https://www.ventanaresearch.com/value-indexes.

Domopalooza 2019 marked the first annual user conference after Domo went public, but the energy, excitement and new feature announcements have not slowed. With thousands in attendance and growing fast, this year's conference focused on five key areas: digitization, real time connectivity, driving insight based actions, applying AI & machine learning, and building applications. All of these announcements are aimed at broadening the workloads supported by Domo.

About 10 years ago, social media tools like Facebook, Twitter and LinkedIn introduced a wave of collaborative analytics and BI capabilities. We saw chat streams associated with specific analyses that users could like or endorse. The number of contributions a user made to the community was part of his or her profile so others could accordingly weigh the importance of the input.

I am happy to offer some insights on Qlik drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated Qlik and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to offer some insights on Microsoft drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated Microsoft and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to offer some insights on Looker drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated Looker and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to offer some insights on Tableau drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated Tableau and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to offer some insights on Oracle drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated Oracle and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to share some insights gleaned from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Embedded Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. Drawing on our benchmark research and expertise, we apply a structured research methodology built on evaluation categories that are designed to reflect the real-world criteria incorporated in a request for proposal to vendors in analytics and business intelligence. Using this methodology, we evaluated vendor submissions in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). This research-based index is the first such evaluation to assess the full business value of embedded analytics and business intelligence software. You can learn more about our Value Index as an effective vendor selection and RFI/RFP tool at https://www.ventanaresearch.com/value-indexes.

I am happy to offer some insights on Salesforce.com drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated Salesforce.com and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

Analytics and business intelligence (BI) play an instrumental role in enabling an organization’s business units and IT to utilize its data in both tactical and strategic ways to perform optimally. To accomplish this, organizations must not only access the data, generate and apply insights from analytics, and communicate the results, they also must ensure that the analytics are presented in a way that leads to action. One of the most effective ways to do this is to embed analytics into business processes and applications.

I am happy to share some insights gleaned from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Mobile Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. Drawing on our benchmark research and expertise, we apply a structured research methodology built on evaluation categories that are designed to reflect the real-world criteria incorporated in a request for proposal to vendors in analytics and business intelligence. Using this methodology, we evaluated vendor submissions in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). This research-based index is the first such evaluation to assess the full business value of analytics and business intelligence software. You can learn more about our Value Index as an effective vendor selection and RFI/RFP tool at https://www.ventanaresearch.com/value-indexes.

For analytics to be effective, they need to be available to line-of-business personnel as needed in their normal course of conducting business, which today means providing rich mobile access to analytics through phones and tablets to support a mobile workforce seeking to conduct business in any location at any time. Workers today expect these mobile capabilities, which means organizations must make choices to provide analytics and BI platforms that can deliver them.

I am happy to offer some insights on SAS drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated SAS and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to offer some insights on Yellowfin drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated Yellowfin and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to share some insight on Domo drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements.  The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated Domo and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

MicroStrategy recently held their annual user conference, MicroStrategy World 2019. This year's conference brought 2,100 customer attendees plus partners to the Phoenix Convention Center in Phoenix, AZ. The big news of the event was the introduction of MicroStrategy HyperIntelligence™, a platform tool designed to directly inject analytics into business applications.

I am happy to offer some insights on SAP drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated SAP and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to share some insight on BOARD drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated BOARD and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to share some insights on Infor based on our latest market Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated Infor and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to share some insights about IBM drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated IBM and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to offer some insights on MicroStrategy drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence. We evaluated MicroStrategy and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to offer some insights on Information Builders drawn from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. We utilized a structured research methodology that includes evaluation categories designed to reflect the breadth of the real-world criteria incorporated in a request for proposal (RFP) and vendor selection process for analytics and business intelligence software. We evaluated Information Builders and 14 other vendors in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). To arrive at the Value Index rating for a given vendor, we weighted each category to reflect its relative importance in an RFP process, with the weightings based on our experience and data derived from our benchmark research on analytics and business intelligence.

I am happy to share some insights gleaned from our latest Value Index research, which provides an analytic representation of our assessment of how well vendors’ offerings meet buyers’ requirements. The Ventana Research Value Index: Analytics and Business Intelligence 2019 is the distillation of a year of market and product research efforts by Ventana Research. Drawing on our benchmark research, we apply a structured research methodology built on evaluation categories that are designed to reflect the real-world criteria incorporated in a request for proposal to vendors in analytics and business intelligence. Using this methodology, we evaluated vendor submissions in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). This research-based index is the first such evaluation to assess the full business value of analytics and business intelligence software. You can learn more about our Value Index as an effective vendor selection and RFI/RFP tool at https://www.ventanaresearch.com/value-indexes/inclusion.

Ventana Research provides unique insight into the analytics and business intelligence (BI) industry. This is important, as its processes and technology play an instrumental role in enabling an organization’s business units and IT to utilize its data in both tactical and strategic ways to perform optimally. To accomplish this, organizations must provide technology that can access the data, generate and apply insights from analytics, communicate the results and support collaboration as needed.

IBM's Analytics University (held in both Miami and Stockholm) brought about some large changes. Big announcements this year included a consolidation of IBM's Watson Analytics into Cognos 11.1, helping provide some clarity to their analytics offerings, along with new visualizations and better data preparation. This also includes a new conversational assistant to help generate narrative explanations of displays and interactive queries. For the full breakdown of IBM's Analytics University 2018, and my analysis of all the largest announcements, watch my latest hot take.

Once again I attended Tableau's Users Conference, along with 17,000 other attendees, affectionately self-referred to as "data nerds". Pushing the envelope in data capabilities and access, Tableau introduced the "Ask Data" feature, allowing users to prose natural language queries and receive a response, along with new data preparation capabilities and other enhancements to help data analysts. Further, Tableau announced new developer enhancements including a new developer program to better align tools built for Tableau with Tableau's interface. For the full breakdown of Tableau User Conference 2018, and my analysis of all the largest announcements, watch my hot take video.

This year, Teradata rebranded the Teradata users conference from "Partners" to "Analytics Universe", and there is a reason for it. For decades, Teradata has represented the high end of the analytic database, but new innovations and technologies are adding flexibility to Teradata's licensing as they compete. For the full breakdown of Teradata's Analytics Universe 2018, and my analysis of all the largest announcements, watch my hot take video.

In 2017 Strata + Hadoop World was changed to the Strata Data Conference. As I pointed out in my coverage of last year’s event, the focus was largely on machine learning and artificial intelligence (AI). That theme continued this year, but my impression of the event was of a community looking to get value out of data regardless of the technology being used to manage that data. The change was subtle: The location was the same; the exhibitors were largely the same; attendance was similar this year and last. But there was no particular vendor or technology dominating the event.

All too often, software vendors view analytics as the end rather than the beginning of a process. I’m reminded of some of the advanced math classes I’ve taken in which the teaching process focused on a few key aspects of a mathematical proof or solution, leaving the rest of the exercise to be worked out by the students. In other contexts, you may hear people say the numbers speak for themselves.

We at Ventana Research recently published our research agendas for 2018. The world of data and information management continues to evolve, as does our research on the use of these technologies to improve your organization’s operations. Relational databases are no longer the only viable enterprise data store as more organizations adopt a polyglot database infrastructure. And while their exact form may still be changing, as I have recently written, big data technologies are here to stay. Our Data and Analytics in the Cloud Benchmark Research indicates that an increasing number of organizations are opting for cloud-based deployments: A modern data infrastructure includes a hybrid of on-premises and cloud deployments for 44 percent of organizations. Our upcoming research will track how these changes are affecting data- and information-management processes.

We at Ventana Research recently published our research agendas for 2018. Analytics and business intelligence are evolving and so is our research on their use across practice areas. Earlier research has shown that analytics can deliver significant value to organizations; for example, our predictive analytics research shows that 57 percent of organizations reported achieving a competitive advantage and half created new revenue opportunities with predictive analytics. Waves of investment in self-service analytics have propelled the market for analytics tools, significantly empowering line-of-business organizations to create their own analytics and set their own analytic priorities. But organizations are also beginning to recognize some of the limitations of current analytics implementations – for self-service, for example. Our Data Preparation Benchmark Research reveals that fewer than half (42%) of organizations are comfortable allowing business users to work with data not prepared by IT. Our research this year will continue to explore both the successes and challenges organizations face as they continue to use analytics and BI.

Ventana Research recently published the findings of our benchmark research on Data Preparation, which examines the practices organizations use to accomplish data preparation. We view data preparation as a sequence of steps: identifying, locating and then accessing the data; aggregating data from different sources; and enriching, transforming and cleaning it to create a single uniform data set. Using data to accomplish organizational goals requires that it be prepared for use; to do this job properly, businesses need flexible tools that enable them to enrich the context of data drawn from multiple sources and collaborate on its preparation as well as ensure security and consistency. Users of data preparation tools range from analysts to operations professionals in the lines of business to IT professionals.

I recently attended SAP TechEd in Las Vegas to hear the latest from the company regarding its analytics and business intelligence offerings as well as its data management platform. The company used the event to launch SAP Data Hub and made several other data and analytics announcements that I’ll cover below.

The Strata Data Conference is changing and it’s changing in a good way. At the recent Strata Data Conference in New York, Mike Olson, chief strategy officer at Cloudera, which co-sponsored the event, commented that at prior events we used to talk about the “Hadoop zoo animals,” meaning the various components of the Hadoop ecosystem of which I have written previously. Following last fall’s Strata event, I observed that the conference was evolving to focus on the use of data. Advancing that evolution, this year’s event focused on a particular type of usage: artificial intelligence (AI) and machine learning. The evolution from a focus on zoo animals to a focus on business value using advanced analytics shows further maturation of the big data market.

There’s been some speculation in the market that Hadoop may be disappearing. Some of this speculation has been driven by vendors that have recently downplayed Hadoop in their marketing efforts. For example, the Strata+Hadoop World conference is now known as the Strata Data Conference. The Hadoop Summit is now known as the Dataworks Summit. In Cloudera’s S-1 filing with the SEC for its initial public offering, the term “Hadoop” appears only 14 times, while the term “machine learning” appears 83 times. So, if some of the vendors that created the market appear to be pivoting away from Hadoop, does your organization need to do something similar, or is there a role for Hadoop in your IT architecture?

Recently Hortonworks announced some significant additions to its products at the DataWorks Summit. These additions reflect the fact that the big data market continues to evolve, as I have previously written.

Natural language generation (NLG), the process of generating text or narratives based on a set of data values, can reach a broader audience. NLG narratives can be used for a variety of purposes, but in this perspective I focus on how NLG can be used to enhance business intelligence (BI) processes. In the case of BI, NLG can be used to explain what has happened and why it is happening, and even what actions to take. The NLG narratives can be understood by a broader range of business users than the tables and charts of data that are the typical output of most BI applications or analytics tools.

Many organizations continue to struggle with preparing data for use in operational and analytical processes. We see these issues reported in our Data and Analytics in the Cloud benchmark research, where 55 percent of organizations identify data preparation as the most time-consuming task in their analytical processes.  Similarly, in our Next-Generation Predictive Analytics research, 62 percent of companies report that they’re unsatisfied because data needed for access or integration is not readily available. In our Big Data Integration research, 52 percent report spending that in working with big data integration processes, they spend the most time reviewing data for quality and consistency.  And nearly half of companies (48%) report this same issue in our Internet of Things research. We are currently conducting further research into this critical issue with our Data Preparation benchmark research.

This is my second analyst perspective based on our IoT Benchmark Research. In the first, I discussed the business focus of IoT applications and some of the challenges organizations are facing. Now I’ll share some of the findings about technologies used in IoT applications and the impact those technologies appear to have on the success of users’ projects.

This year various types of organizations are embracing machine learning like it is going out of style – or maybe it would be better to say coming into style. And now with a little investigation on LinkedIn finds over half million professionals with machine learning in their job title. Machine learning is the application of specific data science algorithms that become more accurate as the system records more outcomes and processes more data. This improvement is referred to as “learning,” hence the name. There are good reasons machine learning is growing so rapidly, but there are pitfalls to avoid as well.

Informatica reintroduced itself to the world at its recent customer conference, Informatica World, in San Francisco. The company took advantage of the event to showcase its new branding in an effort to change the way customers think about the company. Informatica has been providing information services in the cloud for more than a decade. Even though cloud revenue comprises a minority of Informatica’s business, in absolute terms, the revenue is significant, and company executives want the public to recognize Informatica as a leader in cloud-based data management services for enterprises. Presenters also made notable product announcements, discussed below, including the application of machine learning to the data management process.

I recently attended the MicroStrategy World conference, which was held in Washington, D.C. and it celebrate its 20th anniversary, which is why MicroStrategy hosted the event near its headquarters. Over the past 20 years, the market and technology for business intelligence and analytics have significantly changed, and in several changes, MicroStrategy has been at the forefront. Now is a good time to examine the company’s position in the market and its latest offerings in context of the analytics market direction that I recently presented.

Some 3,000 people attended Domo’s recent customer event, called Domopalooza. That’s nearly double the attendance of the previous event, which my colleague Mark Smith covered. Formerly a bit “stealthy,” Domo has started sharing more information, some of which I’ll pass along, as well as observations about product announcements made at the event.

I recently attended SAS Institute’s analyst relations conference. There the company provided updates on its financial performance and its Viya platform and a glimpse into some of its future plans.

The Internet of Things (IoT) is a technology that extends digital connectivity to devices and sensors in homes, businesses, vehicles and potentially almost anywhere. This advance enables virtually any device to transmit its data, to which analytics can then be applied to facilitate monitoring and a range of operational functions. IoT can deliver value in several ways. It can provide organizations with more complete data about their operations, which helps them improve efficiencies and so reduce costs. It also can deliver a competitive advantage by enabling them to reduce the elapsed time between an event occurring and operational responses, actions taken or decisions made in response to it.

Big data initially was characterized in terms of “the three V’s,” volume, velocity and variety. Nearly five years ago I wrote about the three V’s as a way to explain why new and different technologies were needed to deal with big data. Since then the industry has tackled many of the technical challenges associated with the three V’s. In 2017 I propose that we focus instead on a different letter, which includes these A’s: analytics, awareness, anticipation and action. I’ll explain why each is important at this stage of big data evolution.

Big data has become an integral part of information management. Nearly all organizations have some need to access big data sources and produce actionable information for decision-makers. Recognizing this connection, we merged these two topics when we put together our recently published research agendas for 2017. As we plan our research, we focus on current technologies and how they can be used to improve an organization’s performance. We then share those results with our readers.

Ventana Research analysts recently published our research agendas for 2017. As we put together these plans we think about the forces that are shaping the markets that we cover and then craft agendas that study these issues to provide insights for our community. I’ve been working in the business intelligence (BI) and analytics market for nearly 25 years, and throughout that time the industry has been trying to make analytics useful to increasingly wider audiences. That focus continues to today. Better search and presentation methods, including visual discovery and natural-language processing, are promising ways to engage more users. We also see organizations supporting their users in specific functional roles with relevant and accessible analytics. My colleagues examine these issues as part of their agendas in the Office of Finance, Sales, Marketing, Customer Experience, Operations and Supply Chain, and Human Capital Management. While their agendas include analytics within specific domains, my own research focuses on a range of analytics issues across domains including cloud computing, mobility, collaboration, data science and the Internet of Things.

The business intelligence market is bounded on one side by big data and on the other side by data preparation. That is, to maximize their performance in using information, organizations have to collect and analyze ever increasing volumes of data while the tools available are constantly evolving in the big data ecosystem that I have written about. In our benchmark research on big data analytics, half (51%) of organizations said they want to access big data using their existing BI tools. At the same time, as I have noted, end users are demanding self-service access to data preparation capabilities to facilitate their analyses.

The big data market continues to evolve, as I have written previously. Vendors are attempting to differentiate their offerings as they seek to encourage customers to pay for technology that they could potentially download for free.

IBM recently held its inaugural World of Watson event. Formerly known as IBM Insight, and prior to that IBM Information on Demand, the annual event, attended by 17,000 people this year, showcases IBM’s data and analytics and the broader IBM efforts in cognitive computing. The theme for the event, as you might guess, was the Watson family of cognitive computing products. I, for one, was glad to spend more time getting to know the Watson product line, and I’d like to share some of my observations from the event.

More than 13,000 self-described “data and visualization nerds” gathered in Austin, TX, recently for Tableau Software’s annual customer conference. In his inaugural keynote, Tableau’s new CEO, Adam Selipsky, said that nearly 9,000 were first-time attendees. I was impressed with the enthusiasm of the customers who had gathered for the event, cheering as company officials reviewed product plans and demonstrated new features. This enthusiasm suggests Tableau has provided capabilities that resonate with its users. Among other things, the company used the conference to outline a number of planned product enhancements.

Fall is a busy time for software industry analysts. It’s a season filled with vendors’ user conferences and some industry conferences. Throughout the course of attending these events I’ve come to the realization that big vendors are often considered the Rodney Dangerfield of the software industry: They get no respect. What I mean by no respect is revealed in snarky social media comments, less enthusiastic coverage by tech media than smaller vendors get and a general sense that big vendors don’t do anything new with their development efforts. However, I suggest this is a shortsighted view of the software world. Smaller vendors serve a valuable function as a source of innovation for the industry, but they get a disproportionate share of attention. I suggest the big vendors deserve businesses’ attention, too, when they consider new software purchases.

Data preparation is critical to the effectiveness of both operational and analytic business processes. Operational processes today are fed by streams of constantly generated data. Our data and analytics in the cloud benchmark research shows that more than half (55%) of organizations spend the most time in their analytic processes preparing data for analysis – a situation that reduces their productivity. Data now comes from more sources than ever, at a faster pace and in a dizzying array of formats; it often contains inconsistencies in both structure and content.

I recently attended .conf2016, Splunk’s seventh annual user conference. Splunk created the market for analyzing machine data (shorthand for machine-generated data), which consists of log files and event data from various types of systems and devices. Our big data analytics benchmark research shows that these are two of the most common sources of big data that organizations analyze. This market has proven to be fertile ground for Splunk, growing steadily with revenues more than doubling over the previous two fiscal years. Machine data is also the backbone for the Internet of Things (IoT) and operational intelligence, which form the basis of forthcoming benchmark research from Ventana Research.

I recently spent time at Strata+Hadoop World 2016 in New York. I attended this event and its predecessor, Hadoop World, off and on for the past six years. This one in New York had a different feel from previous events including the most recent event in San Jose at the end of March. Perhaps because of its location in one of the financial and commercial hubs of the world, the event had much more of a business orientation. But it’s not just location. Past events have been held in New York also, and I see the business focus as a sign of the Hadoop market maturing.

I recently attended Oracle OpenWorld for the first time in several years. The message at this year’s event was clear: Oracle is all in on the cloud. I had heard the message, but I didn’t get the full impact until I arrived at the Moscone Center in San Francisco. All signage at the event contained the word “cloud,” and Oracle issued 18 press releases in conjunction with OpenWorld related to cloud computing. I also found out that Oracle has its own definition of “cloud.”

Teradata recently held its annual Partners conference, at which gather several thousand customers and partners from around the world. This was the first Partners event since Vic Lund was appointed president and CEO in May. Year on year, Teradata’s revenues are down about 5 percent, which likely prompted some changes at the company. Over the past few years Teradata made several technology acquisitions and perhaps spread its resources too thin. At the event, Lund committed the company to a focus on customers, which was a significant part of Teradata’s success in the past. This commitment was well received by customers I spoke with at the event.

Ventana Research has newly published its Mobile Analytics and Business Intelligence 2016 Value Index. The Value Index provides a comprehensive evaluation of vendors and their product offerings across seven categories. In performing that analysis, I realized that this software category is at a crossroads. Once an optional capability often reserved for executives, mobile analytics is becoming a requirement of business users across organizations. The blurring of lines between work and personal lives has provoked a change from single device BI to BI on multiple devices including smartphones and tablets as well as laptops and desktops. From a platform standpoint, the adoption of HTML5 is contesting the prevalence of native mobile applications.

It’s part of my job to cover the ecosystem of Hadoop, the open source big data technology, but sometimes it makes my head spin. If this is not your primary job, how can you possibly keep up? I hope that a discussion of what I’ve found to be most important will help those who don’t have the time and energy to devote to this wide-ranging topic.

Data virtualization is not new, but it has changed over the years. The term describes a process of combining data on the fly from multiple sources rather than copying that data into a common repository such as a data warehouse or a data lake, which I have written about. There are many reasons for an organization concerned with managing its data to consider data virtualization, most stemming from the fact that the data does not have to be copied to a new location. It could, for instance, eliminate the cost of building and maintaining a copy of one of the organization’s  big data sources. Recognizing these benefits, many database and data integration companies offer data virtualization products. Denodo, one of the few independent, best-of-breed vendors in this market today, brings these capabilities to big data sources and data lakes.

Predictive analytics is a rewarding yet challenging subject. In our benchmark research on next-generation predictive analytics at least half the participants reported that predictive analytics allows them to achieve competitive advantage (57%) and create new revenue opportunities (50%). Yet even more participants said that users of predictive analytics don’t have enough skills training to produce their own analyses (79%) and don’t understand the mathematics involved (66%). (In the term “predictive analytics” I include all types of data science, not just one particular type of analysis.)

Qlik helped pioneer the visual discovery market with its QlikView product. In some respects, Qlik and its competitors also spawned the self-service trend rippling through the analytics market today. Their aim was to enable business users to perform analytics for themselves rather than building a product with the perfect set of features for IT. After establishing success with end users the company began to address more of the concerns of IT, eventually creating a robust enterprise-grade analytics platform. This approach has worked for Qlik, driving growth that led to an initial public offering in 2010. The company now generates more than half a billion dollars in revenue annually, making it one of the largest independent analytics vendors. Of which based on their company and products was rated a Hot Vendor in our 2015 Value Index on Analytics and Business Intelligence and one of the highest ranked in usability.

It has been more than five years since James Dixon of Pentaho coined the term “data lake.” His original post suggests, “If you think of a data mart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state.” The analogy is a simple one, but in my experience talking with many end users there is still mystery surrounding the concept. In this post I’d like to clarify what a data lake is, review the reasons an organization might consider using one and the challenges they present, and outline some developments in software tools that support data lakes.

The emerging Internet of Things (IoT) is an extension of digital connectivity to devices and sensors in homes, businesses, vehicles and potentially almost anywhere. This innovation means that virtually any device can generate and transmit data about its operations – data to which analytics can be applied to facilitate monitoring and a range of automatic functions. To do these tasks IoT requires what Ventana Research calls operational intelligence (OI), a discipline that has evolved from the capture and analysis of instrumentation, networking and machine-to-machine interactions of many types. We define operational intelligence as a set of event-centered information and analytic processes operating across an organization that enable people to use that event information to take effective actions and make optimal decisions. Ventana Research first began covering operational intelligence over a decade ago.

On Monday, March 21, Informatica, a vendor of information management software, announced Big Data Management version 10.1. My colleague Mark Smith covered the introduction of v. 10.0 late last year, along with Informatica’s expansion from data integration to broader data management. Informatica’s Big Data Management 10.1 release offers new capabilities, including for the hot topic of self-service data preparation for Hadoop, which Informatica is calling Intelligent Data Lake. The term “data lake” describes large collections of detailed data from across an organization, often stored in Hadoop. With this release Informatica seeks to add more enterprise capabilities to data lake implementations.

I recently attended the SAS Analyst Summit in Steamboat Springs, Colo. (Twitter Hashtag #SASSB) The event offers an occasion for the company to discuss its direction and to assess its strengths and potential weaknesses. SAS is privately held, so customers and prospects cannot subject its performance to the same level of scrutiny as public companies, and thus events like this one provide a valuable source of additional information.

Last week I attended Spark Summit East 2016 at the New York Hilton Midtown. It revealed several ways in which Spark technology might impact the big data market.

The big data market continues to expand and enable new types of analyses, new business models and new revenues streams for organizations that implement these capabilities. Following our previous research into big data and information optimization, we’ll investigate the technology trends affecting both of these domains as part of our 2016 research agenda.