Data quality can make or break efforts to bring artificial intelligence to IT operations

AIOps, or artificial intelligence for IT operations, may be just what the doctor ordered for beleaguered IT shops. Applying advanced automation to countless rote IT functions will free up IT departments to concentrate on the bigger and more meaningful things, such as digital transformation and promoting continuous integration and deployment of software.

However, there's a problem: AIOps requires the right kind of data at the right time, but much of this data either isn't ready or needs a quality overhaul. While AIOps functions on data points such as system logs and metrics, historical performance, event data, streaming real-time operations events, incident-related data, and ticketing, much of this data may be incomplete or hidden away in silos. In short, if data isn't up to par, AIOps may flop, or worse yet, steer technology decisions in the wrong direction.  

Enter an emerging methodology on the scene that specifically addresses this, known as robotic data automation, or RDA, as identified in a Forbes piece by Shailesh Manjrekar. While its close cousin, robotic process automation (RPA), automates business processes, data workflows, and user tasks, RDA focuses on automating data pipelines with bots.

Bringing RDA in to support AIOps was the gist of a recent webcast, in which Valerie O'Connell, research director at Enterprise Management Associates (EMA), joined Bhaskar Krishnamsetty, chief product officer at CloudFabrix, to make the case for this new approach.

While RDA has potential to increase the availability and quality of data available to AI in all forms of business applications, the panelists focused on its impact on IT operations themselves. The forms of automation that are supported through AIOps, as found in a recent EMA survey, include "workflow across IT" is the most oft-cited use case (60%), followed by "runbook or IT process automation," adopted within about half of AIOps scenarios (49%). Another 43% turn to AIOps for more intelligent alert-driven notifications.

The IT managers surveyed see value delivered as a result of AIOps – 62% rated its value as high to very high, O'Connell says. AIOps helps improve IT/business alignment, the quality of IT and business services, and the end-user and customer experience. 

However, AIOps is difficult to implement, she continues. "Most people found it challenging. The benefits and the gains are almost guaranteed, but equally almost without exception it is going to be complex and difficult." The primary challenges include data accuracy or accessibility, conflicts within IT, fear or distrust of AI, and skills availability.

O'Connell zeroed in on the data difficulties associated with AIOps. "Data is either going to be an obstacle or a shortcut to get to your AIOps benefits," she says. "If you can get a good handle on your data and your data issues, then you directly take a strike against the complexity of AIOps implementation," she explains. 

The success of AIOps is inexorably tied to "data, data, data, and how well you can handle and process the data," Krishnamsetty agrees. One of the most vexing issues is data access and acquisition, he points out. "You want to pull data from your AWS environment, or your application performance monitoring tools, or your log analytics tool. But all this data is in different formats."

RDA addresses the data challenges associated with AIOps, Krishnamsetty continues. "If you don't have the proper data, it's garbage-in, garbage-out. However, powerful your machine learning algorithms are, if your data quality is poor, you are not going to get good insights and analytics."

For example, "if you look at any raw alerts coming from any of your management or monitoring systems, you will know how sparse the data is," he illustrates. "A human can't make a quick decision on it unless it is automatically enriched. The data is incomplete. What application, what infrastructure, and so forth."

RDA also helps address the skills gap, which is in short supply for assuring the quality of data that is fed into AI systems, he continues. "Unless a platform provides out-of-box automated data operations capabilities, you have to depend on expensive data engineers or data scientists." No-code and low-code platforms that provide self-service capabilities to citizen developers is an important trend for 2022.

Data observability is often dependent on how many people can be thrown at assuring quality in the data pipeline, either through hiring more staff or engaging with consulting firms, he points out. "Our customers sometimes have to put a lot of the data engineers to do a lot of data exploration, data preparation, and data enrichment manually. It increases total cost of ownership, it increases time to value. That's where a lot of AIOps implementations are falling short."

With RDA, software bots can be deployed within data pipelines to "simplify and abstract a lot of data operations and data or machine learning operations," he says. This is the key to data automation. "By using software bots within pipelines and automated workflows, you can achieve data quality for AIOps."