Metafor Software CTO talks about challenges in anomaly detection

Metafor Software's CTO discusses anomaly detection and why he believes the approaches in use today lead to too many false positives and let real issues go through without being detected.

Metafor Software's founder and CTO, Dr. Toufic Boubez, spent some time with me recently discussing the challenges performance monitoring and management suppliers have in detecting performance anomalies, learning the root cause and offering a real-time, scalable service enterprises can rely on.

What he has to say makes a great deal of sense and explains what I've seen in the industry quite well.

Who is Metafor Software?

The company describes itself and its mission in the following way:

"Metafor’s machine learning technology analyzes the streams of real-time data generated by your IT infrastructure and applications to accurately identify anomalous behavior so you can fix problems at the first sign of trouble. Metafor automatically establishes what normal behavior looks like and uses advanced unsupervised, non-parametric behavioral analytics to identify activity indicative of security threats and impending performance problems."

It appears to me that they have looked at the performance monitoring problem that everyone sees and have thought a bit differently about it. This is likely to lead to solutions that are a bit different as well.

Correlation of events is the challenge

Metafor Software sees a problem with today's approaches to gathering operational data from many different IT resources, cleansing and formatting the data in real time so that it can be analyzed for potential and real problems. Also how this data is correlated and some basic assumptions about the data has led to performance monitoring and management solutions producing too many false positives and allowing some real issues to go undetected.

Metafor Software has started over with a fresh viewpoint. While Dr. Boubez wouldn't disclose technical descriptions of much of his company's technology, he has presented his views in a video. It can be found here.

What is Metafor doing that is different from all of the others?

Although the complete details of what Metafor's software does are not yet clear, Dr. Boubez gave a few clues including the following:

  • Thresholds, or known state descriptions, are often a problem. Where and how they are set often produce problems in observing systems in operation. When the setting is too low, IT administrators are inundated with alerts that often point out known problems or acceptable levels of performance. When the setting is too high, real problems are not detected until a failure occurs.

  • They are based upon several fallacies including the following:

    • Underlying systems are static — an obvious fallacy in the world of highly virtualized or cloud-based resources. 

    • The limits of system behavior can be defined by static rules and thresholds, regardless of whether those rules have been set by IT administrators or were generated through some form of machine learning.

Snapshot analysis

I really enjoy speaking with representatives of technology start-ups, such as Metafor Software. They are often addressing issues that have fallen through the cracks and that others have not yet addressed.

My conversation with Dr. Boubez was fast paced and very interesting. He's clearly a man on a mission. I'm hoping to speak with an enterprise that uses Metafor's technology to learn more. Until then, we'll all have to watch and wait.