Logz.io moves toward application observability in the cloud, raises questions on open source

Logz.io is an interesting vendor. Founded in 2014, it has managed to raise about $100 million and build a team of 250 people to date, on the premise of offering an open-source based solution for log management. 

Today in AWS reInvent, Logz.io announced the latest addition to its portfolio: a cloud observability platform, that correlates metrics and logs to speed up investigative work and time to resolution. The platform is based on Grafana and is currently available in Beta.

ZDNet caught up with Logz.io CEO, Tomer Levy, to discuss the new platform, business strategy, and open source.

Logz.io moves toward application observability in the cloud

Logz.io calls its cloud observability platform the culmination of its three product offerings. The company's first product was Log Management. The idea was to provide the ELK Stack (Elastic-Logstash-Kibana) as a fully managed service and enhance it with advanced data analytics features to increase developer productivity and decrease time to resolution. 

From there, the company went on to offer Cloud SIEM (Security Information and Event Management). Cloud SIEM aims to provide simple DevOps-native threat detection and analytics built solution, again built on the ELK Stack. In other words, Logz.io went from offering a platform, to offering a domain-specific solution on that platform. 

Today's offering goes one step further in the platform direction. So far, Logz.io was able to ingest and offer services on data contained in logs. Infrastructure Monitoring adds metrics to the mix. 

Levy's contention is that observability is layered on 3 pillars: monitoring, troubleshooting, and security. Logz.io took a progressive approach to addressing these pillars. It started with building the infrastructure to run ELK as a managed service. Then it added integration capabilities to ingest data from various sources, such as databases and servers.

Subsequently, the analytics capabilities were developed, including machine learning. Then, a SIEM solution was built on this foundation - essentially, sophisticated pattern matching, integrating data from various sources. Now, going beyond logs, application metrics are added too.

Logz.io says this enables complete visibility into Kubernetes and distributed cloud workloads. One of the features Levy emphasized was root cause analysis. Although Logz.io does not offer this as an out of the box functionality, Levy said it provides meaningful alerts to users, enabling them to correlate data across the board and investigate.

Built on open-source best of breed

The new platform utilizes Grafana, which is interesting for several reasons. As engineering teams build and ship code faster, they employ technologies such as Kubernetes and serverless, resulting in application stacks that are distributed, abstracted, and difficult to monitor.

As a result, achieving observability in modern IT environments has become cumbersome, and time-consuming. To solve this issue, engineers prefer open-source tools, because they are accessible, easy to set up, community-driven, and purpose-built. Also, they are cloud-native and easy to integrate with modern infrastructure.

The above is a near-verbatim excerpt of Logz.io's press release, yet few people would disagree with that. Time and again we have referred to the fact that open source is becoming the new normal, also in enterprise software. Logz.io chose to build on ELK and Grafana not just because they were available, but also because it wanted to capitalize on them. 

Levy said that people find Logz.io easy to use because it's built on platforms they are familiar with already. Solid platforms with vibrant communities. For us, this begged the question: Is Logz.io competing with Elastic and Grafana Labs, the vendors who build ELK and Grafana? Is Logz.io giving something back to those communities, or should it, and how?

As far as the competition part is concerned, Levy's answer was straightforward. He admitted there is some kind of competition, but his point of view is that both ELK and Grafana are horizontal, domain-agnostic solutions, and neither of those is enough in itself. By contrast, Levy said, Logz.io is a vertical, domain-specific, best-of-breed solution.

As far as giving back goes, Levy said Logz.io is (by its estimate) the number one contributor in ELK in terms of content. Levy said Logz.io people contribute in terms of things such as tutorials, documentation and the like.

We need to talk about open source licensing and monetization

Certainly, Logz.io is an interesting solution. The company shows healthy growth, and the platform makes sense. But the strategy also raises some questions. Many people, ourselves included, have elaborated on how cloud vendors taking open source software and offering it as a managed service are in effect competing with the vendors who build the software.

Is what Logz.io does much different? Elastic, for example, is also offering a SIEM solution, covered recently by fellow ZDNet contributor Tony Baer. Grafana Labs recently secured funding to develop a platform along the lines of the one Logz.io unveiled today. And it sounds like even the technical architecture partially overlaps, for example in using Jaeger, which Grafana includes in its "big tent" approach. 

Where does one draw the line between competing in the same market, or overlapping markets, and unfair competition? Is contributing content enough to make up for it all? Would Elastic and Grafana Labs, or others who may find themselves in their shoes, be justified to react by changing their licenses to prohibit what the Logz.io of the world do? What would happen then?

We don't have the answers to those questions. What we see emerging here, however, is a cause for concern. Some vendors focus on building open source infrastructure, while others take this infrastructure and provide value-add services on top of it. For them, the cost of building this infrastructure is an externality that enables them to compete efficiently.

There's nothing wrong with building value-add services. But what happens if the builders do not just skip giving back to the infrastructure, but also compete against it? Would Logz.io, for example, be happy to take over (part of) the engineering and R&D cost for ELK and Grafana, or share what is built on top of those? The answer seems to be "no".

The question is then, is this sustainable? If everyone does this, the infrastructure is either going to collapse or end up being appropriated. If this is not something we want to see, we need to talk about open source licensing and monetization. 

ADDENDUM, December 4, 2019: Following the publication of the article, we received the following statement by logz.io CEO Tomer Levy:

"We'd love to contribute more to any part of these communities and we already do so. We have dozens of open-source repositories we contribute back to the community with. We're investing a significant part of logz.io engineering time in open-source projects such as Apollo, Sawmill and many other significant parts of the ELK open-source stack were developed by us. These are 100% open source" . 

Big Data

How to find out if you are involved in a data breach (and what to do next)

Fighting bias in AI starts with the data

Fair forecast? How 180 meteorologists are delivering 'good enough' weather data

Cancer therapies depend on dizzying amounts of data. Here's how it's sorted in the cloud

• How to find out if you are involved in a data breach (and what to do next)
• Fighting bias in AI starts with the data
• Fair forecast? How 180 meteorologists are delivering 'good enough' weather data
• Cancer therapies depend on dizzying amounts of data. Here's how it's sorted in the cloud