Open source observability marches on: New Relic and Grafana Labs partnership brings benefits to developers

New Relic is one of the leaders in Application Performance Monitoring (APM), which has been on a pivot to observability. Grafana Labs, makers of popular open-source dashboarding platform Grafana, has been on an growth course for a while now.

Today, the two vendors announced an ongoing partnership they claim will drive advanced open instrumentation and visibility for all developers and software teams. The companies delivered new integrations designed to empower engineering teams to solve problems even faster, as well as a free trial of Grafana Enterprise for new and existing New Relic customers.

We'd be lying if we said we saw this coming. In retrospect, however, the partnership seems to makes sense. ZDNet connected with Grafana Labs CEO Raj Dutt and New Relic Chief Product Officer Bill Staples, and discussed the specifics of the partnership as well as the broader observability landscape.

A partnership that makes sense

New Relic has been on a reinvention course for a while now. 2019 marked a pivot to observability, embracing A.I. and open source. The company has updated its New Relic One platform which instruments IT environments and applications, with CEO Lew Cirne noting this is designed to make New Relic easier to consume and address the convergence of logs, infrastructure, and APM.

Grafana's recent release, on the other hand, brought enhancements to simplify the development of custom plugins and increase the power, speed, and flexibility of visualization. Composability, or the the ability to integrate data from a variety of data sources which Dutt refers to as a "big tent philosophy", is key for Grafana.

Staples noted that New Relic is trying to eliminate both functional and the economic barriers to making observability ubiquitous, referring the simplified pricing, interface, portfolio, and a free tier for the New Relic One platform just unveiled. He went on to add that New Relic realized that probably the most popular and prolific visualization and dashboard platform in the world is Grafana:

"A lot of our customers use Grafana in conjunction with New Relic. We wanted to have a great story there, as developers embrace our observability platform and the great economics. We want them to have great visualization on top of that. In addition to our own dashboard, we wanted to open it up to Grafana. So we reached out, and the rest is history".

For Dutt, the interesting thing within the partnership is that New Relic is now offering Prometheus and PromQL as a method of querying data on the New Relic Telemetry Data Platform (TDP). Prometheus is one of the most interesting, and the fastest growing data source and community in the big tent ecosystem, and Grafana Labs is one of the main contributors, he went on to add.

Αs part of the partnership, New Relic is providing a PromQL-like interface to the metrics in their telemetry data platform. Grafana speaks native PromQL, and can be used to bring New Relic data together with other data, whether that's in open source Prometheus, in tools like Graphite or other commercial vendors like DataDog or Stackdriver or Azure monitor, said Dutt.

Built on open source Prometheus

PromQL is the query language used by Prometheus, and it was a cornerstone for the integration between Grafana and New Relic's TDP. Prometheus users can use the Prometheus remote write capability to send metric data directly to New Relic's TDP with a single configuration change. Grafana open source users can now add TDP as a Grafana data source using Grafana's native Prometheus data source.

This enables teams to enjoy New Relic's up-to 13 months of retention for their Prometheus metrics while continuing to use their existing Grafana dashboards and alerts. With New Relic's new PromQL-style syntax, Prometheus users no longer need to learn a new query language.

Grafana Enterprise customers using Grafana's New Relic data source plugin will enjoy updates designed to support New Relic's latest NRQL native query language capabilities. The plugin enables users to query any data stored in TDP using NRQL to build dashboards in Grafana Enterprise. Plus, new and existing New Relic customers get a free trial of Grafana Enterprise for 30 days.

This means there are now 3 ways to access New Relic data via Grafana: NRQL, PromQL, and Grafana's own data processing and transformation layer. New Relic essentially built a translation layer between NRQL and PromQL, which made the integration with Grafana easy.

You may be wondering what the difference between storing your telemetry data in Prometheus versus New Relic is, now that the option is there. Staples said that what TDP offers extra is the ability to extend retention, provide additional scale, plus private key encryption of data a fully managed solution.

Dutt on his part noted that Prometheus' remote write functionality is something Grafana labs developed about three years ago as part of its participation within the Prometheus community. He went on to add that the use of Prometheus is a way of recognizing the growth of Prometheus, as it's become the de facto standard for metrics in the observability community.

The future of observability, open source, standards, and AI

Grafana Labs is clearly big on Prometheus. New Relic, after announcing the intention to get involved in OpenTelemetry, has company executed on it. Staples said New Relic believes the future of instrumentation is open and open standard, and they are the number three contributor today to OpenTelemetry project after Microsoft and Splunk.

Staples said New Relic is committing engineers and resources to OpenTelemetry, and wants to fully supported it out of the box package when it's released. TDP currently has beta support for OpenTelemetry. New Relic also open sourced existing agents and integrations, and shared the decade plus of IP that we have in instrumentation with the community, Staples added, noting those projects are now ran fully in the open and will conttinue to do so.

Interestingly, another proprietary vendor, Sumo Logic, just announced support for OpenTelemetry too. Grafana Labs, however, is not entirely sold on OpenTelemetry. They're watching it closely, but they believe more in open source than in open standards, said Dutt.

Dutt finds OpenTelemetry interesting because it's tried to combine things like tracing and metrics together, and now supports both metrics, logs and traces as kind of first party data sources and first party telemetry types. However, his preference for open source rather than "committee driven standards" is clear.

In addition to open sourcing agents similar to New Relic, Grafana Labs also open sources all its backends, including Cortex. Cortex is CNCF project that's a scale out Prometheus back end that does long term storage. Dutt described it as a scalable Prometheus offering that runs on Grafana cloud. So there is competition, and different views there.

Wrapping up, we asked Staples and Dutt to comment on the themes that emerged as key for the future of observability in a recent survey: open source, AI and machine learning, cloud and serverless. Open source is a kind of no brainer here, as it's core to Grafana, and something that New Relic is consciously trying to build on.

As far as AI and machine learning go, Dutt acknowledged them, but noted there's lots of hype, and he doesn't believe that it's going to be a replacement for talented SREs anytime soon. Staples emphasized New Relic's applied intelligence product, which uses AI and ML based approaches to discover anomalies and take automated actions based on the data.

Overall, both executives seemed to be on the same page regarding the main themes for the observability space, even if how they approach them may be a bit different. New Relic's turn to open source is going strong, while Grafana Labs continues to execute on its open source legacy path.