Open source observability, meet data transformation: Grafana 7.0 promises to connect, unify, and visualize all your data
Grafana Labs, makers of popular open-source observability platform Grafana, announced the general availability of Grafana 7.0. This comes only a few months after Grafana Labs scored $24 million in Series A funding to double down on open-source strategy and build what it dubs the world's first open and composable observability platform.
Featured
Today's release brings enhancements to simplify the development of custom plugins and increase the power, speed, and flexibility of visualization. Open source Grafana is among the world's most popular dashboard solutions and boasts nearly 600,000 active installations and millions of dashboards in use across the globe.
Grafana Labs' plans seem to be well underway. ZDNet connected with CEO Raj Dutt to discuss Grafana 7.0, and the road forward.
There's a plugin for that: Use your programming language of choice, ingest any data, from anywhere
Like many of us at this time, Dutt is conducting business from home. The difference is, it sounds like Dutt's business is going well, and Dutt seemed to have a genuine enthusiasm about it. Having seen what Grafana 7.0 brings to the table, and heard how Grafana Labs has been faring, we think the enthusiasm may be warranted.
Dutt noted that Grafana 7.0 is an accumulation of effort commencing after 6.0 spanning nearing 18,000 commits and 3,699 pull requests from 362 contributors around the world. Additionally, there are hundreds of company, commercial, and community data-source plugins and thousands of sample/starter dashboards supporting users' needs both on-premise and in the cloud.
This is the first thing that stands out about Grafana 7.0. Like any open-source product, a good part of Grafana's success is owed to its open source community and the contributions the community brings. Grafana Labs is very much aware of this, which is why they emphasized the simplification of the plugin development framework for Grafana 7.0.
What Grafana Labs means by composability is the ability to integrate data from a variety of data sources. Image: Grafana Labs
Grafana 7.0 features new plugins as a result of partnerships across Google (Stackdriver/Cloud Monitor), Microsoft (Azure Monitor), and Amazon (Cloudwatch), plus logs support with open source Loki and tracing inputs from Zipkin and Jaeger. That's the "what," and it's interesting in and by itself, as it adds more data sources to Grafana's "Big Tent" open source ecosystem.
Equally interesting is the "how." Dutt put that in perspective quite vividly, by saying that what used to take 1.000 lines of code to develop, can now be done with 100 lines of code. That's an order of magnitude less, so we wondered how Grafana 7.0 achieved that. While up to now Grafana was focused on time series data, the new version adopts a broader view.
Grafana 7.0 has new component libraries, tools, data structures, and a completely rebuilt common and unified data and plugin framework based on Apache Arrow. The consistent data structure brought by Apache Arrow reduces the effort required to develop plugins. Arrow is a language-agnostic software framework for developing data analytics applications that process columnar data.
Equally important from a developer-friendliness point of view, Dutt noted that Grafana's execution environment for plugins is programming language agnostic. Users can write their plugins in any programming language, and Grafana will execute them. It sounds like a custom virtual machine, and it hardly gets more developer-friendly than that.
There's a data transformation for that: Process and shape your data any way you want them
The second noteworthy feature of Grafana 7.0 is its data transformation capabilities. Grafana plugins can ingest data from various sources, but as Dutt said, up until now if the data you ingested were not in a format that worked for you, you were kind of out of luck. Grafana 7.0 changes that, by introducing the ability to process and transform data.
Grafana already offered the option to visualize ingested data, but now users can also apply data processing rules to transform the data before they visualize it. Dutt said that a shared set of common data operations that were previously duplicated as custom features in different places are now part of Grafana's data processing pipeline, and something all data sources and panels can take advantage of.
To achieve this, Grafana introduced a query processing and transformation language and execution environment, which is a pretty impressive feat. In a way, this sounds like a streaming data ingestion and processing framework within Grafana, so we wondered if this is what it's based on. Although Dutt acknowledged the similarity, he noted this is not based on a streaming data platform.
Grafana 7.0 offers new data ingestion and processing capabilities, as well as GUI improvements
Queries are automatically generated behind the scenes, while users work in a GUI to map and transform their data. Transformations include renaming, summarizing, combining, and performing calculations from different panels, putting time-series labels into columns, reusing and refining query results across other panels, and more.
This is particularly useful for data sources that do not have their own data processing capabilities, such as logs for example. At execution time, queries run against incoming data, and results are displayed on Grafana's dashboards. Users can see, export, and perform simple transformations on the underlying source data.
Users can also drill into query execution details for faster troubleshooting. Dutt said that at a later point the plan is to give power users the ability to intervene behind the scenes, and write their queries directly, or optimize auto-generated queries.
Concluding with the highlights of the new and noteworthy in Grafana 7.0, a new tracing viewer to complement the existing support for metrics and logs enables users to trace the path of a single request through a distributed system. Some more usability enhancements have been introduced, along with the option to search, discover, and secure dashboards for Grafana's Enterprise version.
Growth, in the cloud and on-premise
Speaking of enterprise, Dutt also touched upon the business side of Grafana Labs' path since the Series A funding. The company has grown from 80 to 140 people, and the effects of the lockdown are minimal, Dutt said, because it has always been a remote-first company.
Revenue has grown over 50 percent, Dutt went on to add. What's interesting in the broader scheme of things is where this growth is coming from. According to Dutt, it's a 50/50 split between Grafana's Enterprise version running on-premises, and Grafana Cloud. This is interesting dynamics, especially considering Grafana Labs' cloud-first strategy, and the overall trend favoring cloud-first, too.
Dutt delved further into this, explaining that Grafana Cloud has been bringing many more customers, but Grafana Enterprise is where the big customers are. The explanation that Dutt offered is that beyond the usual reasons that may keep organizations off the cloud, such as compliance and security, there is another, often overlooked factor: data gravity.
These days, we typically use the term data gravity to refer to the fact that as more data moves to the cloud, data storage and processing software follows, too. But it can also work the other way round, Dutt pointed out. For large organizations with lots of on-premise data, it makes much less sense to send everything to some cloud.
Installing a version of data storage and processing software on-premises makes things much simpler if most of your data lives there. Except that on-premise, Dutt was quick to clarify, could actually mean your own AWS or Azure or Google Cloud storage. Anything that is not part of your software vendor's offering, basically. Not exactly what we typically relate to by using the term on-premise, but overall the notion does make some sense.
Cloud or on-premise, however, Grafana Labs seems set on a growth path.