The rise of first-mile observability: Calyptia wants to enable enterprises to log all the things

Cloud-native is the name of the game for application development. The creators of the Fluent Bit and Fluentd popular open-source projects for cloud-native observability are launching an offering aimed at the enterprise

More and more applications, with more and more microservices and back-ends, deployed over more and more clouds, generating more and more logs. This is what developers have to deal with every day, and it makes observability harder and harder.

If you think monitoring and debugging applications in this setting is hard, you’re right. But then of course, we have observability, which is not monitoring, but everyone agrees is important. Confused yet?

Observability definitions abound. A simple one that seems to the point is “the ability to measure a system’s current state based on the data it generates, such as logs, metrics, and traces”.

Fluentd and Fluent Bit are widely used open source projects under the auspices of the CNCF, focusing on the logs part of the equation. They work in tandem to collect and process, respectively, information stored in application and service logs.

Combined, Fluentd and Fluent Bit have more than 30K GitHub stars, 2M deployments per day, and 1,3K active contributors. They have been deployed over a billion times and are embedded into industry standard technologies such as Kubernetes and Red Hat OpenShift.

Today, Fluentd and Fluent Bit maintainer Anurag Gupta and former product manager Eduardo Silva announce the launch of Calyptia, a company formed to commercialize Fluent Bit, as well as a $5 million seed round led by Sierra Ventures and Carbide Ventures We caught up with Gupta and Silva to discuss Calyptia’s offering and its positioning in the observability landscape.

Contextualizing logs

As Cindy Sridharan, author of “Distributed Systems Observability” notes, logs, metrics, and traces are often known as the three pillars of observability. While plainly having access to logs, metrics, and traces doesn’t necessarily make systems more observable, these are powerful tools that, if understood well, can unlock the ability to build better systems.

Fluentd is an open source data collector for unified logging layer, enabling users to unify data collection and consumption for a better use and understanding of data. Fluent Bit is an open source logging and metrics processor and forwarder, and it’s what Calyptia’s offering is built around.

As Silva put it, Fluentd and Fluent Bit are “cousins” of projects such as Jaeger and Zipkin, which focus on traces. Logs, Silva went on to add, are a well-known mechanism to enable some kind of visibility into what an applications does.

When it’s time to analyze how an application is behaving, the first place to look is application logs. But in a distributed system setting, such as Kubernetes, that’s not trivial:

“You have a handful of applications, and you do want to perform data analysis,to see how the applications are behaving. You want to take a look at the logs, but in order to solve that, you cannot go to each one of the applications, every single file, and take a look at what this application is doing. You need to have a specialized tool.

You can tell it — Hey, these are the applications. Here are the logs. Please take the logs and centralize them in a database or a cloud service so we can perform data analysis. So Fluentd and Fluent Bit are this specialized tool that can collect this log information, process it and centralize it in one or multiple endpoints for analysis”, Silva said.

The power of the combo comes from the fact that it does not just collect and forward data stored in logs, but it also applies transformation and parsing logic, Gupta added. This can be done using things such as Lua scripting, or adding specific key value pairs in the Kubernetes space.

What Fluent Bit and Fluentd have been doing for the past 10 and 5 years, respectively, is adding the ability to talk to APIs within Kubernetes and reach all that data. This helps contextualize the data in logs, and make it more meaningful, so diagnosing and debugging becomes faster and easier.

A key advantage of working with Fluentd and Fluent Bit is the ability to work with multiple back-ends, Gupta noted. The tools are vendor agnostic when it comes to back-ends, so data can be routed wherever it needs to go, regardless of what they are or how many they are. Even though back-ends are proliferating, Fluentd and Fluent Bit will be keeping up, Gupta added.

First-Mile Observability

Gupta and Silva, both with long-standing experience in software systems engineering and stints at Microsoft and Oracle, respectively, met in 2017. They worked closely together on Fluentd, and as time went by, they saw the dynamics and decided to found Calyptia to offer commercial services around it in 2020.

Calyptia Cloud is a key part of the Calyptia Enterprise offering, and it includes two things: a management and monitoring layer, and developer tools. The developer tools are aimed at people who just want to use the open source layer, and can help them figure out some key aspects such as regular expressions or whether or not things are working properly, Gupta said.

The management and monitoring offering aggregates all the information that Fluentd and Fluent Bit have at the edge layer, contextualizes and visualizes it. This can help answer questions such as — where is the data routed to, how much data is routed, to which IPs, why did the data choose a specific path, or why was this volume of data sent.

The goal is to also be able to have predictive analytics. In this case, this means being able to surface such insights ahead of the data landing into that back-end, and then getting billed for it, as Gupta said. As for Calyptia Cloud’s own back-end, the service is also agnostic, being able to operate with anything ranging from New Relic and Splunk to Amazon and Google object storage or PostgreSQL.

Observability can be seen as a journey, as per Gupta. People can start with simple monitoring, then evolve, and start adding more data sources. The journey starts at the place where the data is created and continues towards the place where debugging, diagnosing and getting insights happens.

Calyptia sees the place where data gets generated, contextualized, processed and routed as the first mile in that journey. This, Gupta said, is where Fluentd and Fluent Bit have been operating, and this is also what Calyptia is aiming at.

Silva on his part described observability as a continuum, with Calyptia addressing the data collection and pre-processing end of the spectrum, and tools such as Grafana addressing the visualization end. Calyptia works side-by-side and integrates with all of them, he went on to add. That may be so, but it does look to us like at least some overlap does exist.

Gupta and Silva mentioned that they spent a lot of time on calls with people from the likes of AWS, Google Cloud and Microsoft, as well as enterprises looking to solve observability at scale, and open source project contributors. This helped shape their offering, which is now used at places like Fidelity or Neiman Marcus.

In addition to Calyptia Cloud, Calyptia Enteprise can be self hosted on any environment. Calyptia offers a subscription model, as well as consulting services and support. The company aims to use the funding to increase its current headcount of 16, and invest in the continued development of its software.