Download this case study in PDF format
Overview
Polar Signals, a company specializing in performance profiling and observability tools including Parca – which also happens to use eBPF but is distinct from this case study – uses a variety of cloud-native technologies internally, such as Kubernetes and Cilium to manage its offerings. The company faced a significant challenge in controlling internal cloud costs caused by cross-zone network traffic. Cross-zone traffic, a common issue in cloud environments, can lead to substantial costs, as data transfers between availability zones are often billed at a premium, while same-zone traffic has no associated cost. To address this, Polar Signals developed and implemented a custom in house solution leveraging eBPF technology and were able to reduce their operating costs related to cross-zone traffic by 50%.
Challenge
In mid-2024, Polar Signals discovered that cross-zone traffic accounted for nearly half of their cloud expenses. The existing tools provided by their cloud provider offered limited visibility, only reporting daily aggregated costs without granularity at the pod or workload level. This lack of detailed insights made it challenging to identify the root causes of cross-zone traffic and implement targeted optimizations.
Attempts to use existing, off-the-shelf solutions proved insufficient, as they lacked the specific capabilities needed to focus on cross-zone traffic. Polar Signals required a tool that could accurately monitor and log cross-zone traffic, integrate seamlessly with Kubernetes metadata, and provide real-time metrics to enable immediate feedback on changes.
Without such a tool, Polar Signals faced escalating costs and reduced efficiency in their cloud operations.
Solution
To address these challenges, Polar Signals developed kubezonnet, an open source project designed to monitor and measure cross-zone network traffic in Kubernetes clusters. The solution leverages eBPF to trace network packets and aggregate traffic data.
Key features of kubezonnet include:
- eBPF Integration: Using netfilter postrouting hooks, kubezonnet traces network packets leaving pods, aggregating traffic data over 10-second intervals.
- Centralized Processing: The collected data is sent to a central server, which resolves the source and destination IPs to Kubernetes pods and nodes. This process determines the zones of the nodes and identifies cross-zone traffic.
- Metrics and Logs: The server exposes:
- Prometheus metrics to monitor total cross-zone traffic by pod.
- Flow logs to provide detailed insights into traffic patterns between specific pods.
Polar Signals implemented kubezonnet with minimal overhead and integrated it into their existing Kubernetes ecosystem, thanks to the flexibility, deep visibility, and performance of eBPF.
Why eBPF?
Polar Signals chose eBPF for several reasons, one of which was the organization’s existing familiarity with the technology as they use it extensively in their own offering. For this use case, eBPF’s low overhead, flexibility, and ease of implementation were also key factors. More specifically, since eBPF operates efficiently within the Linux kernel, performance impacts are minimized. Additionally, eBPF seamlessly integrates with Kubernetes metadata, which can enable precise monitoring of cross-zone traffic. And eBPF’s programmability allowed Polar Signals to quickly develop a custom solution tailored to their needs.
Results
The deployment of kubezonnet led to immediate and measurable benefits for Polar Signals:
- Cost Savings: By identifying and addressing inefficiencies, such as excessive cross-zone database traffic and duplicated monitoring workloads, Polar Signals reduced their cross-zone traffic costs by 50%.
- Operational Improvements: Insights from kubezonnet informed infrastructure changes, such as deduplicating monitoring stacks across zones to minimize inter-zone traffic.
- Proactive Monitoring: The availability of detailed metrics and logs enabled Polar Signals to set up alerts, preventing future cost escalations caused by unexpected traffic patterns or spikes.
As a result, cross-zone traffic is no longer a major expense, and Polar Signals continues to optimize their cloud operations using the insights provided by kubezonnet.
Future Plans
Going forward, Polar Signals plans to continue using kubezonnet to monitor and optimize cross-zone traffic on their cloud infrastructure. They are also exploring contributions from the community to address current limitations, such as IPv6 support and enhanced accuracy in traffic metrics. The organization additionally plans to continue to leverage other eBPF-based projects to further enhance their observability and performance profiling capabilities, as it already does with Parca.