Download this case study in PDF version
Overview
At Google’s global operational scale, detecting and troubleshooting localized performance anomalies, such as sporadic I/O latency spikes, presents major infrastructural challenges. Traditional diagnostic tools introduce overhead in active clusters, while alternative methods like deploying custom debug kernels require weeks and frequently fail to capture transient anomalies. To resolve these limitations, Google deployed lightweight, dynamic eBPF tracing programs across its fleet to analyze real-time lock contention and conduct ad-hoc diagnostics safely. This platform allows site reliability engineers (SREs) to root-cause complex tail-latency issues within hours or days rather than weeks. By filtering telemetry data entirely in-kernel and leveraging the safety guarantees of the eBPF verifier, Google achieved unprecedented observability depth with negligible system overhead.
Challenges
Google’s infrastructure operates at a scale where localized performance anomalies can trigger severe bottlenecks in tail latency. Google faced distinct technical challenges when diagnosing fleet-wide performance issues:
- High Overhead of Fleet Diagnostics: Traditional profiling and tracing tools often add CPU and memory overhead, making them less attractive to run continuously inside active, latency-sensitive production clusters.
- Elusive Transient Anomalies: Rare anomalies, such as localized I/O latency spikes or misconfigured cgroups, are notoriously difficult to capture using standard static metrics or ad-hoc tools.
Existing diagnostic approaches were slow and structurally constrained. Root-causing an active issue routinely required engineers to compile and deploy custom debug kernels or attempt complex environment reproductions. These diagnostic processes typically took weeks to execute, disrupted production consistency, and frequently failed to replicate the specific conditions surrounding the transient anomalies.
Solution
To achieve deep, low-overhead fleet visibility, Google designed an observability framework using eBPF to safely inject dynamic diagnostic logic directly into kernel subsystems.
- Real-Time Lock Contention Analysis: To identify multi-core execution bottlenecks without degrading cluster performance, Google deployed lightweight eBPF tracing programs. These utilities hook directly into lock-specific tracepoints and kernel functions to capture stack traces and monitor mutexes and semaphores in real time. To ensure negligible system overhead, performance data is filtered entirely within the kernel. The program only surfaces lock events that breach predefined wait-time thresholds.
- Low-Overhead Telemetry and Ad-Hoc Diagnostics: Continuous infrastructure health monitoring is maintained through a tracing framework that continuously monitors performance metrics and exports them directly using standard eBPF iterators pinned to the eBPF filesystem. For localized, on-demand troubleshooting, Google engineered an internal diagnostic platform that integrates directly with its central cluster management agent. This infrastructure lets SREs safely deploy intermittent, highly targeted eBPF tracing programs to specific production machines in real time. The architecture provides immediate, deep introspection into kernel behaviors, such as block I/O latency and scheduler delays, precisely when and where an anomaly occurs.
- Resolving Implementation Challenges: Deploying dynamic diagnostics fleet-wide required addressing critical security, verifier, and compilation hurdles:
- Verifier Complexity Limits: Detailed telemetry and tracing programs frequently hit in-kernel verifier stack and instruction bounds as they grew. Google resolved this by restructuring code layouts, managing function inlining and non-inlining, bounding loops, and shifting large stack variables into eBPF maps.
- Secure Developer Access: To let internal developers use eBPF tracing safely without granting broad CAP_SYS_BPF permissions across production environments, Google implemented a centralized eBPF management daemon. This control plane validates and authorizes diagnostic scripts through a structured approval workflow.
- Kernel Version Adaptability: Fleet-wide distribution required a deployment pipeline tolerant of kernel discrepancies. Because specific tracepoints or internal functions do not exist uniformly across all operating kernel versions in the fleet, Google’s diagnostic tooling was built to be tolerant of those differences and gracefully degrade when encountering structural variances.
Results
The deployment of eBPF-driven diagnostics significantly accelerated incident response and optimized core production subsystems:
- Diagnostic Deployment Accelerated from Months to Hours: Changes to diagnostic and observability telemetry that previously required months of rollout effort are now safely deployed and active fleet-wide in hours to days using independent eBPF packages.
- Rapid Incident Triage: SRE teams can now root-cause transient performance issues (such as hardware-specific I/O latency or misconfigured cgroups) in hours or days instead of weeks, eliminating the need for custom debug kernels or failed reproductions.
- Production Subsystem Optimization: Using eBPF-based lock profiling, developers successfully identified and resolved complex, multi-core kernel lock bottlenecks within core kernel subsystems in active production loads.
Future Plans
Google is focused on simplifying its internal telemetry infrastructure to make deep kernel insights more accessible to engineering teams:
- Streamlined eBPF Developer Tooling: Google is actively working to make it even easier for internal developers to author and deploy custom eBPF tracing programs for gathering fleet-wide insights.
- Balancing Flexibility and Fleet Overhead: A primary architectural focus for the next generation of diagnostic tooling involves managing the infrastructure tradeoff between tracing flexibility and cluster overhead. Google engineering teams are designing guardrails to tightly limit CPU and memory consumption, minimize cache pollution, and reduce eBPF program load-time overhead during fleet-wide rollouts.
Why eBPF?
Google selected eBPF because traditional observability frameworks could not match its combination of safety, flexibility, and performance:
- Deep Observability with Low Overhead: eBPF provides direct access to internal kernel data structures. This lets Google observe and trace events that would be infeasible for a userspace program. Additionally BPF can easily attach to hot-path, high frequency events, such as filtering out tail-latency events by hooking directly into sched_wakeup and sched_switch tracepoints.
- Dynamic, Safe Ad-Hoc Introspection: eBPF lets engineers inject custom tracing logic dynamically anywhere in the kernel using tracepoints and kprobes to investigate unanticipated issues. This tracing can be attached instantly when an anomaly occurs and detached immediately when no longer needed.
- Guaranteed System Stability: Because diagnostic scripts are evaluated by the eBPF verifier, Google has mathematical certainty that production-deployed tracing tools cannot crash the system, hang the CPU, or corrupt kernel memory.