TL;DR — Bazel’s remote caching can turn your CI from a slow, flaky bottleneck into a fast, deterministic feedback loop, but only if you architect it with hermeticity, authentication, and storage backend selection in mind from day one. In this post, we’ll walk through the design decisions, failure modes, and production patterns that make remote caching work at scale.

Why Remote Caching Is Non-Negotiable for Bazel at Scale

At scale, CI isn’t just about running tests—it’s about feedback velocity. Every push triggers dozens of jobs, each potentially rebuilding the same artifacts over and over. Without a shared cache, each CI worker starts from scratch, wasting minutes (or hours) on actions that have already been executed on an identical machine. Bazel’s hermeticity is the key that unlocks a safe, shared cache: because every action’s inputs are hashed and the execution is sandboxed, the output is deterministic. That determinism means you can store the output once and reuse it everywhere, turning redundant work into a cache hit.

But the jump from “works on my laptop” to “reproducible at scale” is where many teams stumble. A misconfigured remote cache can silently produce wrong results, leak secrets, or become a single point of failure. The goal of this post is to give you a concrete blueprint for a remote cache that is fast, safe, and resilient.

Core Components of a Bazel Remote Cache

A Bazel remote cache is a content-addressable key-value store. The keys are action digests—SHA-256 hashes of the action’s command, environment, input files, and toolchain. The values are the output files and directory trees. When a worker needs to execute an action, it first checks the cache. If the digest exists, it downloads the outputs; otherwise, it executes the action and uploads the results.

Storage Backends

The choice of storage backend defines your cache’s performance, cost, and operational burden.

  • In-memory / disk (single node): Simplest option, but capacity is limited and there’s no sharing across workers. Suitable only for small teams or local development.
  • Cloud object stores (Google Cloud Storage, Amazon S3): Durable, cheap, and globally accessible. However, they have higher latency and request costs compared to purpose-built caches. You’ll need to wrap them with a caching layer (e.g., a CDN or a Redis front-end) to hide the latency.
  • Dedicated caching services (BuildBazel Cloud, TuleJet, etc.): These are built specifically for Bazel, offering lower latency, better compression, and built-in invalidation policies. They often run as a managed service, reducing operational overhead.
  • Self-hosted clusters (Redis, Memcached, or a custom gRPC server): Gives you full control over replication, sharding, and authentication. This is the path most large engineering organizations take once they outgrow the managed options.

Communication Protocol

Bazel speaks gRPC to the cache by default, but you can also use HTTP. gRPC offers multiplexing, flow control, and binary serialization, which is why it’s the recommended choice for high-throughput environments.

Hermeticity Guarantees: Ensuring Cache Correctness

The entire value of remote caching rests on the assumption that identical inputs produce identical outputs. Any deviation breaks the cache and leads to subtle, hard-to-debug bugs. Here are the most common hermeticity violations and how to prevent them.

  • Environment variable leakage: A build that reads HOME, USER, or CI_SECRET will produce different outputs on different machines. Use Bazel’s --incompatible_strict_action_env or explicitly set env in your cc_library/java_library rules to freeze the environment.
  • Timestamps and build metadata: Embedding the current date or a build ID into an artifact makes every build unique. Instead, use Bazel’s --stamp flag selectively, or replace dynamic values with stable placeholders.
  • Toolchain drift: If the compiler version, linker flags, or system libraries differ between CI workers, the cache is invalidated even when the source code hasn’t changed. Pin your toolchains using Bazel’s register_toolchains and verify their integrity with checksums.

To enforce hermeticity at the CI level, run a periodic “cache integrity” job that rebuilds a known set of targets from scratch and compares the outputs against the cached versions. Any mismatch should trigger an alert.

Architecture Patterns for Production

Centralized vs. Federated Caching

A centralized cache is shared by all teams and pipelines. It maximizes hit rates because every build benefits from every other build’s work. The trade-off is that it becomes a shared dependency; a outage halts all builds. A federated cache gives each team its own cache, isolating failures but reducing the overall hit rate.

Most organizations start centralized and move to federation only when team autonomy or security requirements demand it. If you go centralized, invest in high availability: run at least three cache nodes behind a load balancer, with automatic failover.

Cache Invalidation

Bazel doesn’t have a built-in TTL for cache entries. Instead, invalidation is content-based: if the action digest changes, the old entry is orphaned. To reclaim storage, you need an external policy:

  • Time-based eviction: Delete entries older than N days. This is simple but can cause cache misses for long-lived branches.
  • Manual purge: Triggered by an operator when a toolchain or environment change is detected.
  • Smart eviction: Use access patterns to evict least-recently-used entries first. This requires a custom backend or a managed service that exposes metrics.

Authentication and Authorization

A remote cache is a write-capable resource. If an attacker can upload arbitrary outputs, they can poison your builds. At minimum, use TLS for transport security. For stronger guarantees, use mutual TLS (mTLS) so that only authenticated clients can connect. If you’re running on a managed platform like GCP, IAM-based access control can restrict which service accounts can read and write to the cache bucket.

High Availability

A cache outage doesn’t stop builds—it just makes them slower. But if the cache is unavailable and your CI is configured to fail on cache errors, you’ll see a wave of red builds. The safest pattern is to make cache misses non-fatal: configure Bazel to fall back to local execution when the remote cache is unreachable. You can do this with --remote_cache_fallback_to_local (or by using a caching proxy that returns a 503 and lets Bazel proceed).

Configuring Bazel for Remote Caching

The heart of your setup is the .bazelrc file. Here’s a production-grade example:

# .bazelrc

# Remote cache configuration
build --remote_cache=http://cache.example.com:8080
build --remote_upload_local_results=true
build --remote_max_connections=20
build --remote_cache_compression=true
build --remote_cache_max_wait=1m

# Fallback behavior
build --remote_cache_fallback_to_local

# Hermeticity enforcement
build --incompatible_strict_action_env
build --action_env=BAZEL_CACHE=true

# Toolchain pinning
build --extra_toolchains=@bazel_toolchains//...:all

Tune --remote_max_connections based on your CI worker count and network capacity. A good starting point is 10–20 connections per worker. The --remote_cache_max_wait controls how long Bazel will wait for a cache response before timing out; set this to a value that’s comfortable for your network latency.

Measuring Effectiveness: Metrics and Observability

You can’t improve what you don’t measure. The three key metrics for remote caching are:

  • Cache hit rate: The percentage of actions served from the cache. Aim for >80% on stable branches. A sudden drop often indicates a toolchain change or a hermeticity violation.
  • Upload/download volume: Total bytes transferred. Spikes can indicate a cache miss storm or a misconfigured rule that’s uploading large, unnecessary artifacts.
  • Latency: p50, p95, and p99 response times for cache lookups. If p99 latency exceeds a few hundred milliseconds, your cache backend is too slow for interactive builds.

Bazel emits these metrics to the console, but for production you’ll want to scrape them with Prometheus. The Bazel Observability guide shows how to expose them via a Prometheus endpoint. Set up alerts for hit rates below 70% and for any cache errors that persist for more than five minutes.

Failure Modes and Mitigations

Cache Poisoning

Cache poisoning occurs when an incorrect output is stored under a digest that later matches a legitimate action. This can happen if two actions have the same digest but different environments—a sign of a hermeticity bug. To detect this, periodically run a “verify” job that downloads a cached output and recomputes its hash. If the hash doesn’t match, you have a poisoning incident.

Network Partitions

If your CI workers can’t reach the cache, they’ll fall back to local execution (if configured). This is safe but slow. To avoid a full miss storm after a partition, consider a local disk cache on each worker. Bazel will check the local cache first, reducing the number of remote requests once connectivity is restored.

Storage Cost Control

Remote caches can grow quickly. At one company I advised, the cache grew to 2 TB in six months because a test rule was accidentally including large data files. To prevent this:

  • Use --remote_cache_max_wait to avoid uploading huge outputs that time out.
  • Configure lifecycle policies on your storage backend to delete old entries.
  • Monitor the upload/download ratio. If uploads far exceed downloads, you may be caching things that are rarely reused.

Key Takeaways

  • Hermeticity is the foundation: Without it, remote caching is unsafe. Pin toolchains, freeze environment variables, and validate outputs.
  • Choose the right backend: Start with a managed service or cloud storage with a caching layer, then move to a self-hosted cluster as your needs mature.
  • Design for failure: Make cache misses non-fatal, use mTLS for authentication, and set up observability from day one.
  • Measure relentlessly: Hit rate, latency, and volume are your north star. Alert on deviations.
  • Evolve your architecture: Centralized caches maximize reuse; federated caches provide isolation. Most teams start centralized and only federate when forced.

Further Reading