Mastering Build Cache Invalidation with Bazelisk and BuildKit

TL;DR — Bazelisk proxies builds to BuildKit, which stores per‑file digests; cache misses are detected via content‑addressable hashes, and invalidation is triggered automatically when source files change, eliminating stale artifacts and cutting CI times by up to 40%.

Introduction

In modern CI/CD pipelines, build time is often the bottleneck that determines developer velocity. A well‑tuned cache can shave minutes off every pull‑request check, but the same cache becomes a liability when it serves outdated artifacts after a source change. Bazelisk, the lightweight Go wrapper around Bazel, and Docker’s BuildKit, the next‑generation build engine, together provide a robust mechanism for content‑addressable caching and automatic invalidation. This article walks through the architecture, configuration, and real‑world patterns for mastering cache invalidation with Bazelisk and BuildKit, and includes concrete tips you can apply today.

Why Cache Invalidation Matters

• Build latency: Even a 5 % cache miss ratio on a medium‑sized Go monorepo can add 2–3 minutes per pipeline run.
• Flaky tests: Stale binaries or generated code can cause tests to pass on a cached build and fail on a fresh one, eroding confidence.
• Resource waste: Re‑compiling unchanged code wastes CPU, memory, and cloud CI budget.

Understanding how Bazelisk forwards actions to BuildKit, and how BuildKit tracks dependencies via digests, is the first step toward a dependable cache strategy.

Bazelisk: The Bazel‑Aware Wrapper

Bazelisk’s primary job is to locate the correct Bazel version for a repository and forward the command line unchanged. It resolves the version from a WORKSPACE file, a .bazelversion file, or a remote release, then execs the matching Bazel binary. Because Bazelisk is stateless and tiny (≈2 MB), it can be safely installed on developers’ machines and CI runners without impacting build semantics.

Key points:

• Version pinning: Guarantees every CI job uses the exact Bazel release that the project was tested against.
• Remote execution proxy: When BAZEL_REMOTE_EXECUTOR is set, Bazelisk forwards the action to the remote executor—often BuildKit running on a Docker daemon.
• No extra configuration: Existing Bazel BUILD files, .bazelrc entries, and environment variables continue to work.

BuildKit: Content‑Addressable Build Cache

BuildKit, merged into Docker Engine 18.09+, introduces several cache improvements over the legacy Docker builder:

1. Immutable, content‑addressed layers: Each build step’s output is addressed by a SHA‑256 hash of its input files and the step’s command. If any input changes, the hash changes and the layer is considered stale.
2. Secret and credential handling: Secrets are injected only at build time and never baked into layers, reducing leak surface.
3. Cache mount: RUN --mount=type=cache lets you mount a directory (e.g., /root/.cache) that persists across builds, keyed by the hash of the mount’s contents.

When Bazel invokes BuildKit via its remote execution protocol, each Bazel action becomes a BuildKit step. The resulting cache key is deterministic: change a source file, and the hash changes, automatically invalidating downstream steps.

Architecture: Combining Bazelisk and BuildKit for Invalidation

Below is a high‑level diagram of the data flow in a typical CI pipeline:

[Git repo] --> [Bazelisk] --> [Bazel] --> [BuildKit daemon] --> [Docker image / artifact store]
          ^                     |
          |---------------------+--- (cache keys derived from file digests)

Steps:

1. Checkout: The CI runner checks out the repository.
2. Bazelisk invocation: bazelisk build //... launches Bazel with the project‑specific version.
3. Action graph execution: Bazel computes action hashes using source file content, dependency declarations, and environment variables.
4. BuildKit remote execution: Each action is sent to a BuildKit instance (local or remote). BuildKit recomputes the content hash; if it matches a cached layer, the step is skipped and its output is reused.
5. Cache persistence: BuildKit stores layers in a local /var/lib/buildkit directory or a remote registry, indexed by hash.

When a developer pushes a single line change, only the actions that depend on that file receive new hashes; everything else is served from cache. This granular invalidation is what makes the combination so powerful.

Configuring Bazel for BuildKit Integration

To get Bazel to talk to BuildKit, you typically set the --strategy flag or add entries to .bazelrc. A minimal configuration looks like this:

# ~/.bazelrc
build --strategy=Dockerfile=remoteexec
build --executor=remote
remote executor address http://localhost:5000
remote cache dir /tmp/bazel-cache

If you are using the official Bazel Docker image, you can start BuildKit in the background:

dockerd --enable-experimental-features=buildkit &
export DOCKER_BUILDKIT=1

Environment variables that matter:

• BAZEL_REMOTE_EXECUTOR: URL of the remote executor (often BuildKit).
• BAZEL_CACHE_DIR: Path where BuildKit stores local layers.
• DOCKER_BUILDKIT=1: Ensures the Docker CLI uses BuildKit for any docker build invocations.

A practical CI snippet (GitHub Actions) might look like:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Docker BuildKit
        run: |
          export DOCKER_BUILDKIT=1
      - name: Install Bazelisk
        run: go install github.com/bazelbuild/bazelisk@latest
      - name: Build with Bazelisk + BuildKit
        run: |
          BAZEL_REMOTE_EXECUTOR=http://localhost:5000 bazelisk build //...

Common Pitfalls and How to Fix Them

Patterns in Production

Many organizations have standardized on a “cache‑first” pipeline:

1. Warm the cache on every push by running a lightweight bazelisk fetch //... before the full build. This pre‑computes hashes and populates BuildKit’s local store, reducing subsequent build times.
2. Cache versioning via a BUILD_CACHE_VERSION env var. Increment the version whenever you change the build infrastructure (e.g., upgrade Bazel or BuildKit). This forces a full invalidation, preventing subtle incompatibilities.
3. Artifact publishing for shared libraries: publish cached binaries to an internal artifact registry (e.g., Artifactory, GCP Artifact Registry) and configure Bazel’s remote_cache to pull them. Subsequent builds benefit from pre‑built artifacts without recompiling.
4. Canary builds on feature branches: run a short‑circuit build that only compiles changed targets. If the canary passes, the full pipeline can skip unchanged steps, leveraging the cache’s granularity.

These patterns are not Bazel‑specific; they translate well to any CI system that leverages content‑addressable builds.

Key Takeaways

• Bazelisk + BuildKit give you content‑addressable caching out of the box—no custom hash logic needed.
• Cache invalidation is automatic: any change to a source file propagates a new digest, causing downstream steps to recompute.
• Configuration is minimal: a few .bazelrc entries and DOCKER_BUILDKIT=1 are usually sufficient.
• Observe the cache hit ratio (bazelisk query //... --output=label can surface cache status) to tune CI resources.
• Avoid pitfalls by keeping environment consistent, clearing stale layers periodically, and using BuildKit’s secret handling.
• Adopt production patterns like cache warming, versioning, and artifact publishing to maximize build throughput across large teams.

Further Reading

• Bazelisk GitHub repository – installation and version‑pinning guide.
• BuildKit documentation – details on content‑addressable layers and cache mount.
• Bazel remote execution guide – how to configure remote executors and caches.
• Effective CI with Bazel – real‑world case study of cache hit improvements.
• Docker BuildKit secrets best practices – secure handling of credentials inside builds.

Building something like this? I’m an AI/systems contractor open to new projects. Book a 30-min call.