TL;DR — The USE method (Utilization, Saturation, Errors) gives you a three‑point checklist that maps directly onto production metrics. By instrumenting each layer with Prometheus and wiring focused alerts, you can isolate bottlenecks before they cascade into outages.

Production systems rarely break because a single metric spikes; they fail when a resource is over‑utilized, saturated, or error‑prone in tandem. The USE method, coined by Brendan Gregg, translates that insight into a repeatable diagnostic workflow. In this post we unpack the three pillars, wire them into a Prometheus‑Grafana stack, and surface concrete patterns you can adopt on day one.

The Foundations of the USE Method

Utilization

Utilization measures how much of a resource’s capacity is being used. High utilization is a warning sign, but not necessarily a problem—think of a CPU running at 70 % on a web tier that comfortably handles its traffic envelope.

What to watch

  • CPU – node_cpu_seconds_total divided by the number of cores.
  • Memory – node_memory_Active_bytes / node_memory_MemTotal_bytes.
  • Disk I/O – node_disk_reads_completed_total + node_disk_writes_completed_total normalized by device bandwidth.

PromQL example

# CPU utilization per instance (percentage)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

A sustained 90 %+ CPU across a fleet usually indicates that you’re approaching the saturation frontier, which we explore next.

Saturation

Saturation captures how much demand is being queued because the resource cannot keep up. This is where utilization alone becomes deceptive: a CPU at 95 % may still have spare cycles, but a network interface at 70 % with a long queue length is already saturated.

Typical saturation signals

  • Run queue length – node_load1 vs. number of cores.
  • Disk queue – node_disk_io_time_seconds_total or node_disk_io_now.
  • Network drops – node_network_receive_drop_total.

PromQL example

# Average run queue length per core (Linux)
node_load1 / count(count(node_cpu_seconds_total) by (cpu))

When the run‑queue consistently exceeds the core count, the scheduler is queuing tasks and latency will spike even though utilization may still be under 100 %.

Errors

Errors are the failure dimension of the triangle. They surface when utilization or saturation pushes a service into an unexpected state—timeouts, retries, or outright crashes.

Key error metrics

  • HTTP 5xx – haproxy_backend_http_responses_total{code=~"5.."}.
  • gRPC errors – grpc_server_handled_total{grpc_code!="OK"}.
  • Application exceptions – custom counters exported via a client library.

PromQL example

# Rate of 5xx responses per service
sum by (service) (rate(haproxy_backend_http_responses_total{code=~"5.."}[1m]))

A sudden rise in error rate while utilization is stable is a classic symptom of saturation: the service can’t process incoming requests fast enough, leading to timeouts.

Architecture of a Modern Monitoring Stack

Prometheus + Grafana in Production

Most organizations that have adopted the USE method already run Prometheus at scale. A typical production layout looks like:

  1. Exporters on every node (node_exporter, cadvisor, custom app exporters).
  2. Remote write to a long‑term store (Thanos or Cortex) for retention beyond 30 days.
  3. Grafana dashboards that combine the three USE dimensions per resource.
  4. Alertmanager with routing rules that map saturation alerts to on‑call pages.
# prometheus.yml snippet – remote_write to Thanos
remote_write:
  - url: "https://thanos-receiver.example.com/api/v1/receive"
    timeout: 30s
    queue_config:
      capacity: 2500
      max_shards: 200

This architecture decouples short‑term scrape performance from historic analysis, allowing you to back‑track a spike in errors to the exact point where saturation began.

Collecting Metrics at Scale

When you cross a few hundred nodes, scrape latency becomes a bottleneck. Strategies that keep the USE pipeline responsive:

  • Sharding – split targets into multiple Prometheus instances behind a federation layer.
  • Service discovery – use Kubernetes Endpoints or Consul to auto‑register new instances.
  • Relabeling – drop high‑cardinality labels that aren’t needed for USE (e.g., pod IDs) to keep series count low.
# Example: relabel to drop pod UID
--relabel-config="source_labels=[__meta_kubernetes_pod_uid],target_label=__tmp_keep,action=drop"

By trimming unnecessary dimensions, you retain a clean, query‑able dataset for utilization, saturation, and error analysis.

Patterns in Production

Alerting on Saturation

An effective saturation alert combines a threshold with a trend to avoid noise:

# Alert when average run‑queue per core exceeds 2 for 5 minutes
avg_over_time(node_load1[5m]) / count(node_cpu_seconds_total) > 2

The rule is routed to PagerDuty with a severity tag that reflects the resource type (CPU, disk, network). In practice, teams pair this with a run‑book that suggests immediate actions: scaling out the tier, adjusting request concurrency, or flushing disk buffers.

Capacity Planning with Utilization

Utilization trends over weeks give you a data‑driven basis for capacity forecasts. Plot the 95th‑percentile CPU usage per service and overlay projected traffic growth.

# 95th percentile CPU per service over the last 30 days
histogram_quantile(0.95, sum by (le, service) (rate(node_cpu_seconds_total{mode!="idle"}[30d])))

When the projected 95th‑percentile creeps toward 80 %, you schedule a right‑size operation—either add nodes or upgrade instance types—before saturation materializes.

Correlating Errors with Saturation in Real Time

A common production pattern is to create a composite alert that fires only when both saturation and error rates breach thresholds simultaneously. This reduces false positives during benign load spikes.

# Composite alert: high run‑queue *and* rising 5xx rate
(
  avg_over_time(node_load1[2m]) / count(node_cpu_seconds_total) > 1.5
) and (
  sum by (service) (rate(haproxy_backend_http_responses_total{code=~"5.."}[2m])) > 0.05
)

When this alert triggers, the on‑call engineer receives a pre‑filled incident ticket that includes links to the relevant Grafana panels, cutting MTTR dramatically.

Key Takeaways

  • Utilization tells you how busy a resource is; keep an eye on 70‑90 % thresholds for early warning.
  • Saturation reveals where demand exceeds capacity; queue length and drop metrics are more decisive than raw usage.
  • Errors are the symptom that ties utilization and saturation together; a rising error rate often signals hidden saturation.
  • A Prometheus‑Grafana stack with remote write and sharding scales the USE method to thousands of nodes without losing fidelity.
  • Composite alerts that combine saturation and error signals drastically reduce noise and improve incident response times.

Further Reading