TL;DR — Go’s scheduler runs an M:N model: G (goroutines) are multiplexed onto P (logical processors), each holding an M (OS thread). Idle Ps steal runnable goroutines from busy Ps to balance load, which is why a single Go binary can keep hundreds of cores busy without explicit thread-pool tuning. Knowing this model turns mysterious latency spikes into diagnosable problems.

A scheduler you don’t have to think about — until you do

Most Go code reads like a recipe for disaster to anyone who’s spent time tuning Java thread pools or configuring Tokio reactor threads. You start goroutines with go func(), you share memory via channels, and the runtime just handles it. That magic is the runtime scheduler, and for the vast majority of services it stays invisible.

It becomes very visible the moment your service hits one of these walls:

  • p99 latency spikes while CPU is only 60% utilized across the fleet.
  • A microservice takes 800ms to serve a request that should take 5ms.
  • GODEBUG=gctrace=1 shows nothing suspicious, but runtime/pprof reveals goroutine piles stacked on a single channel send.

Every one of those traces back to a scheduling decision. This post walks through the GMP model that makes those decisions, why work stealing is the right algorithm for Go’s workload profile, and how to read the runtime’s output when things go wrong.

The GMP model in one diagram

The runtime schedules three actors:

  • G (goroutine) — a user-level task. Holds its own stack, which starts at 2 KB and grows up to 1 GB by default. Cheap to create; the Go runtime can carry millions concurrently.
  • M (machine) — an OS thread. The thing that actually executes instructions. The runtime maintains a pool of these (GOMAXPROCS active, plus idle ones parked on a sleep list).
  • P (processor) — a logical scheduling context. This is the abstraction most engineers miss. A P holds the run queue, local caches, and bookkeeping state needed to run a G. Critically, GOMAXPROCS caps the number of Ps, not the number of Ms.

The invariant the scheduler protects: a P can run exactly one G on exactly one M at any moment, and the number of Ps is fixed at GOMAXPROCS. Ms come and go as the runtime blocks on syscalls, but Ps are the scarce resource you actually contend for.

This is documented across the Go runtime source and summarized well in the Go scheduler design doc. The design doc is older but the architectural decisions have held.

Why P exists at all

Before Ps were introduced in Go 1.1, the scheduler bound the run queue to Ms directly. That sounds simpler; it was a performance disaster on multi-core hardware because every cache-line transition required global locking on the run queue. Introducing P as a per-CPU scheduling context gave the runtime a place to pin a local run queue and a mcache for allocations, dramatically reducing contention. Anyone migrating code from Go 1.0 remembers the latency cliff when 1.1 shipped.

How a goroutine actually runs

When you write:

go handleRequest(conn)

the runtime:

  1. Allocates a G struct, sized to fit a tiny initial stack (2 KB on modern Go).
  2. Pushes it onto the local run queue of the current P. Local enqueues are lock-free in the fast path.
  3. Returns control to your code. The G doesn’t run yet — it just sits in the queue.

Later, the M currently bound to that P drains the queue, executing each G until it returns, blocks on a syscall, or yields. The actual execution happens because the scheduler is invoked at well-defined preemption points: function calls (in async-preemptible builds), channel operations, GC assists, and explicit runtime.Gosched() calls.

Here’s the part engineers get wrong: a go statement is not a guarantee of immediate or parallel execution. It’s a request to the scheduler. If all Ps are busy running other goroutines, your new G waits its turn.

Work stealing: the heart of the design

Local run queues are 256 entries. When one P’s queue empties, the runtime has a choice: spin hoping work arrives, or find work elsewhere. It chooses the latter — that’s the entire point of work stealing.

The stealing algorithm, simplified:

  1. P_A’s local queue is empty.
  2. P_A calls findrunnable().
  3. It checks its own queue (empty), then the global queue (with rate-limited check to avoid starvation), then randomly picks 4 other Ps and tries to steal half of each one’s run queue.
  4. If a steal succeeds, P_A resumes execution. If all four candidates are empty, it polls the global queue, then the netpoller (for goroutines ready after I/O), then it parks.

The choice of 4 candidates and stealing half is load-balanced, not random. One steal is enough to refill the queue because you’re taking half; the victim keeps half and gets to recover load quickly. The fixed candidate count is a deliberate trade-off — it’s enough to hit busy Ps in practice without spending forever spinning on empty ones.

This is covered in the runtime source for findrunnable and explained in Scalability of the Go runtime by Dmitry Vyukov.

Why “steal half” beats steal-one

Imagine a fork-join workload: one goroutine fans out 1000 children, then joins. If a stealing P takes just one child, the busy P keeps 999 and remains saturated. The thief returns, runs one goroutine, finishes, and is back to stealing. You’ve serialized the system. Stealing half transfers load meaningfully and brings both Ps to comparable utilization.

The same logic makes Go great at request-burst patterns: a CPU-bound spike on one handler doesn’t strand 31 cores doing nothing while core 0 melts.

Patterns in production

HTTP servers are the canonical beneficiary. When a request arrives, an HTTP handler goroutine runs on whatever P the listener hands it to. Under load, requests distribute unevenly — some handlers block on DB calls, others crunch JSON. Work stealing redistributes the parse-heavy handlers to idle Ps without any explicit pool configuration. The gRPC-Go implementation leans on this heavily, which is one reason gRPC services in Go scale so cleanly to hundreds of cores.

Fan-out pipelines (Kafka consumers, image-processing workers) see the same benefit. A worker pool sized at GOMAXPROCS emerges naturally — you don’t need to tune a thread pool because the runtime is the thread pool.

The places the scheduler bleeds latency

Work stealing works brilliantly for uniform, fine-grained, CPU-bound work. It stumbles when those properties break.

1. Incessant syscalls

Every syscall that blocks — a database query over a slow link, an os.File.Read from a cold disk — parks the M. The runtime then detaches a new M to keep the P productive. This is fine for occasional syscalls. It’s catastrophic for code that does hundreds of blocking syscalls per second per goroutine, because:

  • M creation is cheap but not free (a stack, signal masks, thread-local storage).
  • You can hit sched.maxmcount (default 10000), at which point goroutines stack up waiting for an M.
  • The runtime also periodically retires idle Ms back to the pool, which can introduce jitter on the next syscall-heavy request.

If you’re writing a service that does Postgres + Redis + S3 calls per request, the scheduler is doing a lot of invisible work. Sometimes that’s the right answer; sometimes it’s worth wrapping the syscall boundary with a bounded worker pool so you have predictable M count.

2. Tight infinite loops without preemption

Pre-Go 1.14, a tight for { i++ } loop would monopolize its P because there were no preemption points inside. Asynchronous preemption fixed the worst symptom, but tight loops still cause scheduling lag — your goroutine is still on-CPU, just no longer monopolizing. If you suspect this, check with:

GODEBUG=asyncpreemptoff=1 ./yourbinary

If latency improves, your hot loop is preempting itself at unfortunate times. A loop body that calls runtime.Gosched() periodically, or a time.Sleep(0) once per million iterations, gives the scheduler a clean checkpoint.

3. Lock contention

sync.Mutex does not involve the scheduler directly, but contended locks create waves of goroutines waking up on the same P, then sleeping, then waking on another. This is a class of jitter the scheduler can’t fix. Replacing mutexes with channels (or vice versa) rarely helps; the cure is usually reducing the critical section or sharding the lock.

The classic Go wiki on mutex contention is worth keeping open during any investigation.

4. Channel send hot spots

A goroutine that’s blocked on ch <- x lives on a special waiting queue. When the receiver reads, the sender wakes up on the receiver’s P — which might be far away. If you’re piping gigabytes of data between two goroutines, you’ll see the cost of constant rescheduling. Buffered channels amortize this; using a ring buffer or bytes.Buffer per producer often removes a measurable amount of p99.

Reading the scheduler under a microscope

The runtime gives you several blunt instruments. They get sharper when you combine them.

Goroutine profile

go tool pprof -seconds=30 http://localhost:6060/debug/pprof/goroutine

This dumps the stacks of all currently-running goroutines. If you see hundreds of goroutines stacked on the same line — chan send on one channel, mutex.Lock on one mutex — that’s your hot spot. Pair this with contention and block profiles for the full picture.

runtime/pprof labels

Wrap critical sections:

ctx, task := runtime.NewTask(nil, "db.query")
defer task.End()
// ... query ...

These labels surface in traces so you can see which scheduler decisions are tied to which logical operations. The Go execution tracer is the single best diagnostic tool shipped with the runtime — if you haven’t used it for your hot path yet, this is your sign.

GODEBUG=schedtrace=1000

Every second, the runtime logs scheduler state:

SCHED 1000ms: gomaxprocs=16 idleprocs=12 threads=26 ...

If idleprocs stays at zero and p99 is bad, you’re CPU-bound (expected). If idleprocs is high but latency is bad, the scheduler can’t see work it could run — classic blocker/contention signature. If threads keeps growing, you’re churning Ms.

The full debug knobs are listed in the GODEBUG documentation under godebug.

Tuning GOMAXPROCS — when and how

The default of GOMAXPROCS = NumCPU is right for almost every production service. The notable exceptions:

Containerized workloads with shared CPUs. If your pod has a CPU limit of 1.5 cores, you almost certainly want GOMAXPROCS=1. The Go runtime will count the host’s 64 cores and spawn 64 Ps; under throttling, they’ll all compete for 1.5 cores’ worth of time and slow you down. The automaxprocs package from Uber watches cgroup limits and adjusts at runtime.

Latency-critical services on noisy hosts. If your machine has 32 cores but you’re sharing it with another tenant that thrashes, dropping to GOMAXPROCS=16 can yield more predictable p99 by leaving cores idle for the OS scheduler to hand out on demand.

Mostly-idle services that burst. Long-running TCP listeners, connection poolers, queue consumers. These can sometimes benefit from a smaller GOMAXPROCS to reduce context-switch overhead between goroutines — though modern Go has driven this overhead down dramatically, so measure before changing it.

Anything else is cargo culting.

Architecture: where the scheduler fits in a larger system

When you deploy a Go service, the scheduling decisions happen below your code’s awareness but above the OS. In a typical Kubernetes deployment:

┌──────────────────────────────────────────────┐
│ Kubernetes pod (cgroup: cpu=2, mem=4Gi)      │
│  ┌──────────────────────────────────────┐    │
│  │ Go binary                            │    │
│  │   P P P P P P P P (GOMAXPROCS=2)     │    │
│  │   ↕ ↕                               │    │
│  │   M M  + parked Ms                   │    │
│  │   ↕ ↕                               │    │
│  │   goroutines (10k–500k)              │    │
│  └──────────────────────────────────────┘    │
│   Linux scheduler maps 2 M → 2 host CPUs     │
└──────────────────────────────────────────────┘

The Go runtime decides which goroutine runs on which logical CPU. The kernel decides when those logical CPUs actually execute. They’re both schedulers, and they have to agree. The runtime’s GOMAXPROCS honors cgroup CPU limits when you use automaxprocs; otherwise it trusts runtime.NumCPU() at process start.

This is why a single Go binary often outperforms a JVM service at the same CPU quota. The JVM’s thread pool sizing interacts awkwardly with cgroup limits; Go’s per-P model maps cleanly. It’s also why Go’s startup time is so consistent: no JIT, no thread pool to warm up.

Key Takeaways

  • Go uses an M:N scheduler with three actors — G (goroutine), M (OS thread), P (logical processor) — and GOMAXPROCS caps Ps, not threads.
  • Work stealing moves half a run queue from a random busy P to an idle one. It’s the right algorithm for fork-join and request-burst workloads.
  • The scheduler hides the cost of routine blocking I/O but does not hide it forever. Counted syscalls, mutex contention, and tight loops are the failure modes that show up in p99.
  • Diagnose with goroutine, block, and mutex profiles; the execution tracer is the highest-resolution tool available.
  • Leave GOMAXPROCS alone unless you’re in a container with CPU limits (use automaxprocs) or sharing a host with noisy neighbors.
  • The scheduler is not a tuning knob you should reach for. It’s an architectural feature you should design around.

Further Reading