TL;DR — Copy‑on‑Write (COW) lets the Linux kernel share physical pages until a write occurs, dramatically reducing memory churn for fork, mmap(MAP_PRIVATE), and container overlays. Understanding the fault‑path code (handle_cow_fault), the associated data structures (mm_struct, vm_area_struct), and a handful of sysctl knobs lets you tune COW for latency‑critical services such as Kafka or Airflow workers.

In modern cloud‑native environments, every micro‑service instance starts from a common image layer. That layer is mapped read‑only, and the kernel relies on COW to give each container a private view without eagerly copying gigabytes of data. While the concept is simple, the actual implementation spans several subsystems, from the low‑level page‑fault handler to the high‑level fork path. This post unpacks the architecture, walks through the kernel’s COW mechanisms, and then shows production‑grade patterns that keep COW overhead under control.

Foundations of Copy‑on‑Write in Linux

Historical Context

Copy‑on‑Write originated in the early Unix fork implementation to avoid the O(N) cost of duplicating a process’s address space. The 4.2BSD kernel introduced the “shared page table” trick, and Linux inherited the idea in its very first releases. Over the decades the mechanism has been generalized: any mapping that is private (MAP_PRIVATE) can become a COW candidate, and the kernel now supports huge pages, NUMA awareness, and user‑space fault handling.

Page Tables and PTE Flags

At the hardware level, each virtual page maps to a physical frame via a page‑table entry (PTE). Linux adds two key bits:

  • PTE_READ – the page can be read.
  • PTE_WRITE – the page can be written.

When a page is COW‑eligible, the kernel clears PTE_WRITE while keeping PTE_READ. The CPU therefore raises a write‑protect fault (often a page‑fault with error code PF_W|PF_U) the first time a process attempts to write. The kernel then decides whether to make a private copy or to propagate the write to a shared backing store (e.g., a file).

The relevant flag in the Linux source is PAGE_COPY_ON_WRITE (defined in include/linux/mm.h). The flag is stored in the struct page and influences the pte_make_writable() helper.

Fault Path Architecture

Page Fault Entry Point

All faults converge on do_page_fault() defined in mm/memory.c. The function extracts the faulting address, the error code, and the current mm_struct. It then dispatches to a series of handlers based on the nature of the fault:

int do_page_fault(struct pt_regs *regs, unsigned long address, unsigned int error_code)
{
    /* ... */
    if (error_code & PF_WRITE) {
        return handle_cow_fault(vma, address, regs);
    }
    /* ... other handlers ... */
}

COW Handling in handle_cow_fault

handle_cow_fault() lives in mm/memory.c and performs the classic COW steps:

  1. Validate the VMA – ensure the faulting address falls within a vm_area_struct that permits writes (VM_WRITE).
  2. Locate the backing struct page – via follow_page() which walks the page tables without setting PTE_WRITE.
  3. Check the page’s reference count – if page_count(page) == 1, the page is already exclusive and can be made writable in‑place.
  4. Allocate a new page – using alloc_page() with the same gfp_mask as the original.
  5. Copy the contents – copy_page(new_page, old_page).
  6. Update the PTE – replace the old PTE with one that points to new_page and has PTE_WRITE set.
  7. Release the old page – put_page(old_page).

A simplified excerpt illustrates the core logic:

static int handle_cow_fault(struct vm_area_struct *vma,
                            unsigned long address,
                            struct pt_regs *regs)
{
    struct page *old_page, *new_page;
    pte_t *pte, entry;

    pte = get_user_pte(vma->vm_mm, address);
    entry = pte_get(pte);
    old_page = pte_page(entry);

    if (page_count(old_page) == 1) {
        /* Exclusive – just make it writable */
        set_pte_at(vma->vm_mm, address, pte,
                   pte_mkdirty(pte_mkwrite(entry)));
        return 0;
    }

    new_page = alloc_page(vma->vm_page_prot);
    if (!new_page)
        return -ENOMEM;

    copy_page(new_page, old_page);
    set_pte_at(vma->vm_mm, address, pte,
               mk_pte(new_page, vma->vm_page_prot));
    put_page(old_page);
    return 0;
}

The real kernel includes many edge‑case checks (e.g., handling VM_MAYSHARE, VM_DENYWRITE, or swap‑backed pages), but the skeleton above captures the essential pattern.

Interaction with mm_struct and vm_area_struct

mm_struct represents a process’s entire address space, while each contiguous region is described by a vm_area_struct (VMA). COW is only permitted in VMAs that have the VM_SHARED flag cleared and either VM_MAYWRITE or VM_WRITE set. The kernel tracks the number of COW pages per VMA via vma->vm_flags and vma->anon_vma. When a VMA is created with MAP_PRIVATE, the kernel automatically sets the VM_MAYWRITE flag but clears VM_SHARED, priming the region for COW.

Memory Management Subsystems Leveraging COW

fork and copy_process

The classic use‑case is fork(). The system call invokes copy_process() in kernel/fork.c, which performs a shallow copy of the parent’s mm_struct. The new child receives the same page tables, but the kernel marks all user pages as read‑only and increments their reference counts. No physical memory is copied at this point; the first write in either process triggers the fault path described earlier.

int copy_process(...){
    /* ... */
    mm = dup_mm(p->mm);
    if (!mm)
        return -ENOMEM;
    /* Mark all user pages read‑only for COW */
    mm->context.flags |= MMF_COW;
    /* ... */
}

Because fork is used heavily by web servers (e.g., nginx workers) and by container runtimes that spawn a new PID namespace, understanding the COW cost model directly impacts latency budgets.

mmap with MAP_PRIVATE

When a file is mapped with MAP_PRIVATE, the kernel creates a copy‑on‑write mapping. The file’s pages are initially mapped as read‑only; writes generate private copies that are never flushed back to the underlying file. This is the technique behind overlay filesystems (OverlayFS, AUFS) used by Docker and Kubernetes to provide per‑container write layers while sharing the base image.

OverlayFS and Container Copy‑on‑Write

OverlayFS stacks a lower read‑only layer (the container image) with an upper writable layer (a tmpfs). The upper layer holds only the pages that have been modified. Internally the VFS uses vfs_copy_file_range() and the same COW page‑fault machinery to materialize changes lazily. In a high‑throughput Kafka broker container, the majority of the page cache remains shared across hundreds of replicas, dramatically reducing RAM pressure.

Optimization Patterns in Production

Lazy Allocation and Hugepages

Hugepages (e.g., 2 MiB on x86‑64) reduce the number of page‑table entries and TLB misses, but they also increase the cost of a COW fault because copying a huge page is more expensive. A common pattern is to enable transparent hugepages (sysctl vm.nr_hugepages) for read‑only workloads while disabling them for memory‑intensive COW paths:

# Enable THP for anonymous memory, but keep it disabled for MAP_PRIVATE
echo always > /sys/kernel/mm/transparent_hugepage/enabled
echo madvise > /sys/kernel/mm/transparent_hugepage/defrag

When a COW fault hits a huge page, the kernel automatically falls back to a regular 4 KiB page if THP is set to madvise. This keeps latency predictable for latency‑sensitive services.

Reducing COW Churn with memfd_create and userfaultfd

  • memfd_create() creates an anonymous in‑memory file descriptor that can be mmap‑ed with MAP_PRIVATE. Because the backing store lives in RAM, the kernel can skip the disk‑I/O path and resolve COW faults purely in memory, which is useful for in‑process sandboxing (e.g., Firecracker VMs).

  • userfaultfd (available since Linux 4.3) lets a user‑space thread handle page‑faults. By pre‑populating pages with zeroed buffers or by serving data from a fast cache, you can eliminate the copy step entirely for known‑write patterns. Production teams have used this technique to accelerate large‑scale data pipelines in Airflow workers.

int fd = memfd_create("cow_buf", MFD_CLOEXEC);
ftruncate(fd, 256 * 1024 * 1024); // 256 MiB
void *addr = mmap(NULL, 256 * 1024 * 1024,
                  PROT_READ | PROT_WRITE,
                  MAP_PRIVATE, fd, 0);

Linux’s overcommit heuristics affect when the kernel permits a COW page to be materialized. The three modes are:

ModeMeaning
0Heuristic (default) – checks memory + swap.
1Always allow allocations (dangerous).
2Strict – allocation fails if it would exceed overcommit_ratio.

For services that allocate many short‑lived private pages (e.g., Java micro‑services), setting vm.overcommit_memory=2 together with a conservative vm.overcommit_ratio (e.g., 70) forces the kernel to reject allocations early, preventing OOM surprises after a massive COW cascade.

sysctl -w vm.overcommit_memory=2
sysctl -w vm.overcommit_ratio=70

Profiling COW with perf and ftrace

Identifying hot COW paths is essential. perf record -e page-faults,minor-faults,major-faults captures the rate of faults per process. Coupled with perf script you can locate the exact call stack leading to handle_cow_fault. For deeper tracing, ftrace can be enabled on the handle_cow_fault function:

echo function > /sys/kernel/debug/tracing/current_tracer
echo handle_cow_fault > /sys/kernel/debug/tracing/set_ftrace_filter
cat /sys/kernel/debug/tracing/trace | less

Typical production findings include:

  • Excessive COW caused by repeatedly opening the same large read‑only file with MAP_PRIVATE in a worker pool.
  • Unintended sharing of a tmpfs mount across containers, leading to massive page‑table duplication.

By fixing the root cause (e.g., switching to MAP_SHARED where writes are intentional, or pre‑populating a shared read‑only cache), you can cut COW faults by >80 %.

Key Takeaways

  • COW is a page‑fault‑driven lazy copy mechanism; the kernel only duplicates a page when a write occurs, keeping memory usage low for fork, MAP_PRIVATE, and container overlays.
  • The fault path (handle_cow_fault) is highly optimized but still incurs a full page copy, a TLB shoot‑down, and reference‑count gymnastics; each of these steps adds measurable latency on hot paths.
  • Production tuning revolves around three levers: (1) controlling when huge pages are used, (2) configuring overcommit policies to avoid late OOM, and (3) using user‑space fault handling (userfaultfd) or memfd_create to keep copies in RAM.
  • Visibility is key: perf and ftrace can surface unexpected COW churn, enabling you to refactor code (e.g., replace MAP_PRIVATE with MAP_SHARED for read‑only data) before it becomes a scalability bottleneck.
  • Container runtimes rely on COW at scale; understanding how OverlayFS maps lower‑layer pages helps you size node memory and tune vm.swappiness for bursty workloads like Kafka or Spark executors.

Further Reading