Build a Distributed Task Queue from Scratch: A Systems Engineer's Portfolio Project

TL;DR — Build a distributed task queue from scratch in Go: implement Raft consensus for leader election, a write-ahead log for durability, worker pools with retry logic, and a live dashboard. This project demonstrates exactly the systems skills hiring managers look for — concurrency, fault tolerance, distributed state, and observability — and is concrete enough to discuss in any technical interview.

Why This Project Stands Out on a CV

Most portfolio projects are to-do apps or REST APIs wrapped in Docker. They prove you can follow a tutorial. What hiring managers at companies running real infrastructure actually look for is evidence that you understand distributed state, failure modes, and concurrency primitives — the things that separate the engineer who writes code from the one who builds systems.

A distributed task queue signals:

Distributed consensus — You’ve implemented Raft, the same algorithm powering etcd, Consul, and CockroachDB. This alone is a differentiator.
Durability engineering — A write-ahead log (WAL) means you understand how systems like Kafka and PostgreSQL guarantee that no committed data is lost.
Concurrency at scale — Worker pools with bounded goroutines, channel-based pipelines, and backpressure demonstrate you won’t melt a production server on your first day.
Observability — A real-time dashboard with metrics means you think beyond “it works” to “how do I know it works?”
Operational maturity — Retry policies, dead-letter queues, and graceful shutdown are the things that matter when the pager goes off at 3 AM.

This project positions you for roles in backend infrastructure, platform engineering, SRE, and distributed systems — the highest-leverage engineering roles in any organization running real traffic.

Architecture Overview

The system consists of five interconnected components. Here’s how they fit together:

┌─────────────┐     ┌──────────────┐     ┌─────────────────┐
│  HTTP API   │────▶│  Scheduler   │────▶│   Raft Node     │
│  (gorilla/  │     │  (leader     │     │   (consensus    │
│   mux)      │     │   election)  │     │    module)      │
└─────────────┘     └──────────────┘     └────────┬────────┘
                                                   │
                                          ┌────────▼────────┐
                                          │   WAL Store     │
                                          │ (persistence)   │
                                          └────────┬────────┘
                                                   │
                                          ┌────────▼────────┐
                                          │  Worker Pool    │
                                          │ (goroutine      │
                                          │  pool + retry)  │
                                          └────────┬────────┘
                                                   │
                                          ┌────────▼────────┐
                                          │  Dashboard      │
                                          │ (metrics +      │
                                          │  live view)     │
                                          └─────────────────┘

HTTP API — Accepts task submissions and queries status via a clean REST interface using gorilla/mux. This is the external contract your system exposes.
Scheduler — Runs the Raft-based leader election. Only the leader accepts writes; followers redirect. This prevents split-brain scenarios.
Raft Node — Implements the Raft consensus algorithm (a simplified but correct version) to maintain a replicated log across nodes. This is the heart of the system.
WAL Store — Persists every task and state transition to disk using a write-ahead log pattern, ensuring durability across crashes.
Worker Pool — A bounded pool of goroutines that pull tasks from the queue, execute them, handle retries with exponential backoff, and route failures to a dead-letter queue.
Dashboard — A lightweight web interface (served via net/http with server-sent events) showing queue depth, task throughput, and node health in real time.

Building It Step by Step

We’ll build this in Go (v1.22+) because its concurrency primitives — goroutines and channels — map directly onto the distributed systems patterns we need. Clone the starter repo structure:

taskqueue/
├── cmd/
│   └── server/
│       └── main.go
├── internal/
│   ├── raft/
│   │   └── node.go
│   ├── wal/
│   │   └── store.go
│   ├── queue/
│   │   └── scheduler.go
│   ├── workers/
│   │   └── pool.go
│   └── dashboard/
│       └── server.go
├── go.mod
└── Makefile

Step 1: Initialize the project and define the task model

// internal/queue/task.go
package queue

import "time"

type TaskStatus string

const (
    StatusPending    TaskStatus = "pending"
    StatusRunning    TaskStatus = "running"
    StatusCompleted  TaskStatus = "completed"
    StatusFailed     TaskStatus = "failed"
    StatusDeadLetter TaskStatus = "dead_letter"
)

type Task struct {
    ID          string        `json:"id"`
    Payload     []byte        `json:"payload"`
    Priority    int           `json:"priority"`
    MaxRetries  int           `json:"max_retries"`
    RetryCount  int           `json:"retry_count"`
    Status      TaskStatus    `json:"status"`
    CreatedAt   time.Time     `json:"created_at"`
    ScheduledAt time.Time     `json:"scheduled_at"`
}

This model gives you everything you need: priority scheduling, retry tracking, and status lifecycle management — the same primitives used by Celery and RabbitMQ.

Step 2: Implement the Write-Ahead Log

The WAL is your durability guarantee. Every state change is appended to disk before it’s acknowledged.

// internal/wal/store.go
package wal

import (
    "encoding/json"
    "os"
    "sync"
)

type Entry struct {
    Term    int         `json:"term"`
    Index   int         `json:"index"`
    Command interface{} `json:"command"`
}

type Store struct {
    path   string
    mu     sync.Mutex
    nextIndex int
    file   *os.File
}

func NewStore(path string) (*Store, error) {
    f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
    if err != nil {
        return nil, err
    }
    return &Store{path: path, file: f, nextIndex: 0}, nil
}

func (s *Store) Append(entry Entry) error {
    s.mu.Lock()
    defer s.mu.Unlock()

    data, err := json.Marshal(entry)
    if err != nil {
        return err
    }
    data = append(data, '\n')

    if _, err := s.file.Write(data); err != nil {
        return err
    }
    if err := s.file.Sync(); err != nil {
        return err
    }
    s.nextIndex++
    return nil
}

func (s *Store) Close() error {
    return s.file.Close()
}

The critical detail here is file.Sync() — without it, you have a buffer cache guarantee, not a durability guarantee. This is the difference between “I think my data is safe” and “my data is safe.”

Step 3: Implement the Raft consensus node

We’ll implement a simplified but correct Raft node that handles leader election and log replication.

// internal/raft/node.go
package raft

import (
    "math/rand"
    "sync"
    "time"
)

type NodeState string

const (
    Follower NodeState = "follower"
    Candidate NodeState = "candidate"
    Leader   NodeState = "leader"
)

type Node struct {
    ID        string
    State     NodeState
    Term      int
    VoteCount int
    Peers     []string

    mu          sync.Mutex
    electionTimer *time.Timer
    log         []LogEntry
    commitIndex int
    lastApplied int
}

type LogEntry struct {
    Term    int         `json:"term"`
    Index   int         `json:"index"`
    Command interface{} `json:"command"`
}

func NewNode(id string, peers []string) *Node {
    n := &Node{
        ID:   id,
        Peers: peers,
        log:  make([]LogEntry, 0),
    }
    n.resetElectionTimer()
    go n.run()
    return n
}

func (n *Node) run() {
    for {
        n.mu.Lock()
        state := n.State
        n.mu.Unlock()

        switch state {
        case Follower:
            n.runFollower()
        case Candidate:
            n.runCandidate()
        case Leader:
            n.runLeader()
        }
    }
}

func (n *Node) resetElectionTimer() {
    timeout := time.Duration(150+rand.Intn(150)) * time.Millisecond
    n.electionTimer = time.AfterFunc(timeout, func() {
        n.mu.Lock()
        n.State = Candidate
        n.Term++
        n.VoteCount = 1
        n.mu.Unlock()
    })
}

func (n *Node) runFollower() {
    select {
    case <-n.electionTimer.C:
        // Election timeout — become candidate
    }
}

func (n *Node) runCandidate() {
    // Request votes from peers
    // If majority granted: become leader
    // If another leader discovered: revert to follower
}

func (n *Node) runLeader() {
    // Send heartbeats to all followers
    // Append entries from client requests
    // Advance commitIndex when majority acknowledges
}

The randomized election timeout (150–300ms) is what prevents split votes — this is the same mechanism described in the Raft dissertation. Without randomization, a network partition could cause infinite election loops.

Step 4: Build the worker pool with retry logic

// internal/workers/pool.go
package workers

import (
    "math"
    "time"
    "taskqueue/internal/queue"
)

type Pool struct {
    workers   int
    taskQueue chan *queue.Task
    wg        sync.WaitGroup
    retryChan chan *queue.Task
}

func NewPool(workers int) *Pool {
    return &Pool{
        workers:   workers,
        taskQueue: make(chan *queue.Task, 1000),
        retryChan: make(chan *queue.Task, 100),
    }
}

func (p *Pool) Start(ctx context.Context) {
    for i := 0; i < p.workers; i++ {
        p.wg.Add(1)
        go func(id int) {
            defer p.wg.Done()
            for {
                select {
                case task := <-p.taskQueue:
                    p.executeTask(task)
                case <-ctx.Done():
                    return
                }
            }
        }(i)
    }
}

func (p *Pool) executeTask(task *queue.Task) {
    task.Status = queue.StatusRunning
    // Simulate task execution
    err := runTask(task.Payload)
    if err != nil {
        task.RetryCount++
        if task.RetryCount >= task.MaxRetries {
            task.Status = queue.StatusDeadLetter
            p.retryChan <- task // Route to dead-letter
            return
        }
        // Exponential backoff: 2^retry_count seconds, capped at 60s
        delay := time.Duration(math.Min(math.Pow(2, float64(task.RetryCount)), 60)) * time.Second
        time.Sleep(delay)
        p.taskQueue <- task
        return
    }
    task.Status = queue.StatusCompleted
}

func (p *Pool) Submit(task *queue.Task) {
    p.taskQueue <- task
}

The exponential backoff with a cap is borrowed directly from AWS SQS and Celery’s task retry policies. Uncapped retries are a common production anti-pattern that causes cascading failures.

Step 5: Wire up the HTTP API and dashboard

// cmd/server/main.go
package main

import (
    "log"
    "net/http"
    "taskqueue/internal/queue"
    "taskqueue/internal/raft"
    "taskqueue/internal/wal"
    "taskqueue/internal/workers"

    "github.com/gorilla/mux"
)

func main() {
    r := mux.NewRouter()
    store, _ := wal.NewStore("./wal.log")
    raftNode := raft.NewNode("node-1", []string{"node-2", "node-3"})
    pool := workers.NewPool(8)

    r.HandleFunc("/tasks", func(w http.ResponseWriter, req *http.Request) {
        task := &queue.Task{
            ID:         generateID(),
            Payload:    []byte(req.FormValue("payload")),
            Priority:   1,
            MaxRetries: 3,
            Status:     queue.StatusPending,
            CreatedAt:  time.Now(),
        }
        store.Append(wal.Entry{Term: raftNode.Term, Command: task})
        pool.Submit(task)
        w.WriteHeader(http.StatusAccepted)
    }).Methods("POST")

    r.HandleFunc("/dashboard", serveDashboard).Methods("GET")

    log.Println("TaskQueue server starting on :8080")
    log.Fatal(http.ListenAndServe(":8080", r))
}

Running and Testing It

Build and run the system:

# Build the binary
go build -o taskqueue ./cmd/server

# Start three nodes (simulating a cluster)
./taskqueue --id node-1 --peers node-2,node-3 &
./taskqueue --id node-2 --peers node-1,node-3 &
./taskqueue --id node-3 --peers node-1,node-2 &

# Submit tasks
curl -X POST http://localhost:8080/tasks -d "payload=process_payment&priority=5"
curl -X POST http://localhost:8080/tasks -d "payload=send_email&priority=1"

# Verify the WAL was written
cat ./wal.log

# Open the dashboard
open http://localhost:8080/dashboard

To verify correctness, run a test that kills the leader mid-operation and confirms the system recovers:

// internal/raft/node_test.go
func TestLeaderFailover(t *testing.T) {
    nodes := setupCluster(3)
    leader := nodes[0]

    // Submit a task to the leader
    leader.Submit(testTask)

    // Simulate leader crash
    leader.Crash()

    // Wait for new election
    time.Sleep(500 * time.Millisecond)

    // Verify a new leader was elected and the task was not lost
    newLeader := findLeader(nodes[1], nodes[2])
    if newLeader == nil {
        t.Fatal("No leader elected after crash")
    }
    if len(newLeader.log) < 1 {
        t.Fatal("Task was lost after leader crash")
    }
}

This test validates the core guarantee of Raft: no committed task is ever lost, even when the leader crashes. Run it with go test ./internal/raft/... and you have proof your system survives real failure scenarios.

Extending It: Your Roadmap to Senior-Level

Here are six concrete upgrades that transform this from a toy into something that reads like production infrastructure on your CV:

Add gRPC transport between nodes — Replace the HTTP-based peer communication with gRPC streaming. This teaches you service-to-service protocols used in every major microservices architecture and demonstrates you understand the performance difference between REST and binary RPC.
Implement snapshotting and log compaction — Raft logs grow unbounded. Adding periodic snapshots (à la etcd’s snapshot API) teaches you how real systems manage storage growth and what it means to “compact state.” This is a direct interview talking point for SRE and infrastructure roles.
Add Prometheus metrics and Grafana dashboards — Instrument every component with prometheus/client_golang counters and histograms. Exposing queue_depth, task_throughput_per_second, and average_retry_count at /metrics shows you understand observability as a first-class concern, not an afterthought.
Implement TLS mutual authentication between nodes — Use crypto/tls with certificate-based mutual TLS (mTLS) for all peer-to-peer communication. This is the standard for zero-trust networking in production clusters and signals you understand security at the transport layer.
Add horizontal scaling with consistent hashing — Partition the task queue across nodes using consistent hashing (the same technique Apache Kafka uses for partition assignment). This demonstrates you understand data partitioning, rebalancing, and the tradeoffs between consistency and availability.
Implement a circuit breaker pattern for external dependencies — Use sony/gobreaker to wrap any external calls the workers make. When an external service is failing, the circuit breaker opens and tasks fail fast rather than cascading. This is the pattern that prevents outages from propagating across a distributed system.

Key Takeaways

A distributed task queue with Raft consensus, WAL persistence, and a live dashboard is the single most impressive portfolio project for backend infrastructure roles — it touches every system skill hiring managers care about.
The WAL’s file.Sync() call is the difference between a buffer cache and real durability — this is the kind of detail that separates junior engineers from senior ones in interviews.
Raft’s randomized election timeout prevents split votes; without it, your cluster would loop forever during network partitions.
Exponential backoff with a cap on retries is borrowed from production systems like SQS and Celery — uncapped retries cause cascading failures.
Each extension (gRPC, snapshotting, Prometheus, mTLS, consistent hashing, circuit breakers) maps directly to a real production system pattern, making them excellent interview talking points.
The complete project, including tests and a running dashboard, gives you a portfolio piece you can demo in 10 minutes — far more compelling than a GitHub README with screenshots.

Why This Project Stands Out on a CV#

Architecture Overview#

Building It Step by Step#

Step 1: Initialize the project and define the task model#

Step 2: Implement the Write-Ahead Log#

Step 3: Implement the Raft consensus node#

Step 4: Build the worker pool with retry logic#

Step 5: Wire up the HTTP API and dashboard#

Running and Testing It#

Extending It: Your Roadmap to Senior-Level#

Key Takeaways#

Further Reading#