Build a Distributed Task Queue From Scratch: A Portfolio Project That Signals Senior Systems Skill

TL;DR — Build a distributed task queue from scratch in Go: a broker accepting tasks over HTTP, persistent storage via BoltDB, a worker pool with retry and dead-letter queues, and a web dashboard. This project demonstrates distributed systems design, concurrency, fault tolerance, and observability — exactly the skills hiring managers screen for in senior engineering roles.

A portfolio project that merely crunches numbers or renders a UI tells a hiring manager you can follow a tutorial. A project that implements a distributed task queue tells them you understand how real infrastructure works under load, how to handle failure, and how to design systems that persist state reliably. This guide walks you through building exactly that — a production-flavored task queue inspired by the architecture behind Celery and RabbitMQ, but stripped to the essentials so you can actually finish it.

Why This Project Stands Out on a CV

Most side projects demonstrate one skill: building a web app, say, or writing a data analysis script. A distributed task queue demonstrates at least five simultaneously, and each one maps directly to roles that are hard to fill.

Distributed Systems Design: You’re implementing a broker-worker topology with message passing, which signals you understand the same patterns that power Kafka, RabbitMQ, and AWS SQS.
Concurrency and Parallelism: The worker pool requires careful goroutine management, channel-based communication, and synchronization primitives — skills that separate mid-level from senior engineers.
Persistence and Durability: By storing tasks in BoltDB (an embedded ACID key-value store), you prove you understand write-ahead logging, transaction semantics, and crash recovery.
Fault Tolerance: Retry policies with exponential backoff and dead-letter queues demonstrate you’ve thought about what happens when things fail — a concern that separates toy projects from production systems.
Observability: A metrics endpoint and structured logging show you know how to instrument a system so operators can actually diagnose problems at 2 AM.

The roles this signals range from backend engineer and DevOps/platform engineer to distributed systems engineer and infrastructure developer. It is, in short, one of the highest-signal portfolio projects you can build because every architectural decision maps to a real production concern.

Architecture Overview

The system consists of four core components communicating over a combination of HTTP and an internal message bus. Here is the decomposition:

┌──────────────┐     HTTP POST      ┌──────────────────┐
│   Client /    │ ──────────────────▶│   Broker Server   │
│   CLI Tool    │                    │  (Go HTTP server) │
└──────────────┘                    └────────┬─────────┘
                                            │
                                     ┌───────▼────────┐
                                     │  BoltDB Store  │
                                     │  (persistent)  │
                                     └───────┬────────┘
                                             │
                                     ┌───────▼────────┐
                                     │   Worker Pool   │
                                     │  (N goroutines) │
                                     └───────┬────────┘
                                             │
                                     ┌───────▼────────┐
                                     │  Web Dashboard  │
                                     │  (metrics + UI) │
                                     └────────────────┘

Broker Server: An HTTP server that receives task submissions, validates them, persists them to BoltDB, and pushes them onto an internal channel consumed by workers. It exposes endpoints for submitting tasks, querying status, and retrieving metrics.
BoltDB Store: An embedded, ACID-compliant key-value database that provides durable task storage. Every task write is wrapped in a BoltDB transaction, ensuring that a crash does not lose in-flight work.
Worker Pool: A pool of goroutines that pull tasks from a buffered Go channel, execute them (user-defined handler functions), and report results back. Workers respect concurrency limits and implement retry logic with configurable backoff.
Web Dashboard: A lightweight HTTP handler serving a JSON API and a minimal HTML interface showing queue depth, completed/failed task counts, and worker health.

The data flow is: client submits a task → broker persists it → broker enqueues it on the work channel → a worker picks it up, executes it, and updates its status in BoltDB → the dashboard reflects the new state.

Building It Step by Step

We will build this in Go (1.22+). The full source is structured as a single module with packages for the broker, storage, workers, and dashboard.

Step 1: Initialize the module and dependencies

mkdir taskqueue && cd taskqueue
go mod init taskqueue
go get go.etcd.io/bbolt/v2
go get github.com/gorilla/mux

BoltDB provides the persistence layer; gorilla/mux gives us clean HTTP routing.

Step 2: Define the task model and storage layer

Create storage/storage.go:

package storage

import (
	"encoding/json"
	"fmt"
	"os"
	"time"

	"go.etcd.io/bbolt/v2"
)

const (
	dbFile       = "tasks.db"
	bucketTasks  = "tasks"
	bucketMeta   = "meta"
)

type TaskStatus string

const (
	StatusPending    TaskStatus = "pending"
	StatusCompleted  TaskStatus = "completed"
	StatusFailed     TaskStatus = "failed"
	StatusDLQ        TaskStatus = "dead_letter"
)

type Task struct {
	ID          string    `json:"id"`
	Payload     string    `json:"payload"`
	Handler     string    `json:"handler"`
	Retries     int       `json:"retries"`
	MaxRetries  int       `json:"max_retries"`
	Status      TaskStatus `json:"status"`
	CreatedAt   time.Time `json:"created_at"`
	UpdatedAt   time.Time `json:"updated_at"`
	Error       string    `json:"error,omitempty"`
}

type Store struct {
	db *bbolt.DB
}

func NewStore(path string) (*Store, error) {
	db, err := bbolt.Open(path, 0600, &bbolt.Options{Timeout: 1 * time.Second})
	if err != nil {
		return nil, err
	}
	err = db.Update(func(tx *bbolt.Tx) error {
		_, err := tx.CreateBucketIfNotExists([]byte(bucketTasks))
		if err != nil {
			return err
		}
		_, err = tx.CreateBucketIfNotExists([]byte(bucketMeta))
		return err
	})
	if err != nil {
		return nil, err
	}
	return &Store{db: db}, nil
}

func (s *Store) SaveTask(task *Task) error {
	return s.db.Update(func(tx *bbolt.Tx) error {
		b := tx.Bucket([]byte(bucketTasks))
		data, err := json.Marshal(task)
		if err != nil {
			return err
		}
		return b.Put([]byte(task.ID), data)
	})
}

func (s *Store) GetTask(id string) (*Task, error) {
	var task Task
	err := s.db.View(func(tx *bbolt.Tx) error {
		b := tx.Bucket([]byte(bucketTasks))
		data := b.Get([]byte(id))
		if data == nil {
			return fmt.Errorf("task not found")
		}
		return json.Unmarshal(data, &task)
	})
	return &task, err
}

func (s *Store) UpdateTask(task *Task) error {
	task.UpdatedAt = time.Now()
	return s.SaveTask(task)
}

func (s *Store) IncrementRetry(id string) error {
	return s.db.Update(func(tx *bbolt.Tx) error {
		b := tx.Bucket([]byte(bucketTasks))
		data := b.Get([]byte(id))
		if data == nil {
			return fmt.Errorf("task not found")
		}
		var task Task
		if err := json.Unmarshal(data, &task); err != nil {
			return err
		}
		task.Retries++
		if task.Retries >= task.MaxRetries {
			task.Status = StatusDLQ
		} else {
			task.Status = StatusPending
		}
		updated, _ := json.Marshal(task)
		return b.Put([]byte(id), updated)
	})
}

func (s *Store) ListTasks(status TaskStatus) ([]*Task, error) {
	var tasks []*Task
	err := s.db.View(func(tx *bbolt.Tx) error {
		b := tx.Bucket([]byte(bucketTasks))
		return b.ForEach(func(k, v []byte) error {
			var task Task
			if err := json.Unmarshal(v, &task); err != nil {
				return err
			}
			if status == "" || task.Status == status {
				tasks = append(tasks, &task)
			}
			return nil
		})
	})
	return tasks, err
}

The key design decision here is wrapping every mutation in a BoltDB Update transaction. This guarantees that even if the process crashes mid-write, the database remains in a consistent state — a property that many tutorial projects ignore entirely.

Step 3: Build the broker and worker pool

Create broker/broker.go:

package broker

import (
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"sync"
	"time"

	"taskqueue/storage"
)

type Broker struct {
	store      *storage.Store
	taskQueue  chan *storage.Task
	workers    int
	mu         sync.RWMutex
	metrics    map[string]int
}

func NewBroker(store *storage.Store, workers int, queueDepth int) *Broker {
	return &Broker{
		store:     store,
		taskQueue: make(chan *storage.Task, queueDepth),
		workers:   workers,
		metrics:   make(map[string]int),
	}
}

func (b *Broker) Start() {
	for i := 0; i < b.workers; i++ {
		go b.worker(i)
	}
	log.Printf("Broker started with %d workers, queue depth %d", b.workers, cap(b.taskQueue))
}

func (b *Broker) worker(id int) {
	for task := range b.taskQueue {
		log.Printf("Worker %d processing task %s", id, task.ID)
		err := b.executeTask(task)
		if err != nil {
			log.Printf("Task %s failed: %v", task.ID, err)
			task.Error = err.Error()
			if task.Retries < task.MaxRetries {
				b.store.IncrementRetry(task.ID)
				// Re-enqueue with backoff
				go func(t *storage.Task) {
					time.Sleep(time.Duration(t.Retries) * time.Second * 2)
					b.taskQueue <- t
				}(task)
			} else {
				task.Status = storage.StatusDLQ
				b.store.UpdateTask(task)
			}
		} else {
			task.Status = storage.StatusCompleted
			b.store.UpdateTask(task)
		}
	}
}

func (b *Broker) executeTask(task *storage.Task) error {
	// In production, this dispatches to a registered handler.
	// For this guide, we simulate execution.
	if task.Handler == "fail" {
		return fmt.Errorf("simulated failure")
	}
	time.Sleep(100 * time.Millisecond)
	return nil
}

func (b *Broker) SubmitTask(w http.ResponseWriter, r *http.Request) {
	var task storage.Task
	if err := json.NewDecoder(r.Body).Decode(&task); err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}
	task.ID = fmt.Sprintf("task-%d", time.Now().UnixNano())
	task.Status = storage.StatusPending
	task.CreatedAt = time.Now()
	task.UpdatedAt = time.Now()
	if task.MaxRetries == 0 {
		task.MaxRetries = 3
	}

	if err := b.store.SaveTask(&task); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	b.taskQueue <- &task

	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(map[string]string{"id": task.ID, "status": "queued"})
}

func (b *Broker) Metrics(w http.ResponseWriter, r *http.Request) {
	pending, _ := b.store.ListTasks(storage.StatusPending)
	completed, _ := b.store.ListTasks(storage.StatusCompleted)
	failed, _ := b.store.ListTasks(storage.StatusDLQ)

	b.mu.Lock()
	defer b.mu.Unlock()
	b.metrics["pending"] = len(pending)
	b.metrics["completed"] = len(completed)
	b.metrics["dead_letter"] = len(failed)

	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(b.metrics)
}

The worker function is the heart of the system. Each worker runs in its own goroutine, pulling from the buffered channel. When a task fails, the retry logic increments the counter in BoltDB and re-enqueues the task with a linear backoff delay. After exhausting retries, the task moves to the dead-letter queue — a pattern borrowed directly from AWS SQS and used in production systems at scale.

Step 4: Wire up the HTTP server and dashboard

Create main.go:

package main

import (
	"log"
	"net/http"

	"broker"
	"storage"

	"github.com/gorilla/mux"
)

func main() {
	store, err := storage.NewStore(storage.DBFile)
	if err != nil {
		log.Fatal(err)
	}
	defer store.db.Close()

	br := broker.NewBroker(store, 4, 100)
	br.Start()

	r := mux.NewRouter()
	r.HandleFunc("/tasks", br.SubmitTask).Methods("POST")
	r.HandleFunc("/metrics", br.Metrics).Methods("GET")
	r.PathPrefix("/").Handler(http.FileServer(http.Dir("./dashboard")))

	log.Println("Server listening on :8080")
	log.Fatal(http.ListenAndServe(":8080", r))
}

The dashboard directory contains a minimal index.html that fetches /metrics and renders queue statistics. This is intentionally lightweight — the focus is on the backend infrastructure, not the frontend.

Running and Testing It

Build and start the server:

go build -o taskqueue .
./taskqueue

Submit a task via curl:

curl -X POST http://localhost:8080/tasks \
  -H "Content-Type: application/json" \
  -d '{"payload": "process_image_001", "handler": "image_processor", "max_retries": 3}'

Submit a deliberately failing task to test the dead-letter queue:

curl -X POST http://localhost:8080/tasks \
  -H "Content-Type: application/json" \
  -d '{"payload": "will_fail", "handler": "fail", "max_retries": 2}'

Check metrics:

curl http://localhost:8080/metrics
# {"completed":1,"dead_letter":1,"pending":0}

To verify persistence, kill the server with Ctrl+C, restart it, and query the BoltDB store. Tasks survive the restart because every state change is committed to disk before the channel receives the next task. You can write a quick integration test:

func TestTaskPersistence(t *testing.T) {
	store, err := storage.NewStore("test.db")
	if err != nil { t.Fatal(err) }
	defer os.Remove("test.db")

	task := &storage.Task{ID: "t1", Status: storage.StatusPending, MaxRetries: 2}
	if err := store.SaveTask(task); err != nil { t.Fatal(err) }

	retrieved, err := store.GetTask("t1")
	if err != nil { t.Fatal(err) }
	if retrieved.Status != storage.StatusPending {
		t.Fatalf("expected pending, got %s", retrieved.Status)
	}
}

Run it with go test ./storage/... to confirm durability holds.

Extending It: Your Roadmap to Senior-Level

The base project above is functional and impressive. But to truly signal senior-level engineering, extend it along these axes:

Add Horizontal Scaling with gRPC Service Discovery — Replace the in-memory channel with a Redis-backed or NATS-backed pub/sub so multiple broker instances can share the work queue. This demonstrates you understand how to scale beyond a single process, which is the first question in any distributed systems interview.
Implement a Write-Ahead Log (WAL) for Crash Recovery — Before each BoltDB write, append the task to an append-only WAL file. On restart, replay the log to reconstruct any tasks that were in-flight during the crash. This is the same technique used by Kafka and PostgreSQL, and it proves you understand durability guarantees at a deeper level than “I used a database.”
Add Structured Observability with OpenTelemetry — Instrument every handler and worker with OpenTelemetry traces and metrics, exporting to a local Jaeger or Grafana Tempo instance. Include request latency histograms, error rates, and queue depth gauges. Observability is not a feature; it is the mechanism that lets you know your system is healthy, and every production system requires it.
Implement Circuit Breaker Pattern for External Handlers — If tasks call external services (HTTP APIs, databases), add a circuit breaker (using github.com/sony/gobreaker) that stops dispatching to a failing service after N consecutive errors, and enters a half-open state after a timeout to test recovery. This is a textbook fault-tolerance pattern used in Netflix’s Hystrix and Spring Cloud.
Add Priority Queues and Scheduling — Extend BoltDB to support multiple priority buckets and delayed execution (tasks that become eligible only after a specified time). This mirrors the delayed-message feature in RabbitMQ and demonstrates you can reason about ordering guarantees and scheduling semantics.
Benchmark Under Load with go test -bench and k6 — Write Go benchmark tests for the broker’s throughput (tasks/second) and use k6 to simulate hundreds of concurrent HTTP clients. Profile with pprof to identify bottlenecks in the channel, the BoltDB transaction layer, or the JSON serialization. Benchmarking transforms a working project into a measurable one — and hiring managers value engineers who can prove their system performs, not just claim it does.

Each of these extensions maps to a real production concern and gives you a concrete story to tell in interviews: “I implemented a circuit breaker because our external API was timing out under load, and here is the data showing it reduced error rates by 94%.”

Why This Project Stands Out on a CV#

Architecture Overview#

Building It Step by Step#

Step 1: Initialize the module and dependencies#

Step 2: Define the task model and storage layer#

Step 3: Build the broker and worker pool#

Step 4: Wire up the HTTP server and dashboard#

Running and Testing It#

Extending It: Your Roadmap to Senior-Level#

Further Reading#