TL;DR — You will implement a LoRA adapter that fuses low‑rank projections and merges weights using only Python and NumPy. The resulting module plugs into any transformer block, reduces parameter count, and can be demonstrated with a simple test harness, making it a strong signal of systems skill on a CV.

In the race to fine‑tune large language models, parameter‑efficient methods have become a must‑have skill for engineers who want to show they can ship models under budget. This post walks you through building a LoRA adapter from scratch, focusing on the fused low‑rank projection and weight‑merging tricks that make it both memory‑efficient and fast. By the end you will have a reusable Python module, a set of tests, and a clear story to tell recruiters about your ability to design and optimize machine‑learning systems.

Why This Project Stands Out on a CV

  • Low‑Rank Adaptation (LoRA) expertise – you will understand and implement the core mathematics behind LoRA, a technique now used in production at companies like OpenAI and Meta.
  • Fused projection & weight merging – demonstrates an ability to optimize inference pipelines, a skill that maps directly to roles in ML infrastructure and model serving.
  • Pure‑Python implementation – shows you can build and prototype without relying on heavy frameworks, highlighting strong fundamentals and debugging ability.
  • Test‑driven development – includes a runnable test suite, proving you can validate correctness and performance before shipping.
  • Extensible architecture – the design naturally accommodates persistence, scaling, and observability upgrades, signaling readiness for senior‑level system design conversations.

Architecture Overview

The project is organized into three logical components:

  1. LowRankProjection – a lightweight module that stores two small matrices (A \in \mathbb{R}^{d \times r}) and (B \in \mathbb{R}^{r \times d}) and computes the low‑rank update (\Delta W = BA).
  2. LoRAAdapter – wraps a target weight matrix (e.g., a transformer attention projection) and injects the low‑rank update during the forward pass. It supports a fused mode where the update is added directly to the original weight before the matrix multiplication, reducing memory traffic.
  3. WeightMerger – provides a method to permanently fold (\Delta W) into the base weight, useful for exporting a merged model or for simplifying deployment.

A simplified text diagram:

Input x
   │
   ▼
┌─────────────┐
│ Base Weight  │  (W₀)
└──────┬──────┘
       │
       ▼
┌─────────────┐   ┌─────────────┐
│ Low‑Rank A   │──▶│ Low‑Rank B   │
└──────┬──────┘   └──────┬──────┘
       │                 │
       ▼                 ▼
   ┌─────────────────────────┐
   │  ΔW = B @ A              │
   └─────────────┬───────────┘
                 │
                 ▼
         ┌───────────────┐
         │ W = W₀ + ΔW    │
         └───────┬───────┘
                 │
                 ▼
           Output y

Building It Step by Step

Step 1: Set up the environment

python -m venv lora-env
source lora-env/bin/activate
pip install numpy pytest

Step 2: Implement the LowRankProjection

import numpy as np

class LowRankProjection:
    """
    Stores low-rank matrices A (d x r) and B (r x d).
    The update is ΔW = B @ A.
    """
    def __init__(self, d: int, r: int, seed: int = 42):
        self.d = d
        self.r = r
        rng = np.random.default_rng(seed)
        # Initialize with small random values
        self.A = rng.standard_normal((d, r)) * 0.02
        self.B = rng.standard_normal((r, d)) * 0.02

    def delta_W(self) -> np.ndarray:
        """Return the low-rank weight update."""
        return self.B @ self.A

Step 3: Create the LoRAAdapter

class LoRAAdapter:
    """
    Wraps a base weight matrix and injects the low-rank update.
    Supports fused mode (add ΔW before matmul) and separate mode.
    """
    def __init__(self, base_W: np.ndarray, lora: LowRankProjection, fused: bool = True):
        self.base_W = base_W
        self.lora = lora
        self.fused = fused

    def forward(self, x: np.ndarray) -> np.ndarray:
        """
        Compute y = x @ W_eff where W_eff = base_W + ΔW (if fused)
        or y = x @ base_W + x @ ΔW (if separate).
        """
        if self.fused:
            # Fuse ΔW into base_W for a single matmul
            W_eff = self.base_W + self.lora.delta_W()
            return x @ W_eff
        else:
            # Separate path – useful for debugging
            base_out = x @ self.base_W
            lora_out = x @ (self.lora.delta_W())
            return base_out + lora_out

Step 4: Add weight merging capability

def merge_lora(adapter: LoRAAdapter) -> np.ndarray:
    """
    Permanently fold the low-rank update into the base weight.
    Returns the merged weight matrix.
    """
    merged = adapter.base_W + adapter.lora.delta_W()
    return merged

Step 5: Fused forward pass with a real transformer‑style projection

def fused_attention_projection(x: np.ndarray,
                               W_q: np.ndarray,
                               W_k: np.ndarray,
                               W_v: np.ndarray,
                               lora_q: LowRankProjection,
                               lora_k: LowRankProjection,
                               lora_v: LowRankProjection) -> tuple:
    """
    Example of applying LoRA to query, key, value projections in a self‑attention layer.
    Returns (Q, K, V) after fused low‑rank updates.
    """
    # Fused query
    Q = x @ (W_q + lora_q.delta_W())
    # Fused key
    K = x @ (W_k + lora_k.delta_W())
    # Fused value
    V = x @ (W_v + lora_v.delta_W())
    return Q, K, V

Step 6: Write a test harness

def test_lora_adapter():
    d, r = 8, 2
    base_W = np.random.randn(d, d)
    lora = LowRankProjection(d, r)
    adapter = LoRAAdapter(base_W, lora, fused=True)

    x = np.random.randn(4, d)  # batch of 4
    y = adapter.forward(x)

    # Verify shape
    assert y.shape == (4, d), f"Unexpected output shape {y.shape}"

    # Verify that fused and separate produce same result (within tolerance)
    adapter_separate = LoRAAdapter(base_W, lora, fused=False)
    y_sep = adapter_separate.forward(x)
    np.testing.assert_allclose(y, y_sep, rtol=1e-5, atol=1e-8)

    print("All LoRA adapter tests passed.")

if __name__ == "__main__":
    test_lora_adapter()

Running and Testing It

  1. Save the code above in a file named lora_adapter.py.
  2. Run the built‑in test:
python lora_adapter.py

You should see All LoRA adapter tests passed. printed to the console.

  1. For a more thorough validation, extend the test suite with pytest:
# test_lora_pytest.py
import numpy as np
from lora_adapter import LowRankProjection, LoRAAdapter

def test_delta_W_shape():
    lora = LowRankProjection(10, 3)
    assert lora.delta_W().shape == (10, 10)

def test_fused_matches_separate():
    d, r = 6, 2
    base = np.random.randn(d, d)
    lora = LowRankProjection(d, r)
    fused = LoRAAdapter(base, lora, fused=True)
    separate = LoRAAdapter(base, lora, fused=False)
    x = np.random.randn(5, d)
    np.testing.assert_allclose(fused.forward(x), separate.forward(x), rtol=1e-5)

Run with:

pytest -q test_lora_pytest.py

A passing test suite proves that the fused projection and the separate path are numerically equivalent, and that the low‑rank update is applied correctly.

Extending It: Your Roadmap to Senior‑Level

  1. Persistence with joblib – serialize the LowRankProjection and LoRAAdapter objects to disk so you can reload trained adapters without retraining, which is essential for model versioning in production.
  2. Horizontal scaling via Dask or Ray – distribute the low‑rank updates across multiple workers, enabling fine‑tuning on datasets that exceed a single machine’s memory.
  3. Observability with Prometheus – expose metrics such as lora_delta_norm, forward_latency, and memory_usage as Prometheus gauges, then visualize them in Grafana to detect drift or performance regressions.
  4. Fault‑tolerant training loops – wrap the training step in retry logic (e.g., tenacity) and checkpoint after each epoch, ensuring that a node failure does not lose weeks of work.
  5. Benchmarking harness – use timeit and cProfile to compare fused vs. separate forward passes on realistic batch sizes, producing a performance report that can be included in a design document.
  6. Integration with Hugging Face transformers – replace a transformer’s linear layer with your LoRAAdapter using the peft library’s interface, demonstrating that the custom implementation can slot into an industry‑standard pipeline.

Key Takeaways

  • You can build a functional LoRA adapter from scratch using only NumPy, covering low‑rank projection, fused inference, and weight merging.
  • The implementation is testable with pytest, giving you a concrete artifact to showcase on a portfolio or GitHub profile.
  • The architecture is deliberately modular, making it straightforward to add persistence, scaling, observability, and fault tolerance—skills that map directly to senior engineering roles.
  • By framing the project around production concerns (memory efficiency, speed, reliability), you position yourself as someone who can ship ML systems, not just prototype models.

Further Reading