TL;DR — AI engineering is the discipline of turning models into reliable products. That means owning the data, the training loop, the eval harness, the serving surface, and the monitoring that catches silent regressions. This post lays out the end-to-end blueprint most tutorials skip, with concrete patterns from real production stacks like those running on Kafka, Airflow, and Postgres.
Why “AI Engineering” Means More Than Prompting
There’s a difference between someone who can call a model API and someone who can put a model behind a $20M feature. The first is a skill; the second is a discipline. AI engineering is the discipline.
The core problem: models are stochastic, data drifts, GPUs are expensive, latency budgets are unforgiving, and the failure modes are silent. A well-tuned notebook can degrade 12% in production without anyone noticing for six weeks. The engineer who builds the system has to design for that — not just for the happy path on a held-out test set.
In practice, that means treating the model as one component in a larger system: data pipelines that version themselves, training runs that are reproducible, evaluation harnesses that gate releases, serving layers that handle retries and fallbacks, and observability that closes the loop back to data. Each of these layers has been worked out in other engineering domains — the MLOps community has consolidated the patterns — but AI systems have unique quirks worth calling out.
The Five-Layer Production Stack
Every shipped AI system I’ve worked on decomposes into roughly the same five layers. The terminology shifts, but the shape holds:
- Data layer — collection, validation, versioning, and feature stores.
- Training layer — experiment tracking, reproducible runs, and artifact management.
- Evaluation layer — offline metrics, online evals, and release gates.
- Serving layer — inference APIs, batching, caching, and fallbacks.
- Observability layer — logging, drift detection, and feedback loops.
These map onto the well-known MLOps stack diagram that emerged from the community around 2020, but with one important addition: the eval and observability layers are first-class. In older ML stacks they were bolted on after the fact, and that’s why most production models quietly rot.
A Concrete Example: Recommendation System
Let’s anchor the rest of the post in a running example: building a personalized content recommendation system similar to what powers feeds at companies like Netflix or Spotify. This isn’t a toy — it has all five layers, real data volume, and real consequences when it breaks. The patterns here transfer to RAG systems, classification services, and generative pipelines; only the specifics change.
Layer 1: The Data Foundation
Models are downstream of data. If your data layer is sloppy, nothing else matters.
Data Collection and Lineage
Every training example should be traceable back to its source. For recommendations, this means logging every impression, click, dismiss, and dwell event with a stable user ID, item ID, timestamp, and a hash of the feature snapshot used at serve time. A common pattern is to stream these events into Apache Kafka and then sink them into a warehouse like BigQuery or Snowflake.
Why Kafka and not just write straight to a database? Decoupling. The serving layer doesn’t care about batch jobs; the training layer doesn’t care about serving traffic. A topic per event stream lets each consumer evolve independently. As Confluent’s documentation puts it, Kafka is a durable, replayable log — and replayability is what lets you reconstruct training datasets deterministically.
Data Validation
Raw event streams are dirty. Users with no profile, items with no metadata, duplicate events from retries, and timezone-bad timestamps are all normal. You want to catch this at ingestion, not at training time when it’s already too late to fix cleanly.
A practical approach uses something like Great Expectations or Pandera to assert schema and value ranges:
import pandera as pa
events_schema = pa.DataFrameSchema({
"user_id": pa.Column(str, pa.str_matches(r"^u_[a-f0-9]{16}$"), nullable=False),
"item_id": pa.Column(str, nullable=False),
"event_type": pa.Column(str, isin=["impression", "click", "dismiss", "dwell"]),
"timestamp": pa.Column("datetime64[ns]"),
"feature_snapshot_hash": pa.Column(str, pa.str_length(64)),
})
Run this on every batch landing in the warehouse. If a column’s null rate exceeds 1%, fail the job and page someone. Garbage in, garbage out — but worse, garbage that trained the model you shipped last Tuesday.
Feature Stores
Once you have validated events, the next question is: how do you serve the same features at training time and inference time? If those numbers disagree, your offline metrics are lies.
A feature store — Feast, Tecton, or a hand-rolled version on Postgres — solves this by storing features in two forms: a historical table keyed by timestamp for training, and a low-latency online store (Redis or DynamoDB) for inference. The same code computes both. This is the point-in-time join pattern — and it is non-negotiable for any system that wants to trust its offline numbers.
Layer 2: Reproducible Training
A training run is a function of code, data, hyperparameters, environment, and randomness. Reproduce any one of those wrong and you get a different model.
Experiment Tracking
Pick a tracker. MLflow, Weights & Biases, or ClearML — the choice matters less than the discipline. Every run should log:
- Git commit hash of the training code
- Hash or snapshot ID of the training dataset
- Full hyperparameter set
- Environment (CUDA version, library versions, ideally a container digest)
- Metrics at every checkpoint, not just the final number
The single most useful piece of metadata is the data hash. When you ask “why does run 47 perform better than run 48?” you will, eventually, always end up at “the data shifted.” Make that answer instant.
import mlflow
mlflow.set_tracking_uri("https://mlflow.internal.acme.com")
mlflow.set_experiment("recs-v3")
with mlflow.start_run() as run:
mlflow.set_tag("git_sha", subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip())
mlflow.set_tag("dataset_version", dataset.version)
mlflow.log_params({"lr": 1e-3, "batch_size": 4096, "embedding_dim": 128})
for epoch in range(num_epochs):
loss = train_one_epoch(...)
mlflow.log_metric("train_loss", loss, step=epoch)
mlflow.pytorch.log_model(model, "model")
Containerized Training
Don’t run training on your laptop. Don’t run it on a shared dev box. Run it in a container whose digest you record. NVIDIA’s NGC containers give you a reproducible CUDA stack; Docker’s best practices guide covers the rest. Store the image digest in MLflow alongside the run.
Distributed Training
Once your dataset hits a certain size, you need to scale out. For deep models, PyTorch Distributed with FSDP or DeepSpeed’s ZeRO are the standard tools. For tabular ranking models like the recommendation example, XGBoost on Ray or LightGBM with num_threads is often enough. Don’t over-engineer — pick the smallest tool that gets the job done and re-evaluate when training time exceeds your iteration budget.
Layer 3: Evaluation That Actually Predicts Production
This is where most projects fail. They have a beautiful 0.92 AUC on a held-out set and ship it, then watch the business metric flatline. The problem isn’t the model — it’s the eval.
Offline Metrics Aren’t Enough
Offline metrics measure correlation, not value. A model that recommends more clickbait can show great CTR while degrading session depth. You need counterfactual or business-aligned offline metrics, plus a tight feedback loop with online evals.
For recommendation systems, the gold standard is a holdout that measures long-term value, not just next-click. The Netflix team’s write-up on offline-eval pitfalls is a classic reference here. The lesson: don’t trust offline metrics in isolation; use them as release gates, not as the release decision.
Evaluation Harnesses
Build a harness that runs the same evaluation suite on every candidate model before it touches production. This is the champion-challenger pattern from continuous delivery, applied to ML. The harness should produce:
- Standard offline metrics (NDCG, calibration, coverage)
- Slice-based metrics broken out by user segment, content category, and traffic source
- Latency and cost estimates from a dry-run inference
def evaluate_candidate(model, eval_dataset):
metrics = {
"ndcg@10": ndcg_score(eval_dataset, model, k=10),
"calibration_error": expected_calibration_error(model, eval_dataset),
"coverage": catalog_coverage(model, eval_dataset),
}
for segment in user_segments:
metrics[f"ndcg@10.{segment}"] = ndcg_score(
eval_dataset.filter_user_segment(segment), model, k=10
)
return metrics
If any gated metric drops more than X% relative to the champion, the candidate doesn’t ship. Period.
Online Evals and A/B Tests
Offline is a gate; online is the truth. Spin up an A/B test with a proper randomization unit (user ID, not request ID) and a primary metric you’ve decided on before looking at data. The Expanding Frontiers blog from Doordash has a great treatment of how to compute the sample size you need to actually detect realistic effects.
Layer 4: Serving at Production Scale
The serving layer is where most engineers underestimate the work. A model that takes 200ms in a notebook can take 5 seconds in production once you add feature lookups, post-processing, and retries.
Inference Architectures
Three common shapes:
- Online synchronous: client requests wait for inference. Common for chatbots, ranking at request time, and any UX where latency matters. Typical stack: FastAPI or Triton Inference Server in front of the model, with Redis in front of that for features.
- Precomputed batch: rankings computed every few minutes, written to a key-value store, served as lookups. Common for “for you” feeds at scale. Cheaper, but stale.
- Hybrid: precomputed candidates, reranked online by a heavier model. This is the dominant pattern at scale.
For the recommendation example, hybrid is the right answer — produce 500 candidates from a fast two-tower model, then rerank with a heavier cross-encoder for the top 50. The two-tower pattern is well-documented; the production insight is that the reranker is where your real model lives.
Batching, Caching, and Quantization
Three knobs that move latency and cost the most:
- Dynamic batching: collate incoming requests over a few milliseconds and infer as a batch. Triton does this out of the box; in custom serving stacks, vLLM pioneered it for LLMs.
- Caching: identical requests with identical features should hit cache. For ranking, the cache key is
(user_id, request_context_hash). - Quantization: INT8 or FP8 inference typically cuts latency 2–3x with negligible quality loss for ranking models. Worth doing from day one.
Fallbacks and SLOs
Define an SLO. Mine: “95th percentile end-to-end p99 under 200ms; 99.9% availability.” Then define the fallback chain when the model is slow or down:
- Slow: serve a cheaper model (e.g., the two-tower only, no reranker).
- Down: serve a non-personalized baseline (popularity, editorial picks).
- Degraded: serve the previous champion’s outputs from a snapshot.
The Circuit Breaker pattern is the textbook solution. Pair it with Hystrix-style isolation so a slow model can’t starve other services in the request path.
Layer 5: Observability and the Feedback Loop
You’re not done when the model is serving. You’re done when the system can detect when the model needs retraining — and can do it without paging a human.
Logging
Log every inference with: input feature hash, output, latency, and a sampled set of raw features. Don’t log PII — log stable hashed IDs and let joins happen downstream. For high-volume systems, this stream is itself a Kafka topic; for lower-volume, OpenTelemetry traces are fine.
Drift Detection
Two kinds:
- Data drift: input feature distributions shift. Catch with Kolmogorov–Smirnov tests on continuous features and chi-squared on categorical. The Evidently AI library ships standard implementations.
- Concept drift: the relationship between features and labels shifts. Harder to detect — usually you only see it via business metrics lagging. The cure is faster feedback loops, which brings us to the last point.
The Feedback Loop
The most underappreciated AI engineering pattern: closing the loop from production back to training data. Every production inference is a potential training example if you can later attach a label. For recommendations, this is automatic — clicks arrive minutes later. For generative systems, it’s harder — you need explicit user feedback or implicit signals like copy-paste rates. The LangSmith tracing approach for LLM apps is the modern equivalent: log every prompt and response with metadata, then mine for examples where users gave negative feedback.
Patterns in Production
A few patterns I’ve seen pay off repeatedly:
- Shadow deployments: run the new model in parallel with the champion, log its outputs, but never serve them. Lets you evaluate on live traffic distributions without risk. Both LinkedIn and Uber have written about this.
- Staged rollouts: 1% → 5% → 25% → 100%, with automatic rollback on SLO breach. This is just good release engineering applied to ML — but a surprising number of teams skip it.
- Canary models: train a model on a slice of data (e.g., only new users) and compare. Useful for testing hypotheses about data scaling.
- Model registries: a single source of truth for “what model version is in which environment.” MLflow’s Model Registry or a custom one in your CI/CD system.
Key Takeaways
- AI engineering is systems engineering. The model is one component; data pipelines, eval harnesses, serving, and observability matter at least as much.
- Reproducibility is the foundation. Log everything — code, data, environment, hyperparameters. If you can’t reproduce a run, you can’t debug it.
- Offline metrics gate, online metrics decide. Build a release harness that enforces thresholds, then trust A/B tests for the final call.
- Serving is where latency, cost, and reliability collide. Batch, cache, quantize, and design fallbacks before the model breaks in production.
- Observability closes the loop. Drift detection and feedback pipelines turn a deployed model into a continuously improving system.
- Start with the smallest thing that ships. A single retraining job on a schedule beats a perfect architecture that never sees production.