TL;DR — LLMOps is the discipline of shipping and operating LLM-backed applications in production: evaluation harnesses, retrieval pipelines, guardrails, cost controls, and observability for non-deterministic systems. GenAIOps extends the MLOps playbook for generative workloads — content quality, safety, and human feedback loops — and treats the prompt, model, and retrieval index as first-class deployable artifacts.
Most LLM projects die the same way: a great demo, an even greater pitch deck, and a quiet Slack channel where someone asks, “so, how do we put this in front of customers?” The gap between a notebook and a production service is exactly what LLMOps and GenAIOps are designed to close. This post walks through the patterns that actually hold up in production — the ones you only learn after the third on-call rotation.
Why MLOps Alone Isn’t Enough
Classical MLOps assumes you ship a model artifact — a serialized model.pkl or a TensorFlow graph — and serve it behind a stable input schema. The contract is tidy: inputs go in, predictions come out, latency is bounded, and the failure modes are well understood.
LLMs break several of those assumptions at once:
- Non-determinism. The same prompt can produce different outputs across runs, even at temperature 0, due to batching, hardware numerics, and kernel changes.
- Schema drift at the output layer. You ask for JSON, you get JSON — but the keys drift between model versions, and a 3% parser failure rate is enough to break a downstream pipeline.
- Compute cost lives at inference. A traditional ML model is cheap to run compared to its training cost. For LLMs, especially long-context or agentic workloads, inference often dominates the bill — see the cost analyses on OpenAI’s pricing page and Anthropic’s pricing page.
- The “model” is a moving target. You might fine-tune today, switch providers tomorrow, and host an open-weights model the day after. The unit of deployment shifts from a single artifact to a graph: prompt + model + retriever + tools.
- Quality is a distribution, not a number. Accuracy on a labeled set tells you very little about whether your chatbot will embarrass you on a Tuesday afternoon.
LLMOps exists because the operational primitives of MLOps — versioning, CI/CD, monitoring — still apply, but the contents of those primitives have to be reinvented.
The LLMOps Stack: A Practical Layering
Think of an LLM production system as five concentric layers. Each one needs its own tooling, its own SLOs, and its own on-call owner.
┌──────────────────────────────────────────────────────────────┐
│ 5. Product & UX → safety, hallucination, tone │
│ 4. Evaluation & Feedback → golden sets, LLM-as-judge, HITL │
│ 3. Application Logic → prompts, tools, agents, routing │
│ 2. Retrieval & Context → vector DB, re-rankers, filters │
│ 1. Model Gateway → auth, rate limit, caching │
│ 0. Foundation Models → hosted APIs or self-hosted │
└──────────────────────────────────────────────────────────────┘
The bottom layer is “did we pick a viable model?” The top is “would a customer trust this output?” Most outages I have seen live somewhere in layers 2–4.
Layer 0 — Foundation Models
You have three real choices: a hosted frontier API (OpenAI, Anthropic, Google), an open-weights model served on your own GPUs (vLLM, TGI, SGLang), or a hybrid where you burst to a hosted API when self-hosted capacity runs out. The operational profile of each is wildly different.
- Hosted APIs. Almost no infra, but you inherit the provider’s availability, deprecation, and pricing schedule. Capacity is somebody else’s problem until it isn’t.
- Self-hosted open weights. You control latency, data residency, and unit economics. You also own the GPU fleet, the quantization pipeline, and the kv-cache sizing math.
- Hybrid. Most teams at scale land here — a primary self-hosted model for steady traffic, a hosted fallback for spikes. vLLM’s PagedAttention paper is worth reading just to understand why inference engines are not interchangeable.
Whichever you pick, the model is now a versioned dependency, not a constant. Pin it, hash it, and treat upgrades as migrations.
Layer 1 — Model Gateway
Every serious LLM deployment routes through a gateway. The gateway is where you put the cross-cutting concerns that would otherwise be smeared across every microservice:
- Authentication and key management — separate API keys per team, per environment, per vendor.
- Rate limiting and budgets — per-user, per-tenant, per-feature soft and hard caps.
- Caching — exact-match caches for repeated prompts, semantic caches for near-duplicates (see LangChain’s semantic cache guidance).
- Fallback and routing — primary model with cheaper fallback for simple queries, escalation to a stronger model for hard ones.
- PII redaction and prompt logging — strip secrets before the request leaves your VPC.
Tools in this space range from lightweight sidecars (like litellm in proxy mode) to full platforms (like Portkey, Cloudflare AI Gateway, or Kong AI Gateway). The choice matters less than the discipline of funneling all traffic through something.
Layer 2 — Retrieval and Context
Most production LLM systems are RAG systems, whether the team calls them that or not. The retrieval layer is where you trade “the model knows everything” for “the model knows what we tell it.” That trade is almost always worth it: a grounded 8B model usually beats an ungrounded 70B one, and you can update knowledge without retraining.
A production retrieval stack has at least four components:
Document store → Chunking → Embedding + Index → Re-rank → Prompt assembly
Concretely:
- Chunking. The single highest-leverage decision. Fixed-size chunks are easy; structure-aware chunks (by heading, code block, table row) usually win. LangChain’s text splitters and LlamaIndex’s node parsers are the usual starting points.
- Embeddings. Hosted (OpenAI, Voyage, Cohere) or self-hosted (BGE, E5, GTE). Pick a model whose training distribution matches your corpus — code, legal text, and chat transcripts want different embedding models.
- Vector store. This is where named-system anchoring pays off. Production teams overwhelmingly converge on a handful of options: Pinecone, Weaviate, Qdrant, Milvus, and Postgres with pgvector. If you already run Postgres, pgvector is the path of least resistance; if you are doing hybrid search at serious scale, Qdrant or Weaviate tend to win.
- Re-ranking. A cheap embedding-based top-k followed by a more expensive cross-encoder reranker almost always beats either alone. Cohere Rerank and BGE rerankers are common choices.
The “vector database” label obscures a more interesting reality: the best retrieval systems are hybrid, blending BM25 with dense vectors and applying metadata filters before re-ranking. A pure vector DB without lexical fallback will struggle on exact-match queries like product SKUs or error codes.
Layer 3 — Application Logic
This is where prompts, tools, and agents live — and where the actual product behavior is defined. Treat all of it as code:
# A simplified "router + tool call" pattern
def route_and_answer(question: str, tenant_ctx: TenantContext) -> Answer:
plan = planner_llm(question, tools=available_tools(tenant_ctx))
if plan.needs_retrieval:
docs = retriever.search(question, top_k=20, filters=tenant_ctx.filters)
docs = reranker.rerank(question, docs, top_k=4)
context = assemble_context(docs, token_budget=4000)
else:
context = None
response = answer_llm(question, context=context, style=tenant_ctx.style_guide)
return validate_and_enrich(response, schema=Answer)
Three patterns dominate this layer:
- Routing. Cheap models classify incoming traffic and dispatch to the right pipeline. A billing question and a creative-writing prompt should never share a code path — let alone a model.
- Tool use. Function calling is now table stakes; the operational question is who owns the tool registry and how tool failures are surfaced. Every tool call should be logged with its arguments, result, latency, and whether the model actually used it in the final answer.
- Agents. Real agentic systems (multi-step, branching, self-correcting) are rare in production because they are hard to make reliable. The vast majority of “agents” in shipping products are short loops — 1–3 steps — with explicit termination conditions.
Whichever you ship, the prompt is a deployable artifact. Store it in version control, evaluate it before merging, and roll it back the same way you would roll back a model.
Layer 4 — Evaluation and Feedback
This is the layer that separates LLM demos from LLM products. You cannot manage what you cannot measure, and measuring generative systems is hard.
There are three evaluation modalities, and you need all of them:
| Modality | What it measures | Strength | Weakness |
|---|---|---|---|
| Automatic metrics | BLEU, ROUGE, exact match | Fast, cheap, reproducible | Misses semantics |
| LLM-as-judge | Relevance, faithfulness, tone | Scales, correlates with humans | Judge bias, cost |
| Human evaluation | True quality, brand safety | Gold standard | Slow, expensive |
The Anthropic cookbook and OpenAI Evals framework are good starting points. For LLM-as-judge specifically, DeepEval and Braintrust provide out-of-the-box metrics for hallucination, answer relevancy, and toxicity.
A practical evaluation pipeline looks like this:
# eval-config.yaml — illustrative
dataset:
source: s3://golden-sets/support-v3.jsonl
version: 2025.11.0
metrics:
- name: answer_relevancy
type: llm_judge
judge_model: gpt-4o-mini
threshold: 0.85
- name: hallucination
type: llm_judge
judge_model: gpt-4o-mini
threshold: 0.95
- name: schema_validity
type: deterministic
threshold: 1.0
- name: p95_latency_ms
type: runtime
threshold: 3500
sampling:
rate: 0.05
store_traces: true
The trick is wiring evaluation into CI: every prompt change, every model upgrade, every retriever swap runs through this suite and either passes or blocks the deploy. This is the LLMOps equivalent of a unit test gate.
Layer 5 — Product, Safety, and Trust
The top layer is the one executives notice and engineers underestimate. Three things live here:
- Safety filters. Toxicity, PII leakage, prompt injection. The OWASP LLM Top 10 is the canonical reference; treat prompt injection as a first-class threat with detection, isolation, and logging, not as an edge case.
- Tone and brand alignment. A model can be factually correct and still sound wrong. Style guides, example outputs, and tone rubrics belong in evaluation.
- User feedback loops. Thumbs-up/down is fine; structured feedback with reasons is better. Whatever you collect, it should flow back into your golden set and your fine-tuning pipeline.
GenAIOps: What’s Different From LLMOps?
The terms are often used interchangeably, and you will see vendors use both to describe the same platform. Practically, GenAIOps is the broader umbrella — it covers LLMs but also image generation, speech, code generation, and multi-modal agents. The “ops” suffix in both cases means the same thing: production discipline.
Where GenAIOps adds genuine new concerns:
- Multi-model orchestration. A product might use one model to summarize a document, another to generate an image, and a third to caption it. Each has its own SLO.
- Asset pipelines. Generated images, audio clips, and video frames are now first-class artifacts that need storage, CDN delivery, and rights management.
- Human-in-the-loop at scale. Creative outputs almost always need review before shipping. You need review queues, reviewer assignment, and audit trails — the same machinery a content moderation system has, even if your volume is lower.
- Cost attribution. A single user request might fan out to four models. Without per-stage cost attribution, you cannot price the product or detect abuse.
If your system is purely text-in, text-out, “LLMOps” is the accurate label. If you generate anything else — or compose multiple model calls per request — “GenAIOps” is more honest.
Patterns in Production: Three Architectures That Work
Theory is cheap. Here are three architectures I have seen ship successfully at non-trivial scale.
Architecture 1: The Grounded Support Assistant
A classic RAG system for a support knowledge base.
User → API Gateway → Intent Classifier (small model)
├── FAQ path → cached answer
├── Account q. → tool call to billing API
└── How-to q. → retrieve → rerank → answer LLM → guardrail
Key choices:
- pgvector for retrieval because the team already runs Postgres and the corpus is small.
- gpt-4o-mini for routing, gpt-4o for answering hard questions.
- Exact-match cache for the top 200 FAQs, which covers ~40% of traffic.
- Guardrail layer that strips any model output containing PII or unverifiable claims.
Cost per ticket drops by ~70% versus the human-only baseline, latency p95 is under 4 seconds, and hallucination rate (measured by LLM-as-judge against a labeled set) sits around 3%.
Architecture 2: The Multi-Tenant Document Analyst
A B2B product where each customer’s documents live in their own isolated index.
Upload → chunk → embed → per-tenant index (Qdrant)
Query → tenant auth → filter by tenant_id → retrieve → rerank → answer
→ citation check → structured output → audit log
Key choices:
- Per-tenant vector indexes rather than a single shared index with metadata filters. At a few hundred tenants, isolation is simpler than clever filtering and lets you delete a customer cleanly.
- Citation enforcement in the prompt and validated in post-processing: every claim in the answer must reference a chunk ID, and chunks with low similarity are flagged.
- Audit log capturing prompt, retrieved chunks, model output, and user feedback. This is your regulatory defense and your evaluation dataset.
The non-obvious win is the audit log. Six months in, it becomes the most valuable dataset you own.
Architecture 3: The Agentic Workflow Engine
An internal tool that chains multiple model calls with tool use to complete multi-step business processes.
Trigger → Planner → Executor (loop)
├── tool call → result
├── tool call → error → retry/backoff
└── final answer → reviewer (LLM) → human spot-check
Key choices:
- Hard step limit (typically 5–8) and a timeout. Without these, a stuck agent will burn your GPU budget overnight.
- Deterministic planner, stochastic executor. The planner that decides what to do uses a low-temperature model; the executor that decides how to phrase it can be creative.
- Sandboxed tool execution. Agents should never have unrestricted access to production systems. The pattern is the same as IAM for human operators: least privilege, scoped credentials, audited actions.
- Human-in-the-loop checkpoints for any irreversible action. Most “agent” demos skip this; most “agent” production systems cannot.
The lesson from shipping agentic systems: the harder problem is not the model, it is the failure recovery. Plan for the agent to fail, and design the system so a failure is cheap.
Observability for Non-Deterministic Systems
Traditional APM tools answer “is the service up and fast?” LLMOps observability has to answer harder questions:
- Is the model returning worse answers this week?
- Did a prompt change cause a regression we didn’t catch in CI?
- Are certain user segments hitting higher refusal or fallback rates?
- Is a specific tool call failing in a way that’s degrading the user experience?
The canonical tooling in this space includes LangSmith, Langfuse (open source, self-hostable — my usual recommendation), Arize Phoenix, Helicone, and WhyLabs. What they all do, at their core, is capture every request’s full trace — prompt, retrieved context, model output, latency, cost, feedback — and let you slice it.
The minimum viable observability stack:
Trace every production request (input, output, latency, cost)
↓
Sample 5–10% into a labeled evaluation pipeline
↓
Aggregate metrics on a dashboard (quality, cost, latency, refusal rate)
↓
Alert on regressions vs. rolling 7-day baseline
If you implement only one thing from this post, implement this. Everything else in LLMOps is optimization; observability is survival.
Cost Control Without Tears
LLM costs are uniquely easy to lose control of because they scale with usage, and usage of a useful feature tends to grow. A few rules of thumb:
- Cache aggressively. Exact-match caches turn 20–40% of traffic into free retries for many real workloads.
- Route by difficulty. Use a cheap model to classify, then dispatch. A well-tuned router can cut blended cost in half.
- Bound context length. Truncate retrieved documents, cap conversation history, and reject requests over a token limit. Most teams discover that 80% of their cost comes from 20% of inputs that blew past their context window.
- Set per-tenant budgets. A single misbehaving integration can rack up five figures overnight. Per-key rate limits are not optional.
- Track cost per request in your dashboard. If cost-per-request is invisible, it is also unmanageable.
The companies that lose money on LLM products almost never lose it on training. They lose it on inference they didn’t measure.
The Team You Actually Need
LLMOps is not a single role. The teams that ship reliably tend to have:
- A platform engineer who owns the gateway, the caches, and the GPU fleet.
- An ML engineer who owns retrieval, embeddings, and evaluation.
- An applied scientist (or strong senior engineer) who owns prompts, tools, and the evaluation rubric.
- A product engineer who owns the user-facing behavior and the feedback collection.
- A safety/responsible-AI reviewer, even at 0.25 FTE, who signs off on high-risk outputs.
Total headcount can be small — three or four people can run a meaningful LLM product — but the responsibilities must be explicit, otherwise evaluation gets skipped, the gateway becomes spaghetti, and the cost dashboard is someone’s side project.
Key Takeaways
- LLMOps is MLOps adapted for non-deterministic, generative, inference-heavy systems. The principles are familiar; the contents are not.
- Funnel everything through a model gateway. Auth, caching, rate limits, fallbacks, and logging belong in one place, not scattered across services.
- Retrieval is a stack, not a database. Chunking, embeddings, vector store, and re-ranking each have their own failure modes.
- Evaluation is a CI gate, not a quarterly exercise. Golden sets, LLM-as-judge, and human spot-checks together are the only thing standing between a prompt edit and a customer-visible regression.
- Observability is survival. Trace every request, sample into evaluation, alert on distribution shift.
- Bound cost like you bound latency. Caching, routing, context caps, and per-tenant budgets are non-negotiable.
- Treat the prompt, the model, and the retriever index as versioned deployable artifacts. If you cannot roll them back, you cannot ship them confidently.