Scaling Large Language Models with Ray and Kubernetes for Production‑Grade Inference

Table of Contents

1. Introduction
2. Why Scaling LLM Inference Is Hard
3. Overview of Ray and Its Role in Distributed Inference
4. Kubernetes as the Orchestration Backbone
5. Architectural Blueprint: Ray on Kubernetes
6. Step‑by‑Step Implementation
   • 6.1 Preparing the Model Container
   • 6.2 Deploying a Ray Cluster on K8s
   • 6.3 Writing the Inference Service
   • 6.4 Autoscaling with Ray Autoscaler & K8s HPA
   • 6.5 Observability & Monitoring
7. Real‑World Production Considerations
   • 7.1 GPU Allocation Strategies
   • 7.2 Model Versioning & Rolling Updates
   • 7.3 Security & Multi‑Tenant Isolation
8. Performance Benchmarks & Cost Analysis
9. Conclusion
10. Resources

Introduction

Large language models (LLMs) such as GPT‑3, Llama 2, and Claude have moved from research curiosities to production‑critical components that power chatbots, code assistants, summarizers, and many other AI‑driven services. While training these models demands massive clusters and weeks of compute, serving them in real time presents a different set of engineering challenges:

• Low latency – Users expect sub‑second responses even for 30‑token prompts.
• High throughput – A single endpoint may need to handle thousands of concurrent requests.
• Resource efficiency – GPUs are expensive; we must pack as many inference jobs as possible without sacrificing quality.
• Scalability & resilience – Traffic spikes, node failures, and model upgrades must be handled gracefully.

Enter Ray – an open‑source framework for building distributed Python applications – coupled with Kubernetes, the de‑facto orchestration platform for containerized workloads. Together they provide a powerful, production‑grade stack for scaling LLM inference.

In this article we will walk through the entire lifecycle of building a scalable inference service:

1. Understanding the pain points of LLM inference.
2. Learning how Ray abstracts distributed execution.
3. Deploying a Ray cluster on Kubernetes.
4. Writing a model‑serving actor that can be autoscaled.
5. Adding observability, security, and cost‑optimization layers.
6. Benchmarking the solution and discussing real‑world trade‑offs.

By the end you’ll have a complete, reproducible code base that can be dropped into any CI/CD pipeline and run on a cloud‑native GPU fleet.

Why Scaling LLM Inference Is Hard

Before diving into solutions, it helps to enumerate the technical hurdles that arise when serving multi‑billion‑parameter models.

Traditional monolithic Flask or FastAPI services can’t address these issues at scale. We need a system that automatically distributes work across GPUs, performs intelligent batching, and reacts to load changes—all while staying container‑friendly.

Overview of Ray and Its Role in Distributed Inference

Ray provides three core concepts that map directly to the problems above:

1. Actors – Stateful, long‑running objects (e.g., a loaded LLM) that can receive remote method calls.
2. Tasks – Stateless functions that can be executed in parallel.
3. Ray Serve – A high‑level model serving library built on top of actors, supporting request routing, batching, and autoscaling.

Key benefits for LLM inference:

• Zero‑copy GPU sharing – Multiple actors can share the same GPU via Ray’s placement groups.
• Dynamic batching – Ray Serve automatically aggregates incoming requests into batches, reducing per‑token cost.
• Autoscaling – The Ray autoscaler can spin up/down workers based on request queue length.
• Fault tolerance – Actors are recreated on failure, preserving service continuity.

When we combine Ray Serve with Kubernetes, we get a cloud‑native inference platform where the underlying compute nodes are managed by K8s, while Ray handles the fine‑grained distribution of model workloads.

Kubernetes as the Orchestration Backbone

Kubernetes (K8s) offers:

• Pod scheduling – Allocate GPUs using device plugins (e.g., NVIDIA GPU Operator).
• Horizontal Pod Autoscaler (HPA) – Scale pods based on custom metrics such as Ray queue length.
• ConfigMaps & Secrets – Store model checkpoints, API keys, and environment variables securely.
• Rolling updates & rollbacks – Deploy new model versions without downtime.
• Service mesh integration – Use Istio or Linkerd for traffic routing, retries, and observability.

Deploying Ray on K8s essentially means running a head node (the Ray driver) and a set of worker pods that register themselves with the head. The Ray autoscaler runs as a sidecar in the head pod and interacts with the K8s API to create or delete worker pods on demand.

Architectural Blueprint: Ray on Kubernetes

Below is a high‑level diagram (described in text) of the production stack:

+-------------------+       +-------------------+       +-------------------+
|   Client / API   | <---> |  Ingress (NGINX)  | <---> |  Ray Serve Front  |
+-------------------+       +-------------------+       +-------------------+
                                                       |
                +-------------------+------------------+-------------------+
                |                   |                  |                   |
         +------+-----+      +------+-----+    +-------+------+    +-------+------+
         | Ray Actor  |      | Ray Actor  |    | Ray Actor   |    | Ray Actor   |
         | (Llama‑2)  |      | (GPT‑Neo)  |    | (Claude‑1)  |    | (Custom)    |
         +------------+      +------------+    +-------------+    +------------+
                |                   |                  |                   |
          +-----+-----+       +-----+-----+      +-----+-----+       +-----+-----+
          | GPU Node  |       | GPU Node  |      | GPU Node  |       | GPU Node  |
          +-----------+       +-----------+      +-----------+       +-----------+

• Ingress – Exposes a stable HTTP(S) endpoint.
• Ray Serve Front – A single HTTP server that forwards requests to the appropriate Ray actor.
• Actors – Each holds a loaded model in GPU memory; multiple replicas can exist for horizontal scaling.
• GPU Nodes – K8s worker nodes equipped with NVIDIA GPUs, managed by the Ray autoscaler.

Step‑by‑Step Implementation

6.1 Preparing the Model Container

We’ll start by building a Docker image that:

1. Installs Python 3.11, Ray, PyTorch, and the model dependencies.
2. Copies a lightweight script (serve.py) that defines the Ray Serve deployment.
3. Exposes port 8000 for the Serve HTTP endpoint.

Dockerfile

# Use the official NVIDIA PyTorch base image (CUDA 12.1, cuDNN 8)
FROM nvcr.io/nvidia/pytorch:23.09-py3

# Install Ray and other utilities
RUN pip install --no-cache-dir \
    "ray[default]==2.9.0" \
    "ray[serve]==2.9.0" \
    "transformers==4.38.0" \
    "accelerate==0.27.2" \
    "fastapi==0.110.0" \
    "uvicorn[standard]==0.27.0"

# Create a non‑root user
RUN useradd -ms /bin/bash inference && \
    chown -R inference:inference /workspace

USER inference
WORKDIR /workspace

# Copy the serving script
COPY serve.py .

# Expose the Serve HTTP port
EXPOSE 8000

# Entry point – start Ray Serve
CMD ["python", "serve.py"]

serve.py (simplified, full version later)

import os
import ray
from ray import serve
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# -------------------------------------------------
# 1️⃣  Initialize Ray (head or worker mode)
# -------------------------------------------------
if os.getenv("RAY_HEAD", "0") == "1":
    ray.init(address="auto", _node_ip_address=os.getenv("MY_POD_IP"))
else:
    ray.init()

# -------------------------------------------------
# 2️⃣  Define the LLM actor – holds model on GPU
# -------------------------------------------------
@ray.remote(num_gpus=1)  # Guarantees a whole GPU per replica
class LLMActor:
    def __init__(self, model_name: str):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
        self.model = AutoModelForCausalLM.from_pretrained(
            model_name,
            torch_dtype=torch.float16,
            device_map="auto"
        )
        self.model.eval()

    async def generate(self, prompt: str, max_new_tokens: int = 64):
        inputs = self.tokenizer(prompt, return_tensors="pt").to("cuda")
        with torch.no_grad():
            output = self.model.generate(
                **inputs,
                max_new_tokens=max_new_tokens,
                do_sample=True,
                temperature=0.7,
                top_p=0.9,
            )
        return self.tokenizer.decode(output[0], skip_special_tokens=True)

# -------------------------------------------------
# 3️⃣  Deploy the actor with Ray Serve
# -------------------------------------------------
serve.start(detached=True)

# Model name can be overridden via env var
MODEL_NAME = os.getenv("MODEL_NAME", "meta-llama/Llama-2-7b-chat-hf")

# Deploy a replica set (adjust `replicas` via env)
@serve.deployment(
    name="llm",
    route_prefix="/generate",
    autoscaling_config={
        "min_replicas": int(os.getenv("MIN_REPLICAS", "1")),
        "max_replicas": int(os.getenv("MAX_REPLICAS", "4")),
        "target_num_ongoing_requests_per_replica": 4,
    },
)
class LLMDeployment:
    def __init__(self):
        # The actor class is wrapped by Serve automatically
        self.actor = LLMActor.remote(MODEL_NAME)

    async def __call__(self, request):
        json_body = await request.json()
        prompt = json_body.get("prompt", "")
        max_tokens = json_body.get("max_new_tokens", 64)
        return await self.actor.generate.remote(prompt, max_tokens)

LLMDeployment.deploy()

Key points:

• num_gpus=1 ensures each replica gets an exclusive GPU; you can change to fractional GPUs (0.5) if the model fits.
• Autoscaling config – Ray Serve will add/remove replicas based on the number of pending requests.
• detached=True – The Serve instance lives beyond the script’s lifetime, crucial when the pod restarts.

6.2 Deploying a Ray Cluster on K8s

Ray provides a Helm chart that boots a head service and a worker deployment. Below is a minimal values.yaml that we’ll apply.

ray-cluster-values.yaml

image:
  repository: your-registry/llm-ray
  tag: latest
  pullPolicy: IfNotPresent

# Enable GPU support via node selector & resource limits
resources:
  limits:
    nvidia.com/gpu: 1          # Each worker pod gets 1 GPU
  requests:
    nvidia.com/gpu: 1

# Ray autoscaler configuration – this is mounted as a ConfigMap
autoscaler:
  enabled: true
  config: |
    # Ray autoscaler config (YAML)
    cluster_name: llm-ray-cluster
    max_workers: 10
    min_workers: 2
    upscaling_speed: 1.0
    idle_timeout_minutes: 5
    provider:
      type: kubernetes
      namespace: default
    head_node:
      # Head pod runs the container with RAY_HEAD=1 env var
      pod_config:
        spec:
          containers:
          - name: ray-head
            image: your-registry/llm-ray:latest
            env:
            - name: RAY_HEAD
              value: "1"
            - name: MODEL_NAME
              value: "meta-llama/Llama-2-7b-chat-hf"
            - name: MIN_REPLICAS
              value: "1"
            - name: MAX_REPLICAS
              value: "4"
            resources:
              limits:
                nvidia.com/gpu: 1
    worker_nodes:
      pod_config:
        spec:
          containers:
          - name: ray-worker
            image: your-registry/llm-ray:latest
            env:
            - name: RAY_HEAD
              value: "0"
            resources:
              limits:
                nvidia.com/gpu: 1

Deploy the cluster

# Add the Ray Helm repo
helm repo add ray https://ray-project.github.io/kuberay-helm/
helm repo update

# Install the RayCluster CRD (only needed once)
kubectl apply -f https://github.com/ray-project/kuberay/releases/download/v0.5.0/kuberay-operator.yaml

# Deploy the RayCluster using our custom values
helm install llm-ray-raycluster ray/kuberay-cluster -f ray-cluster-values.yaml

The Helm chart spins up a head pod (ray-head) and a pool of worker pods (ray-worker). The Ray autoscaler continuously reconciles the desired number of workers based on the queue length reported by Serve.

6.3 Writing the Inference Service

While the previous serve.py already defines a REST endpoint (/generate), production environments often want a FastAPI wrapper for richer validation, authentication, and OpenAPI docs.

app.py

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
import ray
from ray import serve

app = FastAPI(
    title="LLM Inference Service",
    description="Scalable LLM generation using Ray Serve on Kubernetes.",
    version="1.0.0",
)

# -------------------------------------------------
# Pydantic model for request payload
# -------------------------------------------------
class GenerateRequest(BaseModel):
    prompt: str = Field(..., description="User prompt or system instruction")
    max_new_tokens: int = Field(64, ge=1, le=256, description="Maximum tokens to generate")

# -------------------------------------------------
# Initialise Ray (this runs inside the same container)
# -------------------------------------------------
if not ray.is_initialized():
    ray.init(address="auto")

# -------------------------------------------------
# Deploy the model if not already done
# -------------------------------------------------
@serve.deployment(name="llm", route_prefix="/v1/generate")
class LLMDeployment:
    def __init__(self):
        self.actor = ray.get_actor("LLMActor", namespace="default")

    async def __call__(self, request):
        payload = await request.json()
        prompt = payload.get("prompt", "")
        max_tokens = payload.get("max_new_tokens", 64)
        result = await self.actor.generate.remote(prompt, max_tokens)
        return {"generated_text": result}

# Deploy once; subsequent runs will just attach.
if not serve.get_deployment("llm"):
    LLMDeployment.deploy()

# -------------------------------------------------
# FastAPI endpoint that forwards to Ray Serve
# -------------------------------------------------
@app.post("/v1/generate")
async def generate(req: GenerateRequest):
    # Forward request to the Serve deployment via HTTP
    # (alternatively, use Ray client directly)
    import httpx
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            "http://127.0.0.1:8000/v1/generate",
            json=req.dict(),
            timeout=30,
        )
    if resp.status_code != 200:
        raise HTTPException(status_code=500, detail="Backend error")
    return resp.json()

Why FastAPI?

• Automatic OpenAPI schema (/docs) – useful for internal teams.
• Input validation via Pydantic – prevents malformed prompts.
• Easy integration with authentication middlewares (e.g., API keys, OIDC).

The app.py container can be run alongside the Ray head pod or as a sidecar, exposing port 8080 through a K8s Service.

6.4 Autoscaling with Ray Autoscaler & K8s HPA

Ray’s internal autoscaler reacts to Serve request queue length. However, you may also want K8s to scale the head pod (especially if you run multiple services) or to add more worker nodes to the cluster.

Custom Metrics Adapter

Ray publishes metrics to Prometheus under the ray_serve_* namespace. We can expose the ray_serve_requests_queued metric to the HPA via the Prometheus Adapter.

# prometheus-adapter-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-metrics-config
  namespace: kube-system
data:
  config.yaml: |
    rules:
    - seriesQuery: 'ray_serve_requests_queued{deployment!="",namespace!="",instance!=""}'
      resources:
        overrides:
          namespace: {resource: namespace}
          deployment: {resource: deployment}
      name:
        matches: ""
        as: "serve_queue_length"
      metricsQuery: sum(ray_serve_requests_queued) by (deployment,namespace)

HorizontalPodAutoscaler

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-ray-head-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-ray-head
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Pods
    pods:
      metric:
        name: serve_queue_length
      target:
        type: AverageValue
        averageValue: "5"

The HPA watches the queued request count; when the average queue per head pod exceeds 5, it adds another head replica (which in turn spawns more workers via Ray autoscaler). This two‑layer scaling ensures we can handle massive spikes without exhausting the underlying GPU nodes.

6.5 Observability & Monitoring

A production inference platform must expose:

• Latency histograms – per‑request and per‑batch.
• GPU utilization – via NVIDIA DCGM Exporter.
• Error rates – HTTP 5xx, Ray actor failures.
• Model‑specific metrics – tokens generated, cache hits.

Prometheus Scrape Config (partial)

scrape_configs:
  - job_name: 'ray'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: ray
        action: keep
    metrics_path: /metrics
    scheme: http

Grafana Dashboard

Create a dashboard with panels:

1. ray_serve_request_latency_seconds – latency distribution.
2. nvidia_gpu_utilization – % GPU usage per node.
3. ray_serve_requests_total – cumulative request count.
4. ray_serve_replica_num – current replica count.

Logging

Ray’s internal logs can be shipped to Elasticsearch or Loki using Fluent Bit. Include the RAY_LOG_TO_STDERR=1 env var to force logs to stdout (captured by K8s).

Real‑World Production Considerations

7.1 GPU Allocation Strategies

For most 7‑13 B parameter models, fractional GPUs paired with dynamic batching give the best cost‑performance ratio.

7.2 Model Versioning & Rolling Updates

Ray Serve supports named deployments. To roll out a new model:

# Deploy new version under a different name
@serve.deployment(name="llm_v2", route_prefix="/v2/generate")
class LLMV2Deployment:
    ...

LLMV2Deployment.deploy()

Then update the FastAPI router to point to /v2/generate. Gradual traffic shifting can be performed with an Istio VirtualService that splits 80/20 between v1 and v2.

Kubernetes rolling updates ensure the underlying pods are replaced without downtime. Use readiness probes that check /healthz on the Ray head to guarantee the new pod is ready before traffic is sent.

7.3 Security & Multi‑Tenant Isolation

• Network Policies – Restrict which services can reach the Ray head.
• Namespace Isolation – Deploy each tenant’s model in its own K8s namespace with separate Ray clusters.
• API Keys / OAuth – FastAPI dependencies can validate tokens before forwarding to Serve.
• GPU Resource Quotas – Enforce per‑tenant GPU limits via K8s ResourceQuota.

Performance Benchmarks & Cost Analysis

We evaluated three configurations on an AWS p4d.24xlarge node (8 × NVIDIA A100 40 GB) using Llama‑2‑7B‑Chat.

Key takeaways

• Dynamic batching reduces latency dramatically while keeping GPU utilization > 90 %.
• Fractional GPU allocation yields the best cost‑performance, but you must monitor memory to avoid OOM.
• Ray’s autoscaling can shrink the cluster during off‑peak hours, cutting the hourly bill by up to 60 %.

Conclusion

Scaling large language models for production inference is no longer a “research‑only” problem. By marrying Ray’s fine‑grained distributed execution with Kubernetes’ robust orchestration, you gain:

• Elastic GPU utilization – automatic scaling of model replicas based on request load.
• Low‑latency serving – built‑in dynamic batching and zero‑copy GPU sharing.
• Operational excellence – native K8s tools for rollout, monitoring, and security.
• Cost‑efficiency – fractional GPU assignments and autoscaling keep spend under control.

The end‑to‑end workflow presented—Dockerizing a model, deploying a Ray cluster via Helm, exposing a FastAPI gateway, and wiring up observability—covers everything a modern MLOps team needs to push LLM inference into production with confidence.

Feel free to adapt the code snippets to your own model families, cloud provider, or compliance requirements. As LLMs continue to grow, the principles of distributed actors, dynamic batching, and cloud‑native orchestration will remain the cornerstone of scalable AI services.

Resources

• Ray Documentation – Serve – https://docs.ray.io/en/latest/serve/index.html
• Kubernetes Autoscaling – HPA & Custom Metrics – https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
• NVIDIA GPU Operator for Kubernetes – https://github.com/NVIDIA/gpu-operator
• FastAPI – High‑Performance APIs – https://fastapi.tiangolo.com/
• DeepSpeed – Model Parallelism – https://www.deepspeed.ai/
• Prometheus Adapter for Custom Metrics – https://github.com/kubernetes-sigs/prometheus-adapter