TL;DR — Multi-tenant Kubernetes clusters require fine-grained admission control to enforce namespace isolation, resource quotas, and label ownership at create/update time. By combining Kyverno’s native Kubernetes policy engine with Open Policy Agent Gatekeeper’s constraint-based validation, teams can achieve composable, auditable, and performant tenant isolation without mutating core API objects. This post walks through the architecture, policy patterns, and operational playbooks for deploying multi-tenant admission control at scale.

Building a multi-tenant Kubernetes platform demands that each tenant’s workloads are invisible, restricted, and governed by organizational policy. Admission controllers sit at the gateway of the API server, making them the natural enforcement point for tenant boundaries. Unlike namespace-scoped RBAC or deprecated Pod Security Policies, modern admission control with Kyverno and OPA provides declarative, version-controlled policies that integrate directly into CI/CD pipelines. In this article, we’ll explore how to architect a layered defense: Kyverno for namespace lifecycle and label propagation, OPA Gatekeeper for resource shape and constraint enforcement, and the operational patterns that keep policies maintainable across dozens or hundreds of tenants.

1. Why Multi-Tenant Admission Control Matters

In a typical Kubernetes cluster, multiple teams or customers share the same control plane. Without explicit admission control, a single tenant can inadvertently (or maliciously) consume all available compute, bypass namespace isolation, or deploy resources that violate organizational security standards. The Kubernetes API server processes requests in order: authentication, authorization, then admission. By inserting policies into the admission phase, you gain the ability to reject or mutate requests before they persist, ensuring that tenant boundaries are enforced at the point of creation or modification.

Common failure modes in multi-tenant clusters include:

  • Resource starvation: A noisy neighbor deploying an unbounded Deployment or Job, evicting other tenants’ pods.
  • Label leakage: Workloads missing required tenant labels, causing monitoring and routing systems to misattribute metrics.
  • Cross-tenant resource access: Secrets or ConfigMaps accidentally scoped to the wrong namespace due to missing metadata.namespace constraints.

Admission control addresses these by making policy enforcement a first-class part of the cluster lifecycle. Kyverno and OPA Gatekeeper complement each other: Kyverno operates natively as a Kubernetes controller, understanding Kubernetes resources without requiring external dependencies; OPA Gatekeeper provides a Rego-based constraint language that can express complex, application-level business rules. Together, they form a defense-in-depth model where Kyverno handles the “plumbing” of tenant onboarding, and OPA enforces the “shape” of what each tenant can provision.

2. Kyverno: Kubernetes-Native Policy Enforcement

Kyverno’s design philosophy centers on using standard Kubernetes objects—ClusterRole, Role, CustomResourceDefinitions—to define, distribute, and enforce policies. Because Kyverno policies are themselves Kubernetes resources, they benefit from native tooling: kubectl get policy, Helm charts for deployment, and GitOps workflows via Argo CD or Flux.

2.1 Tenant Label Injection

A foundational pattern in multi-tenant architectures is ensuring every resource carries a tenant_id label. This label flows through to workloads, enabling network policies, resource quotas, and monitoring filters to correctly attribute traffic. Kyverno can inject this label during resource creation or update via a Policy with a mutate rule.

Consider a policy that adds tenant_id to every namespace created in a shared cluster:

apiVersion: kyverno.io/v1
kind: Policy
metadata:
  name: inject-tenant-label
spec:
  rules:
  - name: add-tenant-to-namespace
    match:
      any:
      - resources:
          kinds: ["Namespace"]
    mutate:
      patchType: JSONPatch
      patchJsonPatch:
        - op: add
          path: /metadata/labels/tenant_id
          value: "tenant-xyz"
    apply: {}
    background: false
    generateName: ""

This rule triggers on every Namespace creation. The mutate block applies a JSON Patch that adds the label. If the label already exists, the patch fails by default, which is the desired behavior for strict tenant enforcement. Kyverno’s deny rules can complement this by rejecting namespace creation without an explicit tenant identifier.

2.2 Namespace Defaults and Quota Propagation

Beyond label injection, Kyverno can propagate default resource quotas when a tenant namespace is created. A policy can watch for Namespace events and automatically create a ResourceQuota with tenant-specific limits. This eliminates the manual step of quota provisioning and reduces the risk of a tenant deploying unrestricted workloads.

apiVersion: kyverno.io/v1
kind: Policy
metadata:
  name: tenant-quota-defaults
spec:
  rules:
  - name: create-quota-upon-namespace
    match:
      any:
      - resources:
          kinds: ["Namespace"]
    generate:
      apiVersion: v1
      kind: ResourceQuota
      name: "{{request.object.metadata.name}}-quota"
      spec:
        hard:
          cpu: "2"
          memory: "4Gi"
          pods: "50"
    filter:
      any:
      - key: "annotations Kyverno/tenant-aware"
        value: "true"

The generate block defines the quota manifest that Kyverno creates whenever the filter matches. The hard limits here are illustrative; in production, these would be pulled from a per-tenant configuration store or parameterized via policy parameters.

2.3 Kyverno Policy Testing and Verification

Kyverno ships with kyverno test, a command that evaluates policies against a set of YAML manifests. This integrates seamlessly into CI pipelines. A typical test case might create a Namespace without the expected label and verify that the policy either mutates it or rejects the request, depending on the validate vs mutate configuration. This test-driven approach ensures that policy changes are validated before they reach the cluster.

3. OPA Gatekeeper: Constraint-Based Validation

While Kyverno excels at mutation and lifecycle automation, OPA Gatekeeper specializes in expressive, logic-based validation. Gatekeeper uses Rego, a declarative policy language, to write constraints that check the admission request against complex conditions. These constraints are compiled into ConstraintTemplate objects, which are then instantiated as Constraint objects binding the template to specific cluster roles or namespaces.

3.1 Constraint Templates for Resource Quotas

A common multi-tenant requirement is to enforce CPU and memory limits per namespace. Gatekeeper can validate that any Deployment or Pod spec respects the tenant’s allocated quota. Below is a Rego Constraint Template that ensures a Deployment does not exceed its tenant’s CPU limit:

package kubernetes.admission

import data.kubernetes.admission.quotas

deny [msg] {
    input.review.kind.kind == "Deployment"
    not quotas[input.review.object.metadata.namespace]
    sprintf("namespace %s has no quota defined", [input.review.object.metadata.namespace]) != msg
}

deny [msg] {
    input.review.kind.kind == "Deployment"
    q := quotas[input.review.object.metadata.namespace]
    not input.review.object.spec.template.spec.containers[_].resources.limits.cpu <= q.cpu
    msg := sprintf("Deployment %s exceeds tenant CPU limit of %s", [input.review.object.metadata.name, q.cpu])
}

The quotas data source would be populated externally, perhaps via a Kubernetes ConfigMap or a dynamic admission webhook. This pattern allows operators to define per-tenant limits in a central location, and Gatekeeper automatically enforces them across the cluster.

3.2 Tenant-Aware Scoping with Data Entries

Gatekeeper’s data entries can inject runtime information into Rego evaluations. For multi-tenancy, a typical approach is to pass the requesting user or service account as a data entry, and then use Rego logic to determine the effective tenant. For example:

package kubernetes.admission

default tenant_id := "unknown"

tenant_id := input.review.object.metadata.labels.tenant_id

A Constraint then references tenant_id to look up the appropriate quota. This pattern decouples policy logic from hardcoded tenant names, making it straightforward to onboard new tenants without modifying Rego code.

3.3 OPA Gatekeeper Policy Testing

OPA provides opa test to validate Rego policies against input/output pairs. A typical test suite for a constraint template might include JSON files representing a valid Deployment (within quota) and an invalid one (exceeding CPU limits). The test command returns pass/fail for each case, and failures are surfaced with the exact Rego rule that rejected the request. This feedback loop is critical for iterating on policy logic without causing cluster-wide outages.

4. Architecture: Layered Collaboration of Kyverno and OPA

The most robust multi-tenant admission control architectures separate concerns: Kyverno handles namespace lifecycle, label propagation, and default resource provisioning; OPA Gatekeeper enforces resource-shaped constraints and business logic. This separation reduces policy overlap and makes each engine’s purpose clear.

4.1 Where Each Engine Runs

Kyverno operates as a Kubernetes controller that watches the API server for resource events. It does not require a separate admission webhook deployment; it integrates directly via the Kubernetes admission pipeline. OPA Gatekeeper, conversely, deploys an admission webhook (gatekeeper-admission-controller) that intercepts requests after Kyverno’s native mutations have been applied, if applicable. The order of execution matters: if Kyverno mutates a Namespace to add a label, Gatekeeper’s subsequent validation can reference that label because it sees the mutated state.

4.2 Performance Considerations

Both engines are designed for low-latency enforcement. Kyverno’s policy evaluation is cached per-resource type, and its admission requests are served from the API server’s local cache. OPA Gatekeeper compiles Rego policies into highly optimized decision trees; in production clusters with hundreds of tenants, latency remains sub-millisecond per admission request. However, complex Rego rules with deep data dependencies can introduce overhead. Profiling with opa eval and iterating on rule granularity is recommended.

4.3 Complementary, Not Redundant

A frequent pitfall is writing overlapping policies in both Kyverno and Gatekeeper, leading to confusing error messages when one engine denies what the other allowed. A clear ownership map resolves this: Kyverno owns “what gets created and how it’s shaped initially”; Gatekeeper owns “what is allowed given the current state and business rules.” Documentation and policy naming conventions (e.g., kyverno/inject-tenant-id vs gatekeeper/limit-cpu-per-tenant) make ownership explicit.

5. Patterns in Production: CI/CD, Testing, and Rollout

5.1 Policy Versioning in Git

Treat policies as application code. Store Kyverno Policies and OPA Constraints in a Git repository under policy/ directories, organized by tenant or domain. Use semantic versioning for policy bundles, and leverage GitOps tools to synchronize the desired policy state to the cluster. This approach enables pull-request-based review, change tracking, and rollback.

5.2 Automated Testing with OPA Test and Kyverno Test

Integrate kyverno test and opa test into your CI pipeline. A typical pipeline stage might:

  1. Lint policies using kyverno lint and opa fmt.
  2. Run unit tests against a set of admission review JSON fixtures.
  3. Deploy a temporary test cluster (kind or k3s) and apply policies.
  4. Execute end-to-end scenarios: create a tenant namespace, deploy a workload, verify quotas and labels.

Both tools emit structured output (JUnit-compatible for Kyverno, JSON for OPA) that can be consumed by CI dashboards.

5.3 Gradual Admission Control Rollout

Enabling admission control on a production cluster requires care. Kyverno and Gatekeeper both support a dry-run mode: Kyverno’s --dry-run flag and Gatekeeper’s skipRulesWithLabel: bypass can be used to evaluate policies without enforcing them. A phased rollout strategy might look like:

  1. Evaluation: Deploy policies in dry-run mode, collect audit logs, and iterate on rules.
  2. Warning: Enable policies with validate operations but set ignoreNotFound: true to avoid hard failures on misconfigured resources.
  3. Enforcement: Transition to enforce mode once baseline coverage is confirmed.

Monitoring policy denials via Kyverno’s kyverno get violation and Gatekeeper’s kubectl get constraintviolations provides visibility into real-world impact and guides further refinements.

6. Key Takeaways

  • Admission control is the gateway for multi-tenant isolation, enforcing boundaries at the point of resource creation or modification.
  • Kyverno excels at native Kubernetes policy operations: namespace lifecycle, label injection, and default quota propagation. Its policy-as-code lives as Kubernetes resources, enabling GitOps workflows.
  • OPA Gatekeeper provides expressive, logic-based validation via Rego, ideal

Building something like this? I’m an AI/systems contractor open to new projects. Book a 30-min call.