TL;DR — Apache Pulsar Schema Registry centralizes schema management, enabling safe evolution and strong consistency across producers and consumers in real‑time pipelines. It eliminates schema drift by validating messages against registered schemas and supports backward and forward compatibility rules, so downstream services can upgrade without downtime.

Apache Pulsar has become a de‑facto standard for low‑latency, multi‑tenant messaging, and its Schema Registry layer adds a powerful mechanism for handling evolving data models without breaking downstream consumers. In this post we explore how to integrate the registry, the architectural guarantees it provides, and practical patterns for deploying it at scale.

Overview

Pulsar’s Schema Registry (SR) is an independent service that stores Avro, JSON, or Protobuf schemas and assigns each a unique identifier. Producers embed the schema ID with every message, while consumers retrieve the schema by ID, guaranteeing that every subscriber sees the same version of the data. This design decouples schema evolution from the messaging layer, allowing teams to evolve contracts independently of Pulsar clusters.

Key guarantees provided by the registry:

  • Schema validation – every published message is validated against its registered schema before it lands in a topic.
  • Compatibility modes – configurable rules (NONE, BACKWARD, FORWARD, FULL, FULL_STRICT) that reject incompatible schema changes at write time.
  • Versioned storage – each schema revision is immutable, enabling rollbacks and audits.

Because the registry sits alongside Pulsar but outside the broker process, it can scale horizontally, be secured with TLS, and integrate with identity providers such as OIDC or LDAP.

Architecture & Patterns in Production

A typical production deployment follows a “registry‑as‑a‑service” pattern. The diagram below (conceptual) shows the data flow:

  1. Producer – serializes the record, looks up or registers the schema in SR, and attaches the schema ID to the Pulsar message.
  2. Pulsar broker – forwards the message to the appropriate topic; the broker itself is schema‑agnostic.
  3. Consumer – reads the schema ID from the message, fetches the corresponding schema from SR, and deserializes the payload.

Common patterns

PatternDescriptionWhen to use
Side‑car registryDeploy the SR container alongside each Pulsar broker node; reduces network hops and improves latency.Large clusters where every millisecond of latency matters.
Centralized registrySingle (or multi‑region) SR instance accessed over TLS from all producers/consumers.Most common; simplifies ops and provides a single source of truth.
Schema per topicEach Pulsar topic has its own SR namespace, preventing cross‑topic schema conflicts.Multi‑tenant platforms where isolation is a requirement.
Compatibility enforcementSet the compatibility mode per namespace; the producer will block writes that violate the rule.When downstream consumers cannot tolerate missing fields.

A real‑world example is a clickstream pipeline that ingests events from web front‑ends into Pulsar. The front‑end sends JSON events; the SR stores the evolving event schema. When the product team adds a new user_id field, they register a new version with FORWARD compatibility. Existing consumers that do not yet know the field simply ignore it, while new consumers can read the enriched data immediately.

Integration with Pulsar Functions

Pulsar Functions can be written to interact with the SR directly. For instance, a Python function can call the REST endpoint GET /schemas/{schemaId} to fetch the latest version before processing. This is useful for stateful functions that need to maintain per‑schema state, such as aggregations that differ per schema version.

Code snippet (Python) for fetching a schema:

import requests

def fetch_schema(schema_id: str) -> dict:
    resp = requests.get(
        f"https://schema-registry.example.com/v2/schemas/{schema_id}",
        headers={"Authorization": "Bearer $TOKEN"},
    )
    resp.raise_for_status()
    return resp.json()

Integration with Pulsar Topics

Registering a schema is straightforward via the HTTP API or the Pulsar CLI.

# Register an Avro schema for the topic "persistent://public/default/clickstream"
pulsar schema register \
  --type avro \
  --schema-file clickstream.avsc \
  --name clickstream \
  --url http://schema-registry:8080

Once registered, producers can publish messages without specifying the full schema; they only need the schema ID. The Pulsar client libraries (Java, Python, Go, C++) automatically include the ID in the message properties.

Example (Java producer)

SchemaInfo schema = schemaRegistry.getSchema("clickstream");
Producer<GenericRecord> producer = client.newProducer(Schema.AVRO)
    .topic("persistent://public/default/clickstream")
    .schema(schema)
    .create();

GenericRecord record = SpecificRecord.create(Clickstream.class);
record.put("event_id", "12345");
record.put("timestamp", System.currentTimeMillis());
producer.send(record);

When the schema evolves, the producer can re‑register the new version; existing consumers continue to work as long as the compatibility mode permits the change.

Schema Evolution & Compatibility

Schema evolution is the most delicate part of any streaming system. Pulsar SR offers five compatibility modes, each with distinct semantics:

ModeAllows …Typical use case
NONEAny change, breaking or not.Internal prototypes where breaking changes are acceptable.
BACKWARDNew schema can read old data.Adding optional fields; backward‑compatible additions.
FORWARDOld schema can read new data.Removing optional fields; forward‑compatible deletions.
FULLBoth backward and forward compatible.Most production environments; safe additions + removals of optional fields.
FULL_STRICTSame as FULL, but also enforces field‑order and type‑exactness.When strict Avro compatibility is required.

Choosing the right mode early avoids costly retrofits. For example, a social‑feed platform that frequently adds reaction types would typically use FULL, because new reaction types must be visible to old clients while old reaction types must not cause errors on new clients.

Compatibility checks in practice

When a producer attempts to publish a message with a schema ID that references a newer version, the SR validates the payload against the schema definition. If the payload violates the compatibility mode, the request is rejected with a 409 Conflict response, and the producer can either:

  • Update its code to emit the new field.
  • Downgrade the compatibility mode (not recommended for production).

The SR also stores schema evolution logs, which can be queried to audit who changed a schema and when. This is invaluable for compliance‑driven industries (financial services, healthcare).

Monitoring, Security, and Operations

A production‑grade SR deployment requires robust observability and security hardening.

Metrics

The SR exposes Prometheus metrics under the /metrics endpoint. Key metrics include:

  • schema_registry_requests_total – total number of read/write requests broken down by endpoint.
  • schema_registry_validation_errors_total – count of validation failures, useful for spotting incompatible changes.
  • schema_registry_cache_hit_ratio – indicates how often the client caches schema lookups, affecting latency.

Alert on a sudden spike in validation_errors_total; it often signals an unintended breaking change.

TLS & Authentication

Serve the SR over HTTPS with a valid certificate. Authentication can be configured via:

  • Basic auth (username/password) – simple for internal deployments.
  • OIDC/LDAP – integrate with corporate identity providers for SSO.
  • mTLS – mutual TLS for zero‑trust environments, where both client and server present certificates.

High Availability

Deploy the SR as a replicated cluster (e.g., using Kubernetes StatefulSet) with Raft or similar consensus to avoid a single point of failure. The Pulsar client libraries automatically retry on connection errors, and the registry can be configured with a quorum size to tolerate node failures.

Key Takeaways

  • Centralized schema management eliminates drift and gives producers & consumers a single source of truth.
  • Compatibility modes (BACKWARD, FORWARD, FULL, FULL_STRICT) let you control which evolutionary changes are allowed, preventing runtime failures.
  • Versioned, immutable schemas enable rollbacks and audits, essential for regulated domains.
  • Integration points—Pulsar clients, Functions, and the REST API—make it easy to embed schema checks into existing pipelines.
  • Observability (Prometheus metrics, logs, alerts) and TLS/mTLS are non‑negotiable for production security and reliability.
  • Deployment patterns (side‑car vs. centralized) let you trade off latency against operational simplicity; choose based on your cluster size and latency requirements.

Further Reading