Kafka in Production · Kafka Patterns and Operations
Message Schemas and Safe Evolution
Treat message formats as contracts: version them, evolve them compatibly, and validate messages before producing them.
What you'll build
A versioned OrderPlaced event and a compatibility check that a producer change must pass before deployment: adding an optional field is allowed, removing or renaming a required field is rejected, because consumers of older versions would break.
The idea
- Topics are shared contracts between teams. Changing a message format can break consumers you don't know about.
- Backward compatible changes let new consumers read old messages; forward compatible changes let old consumers read new messages. Adding optional fields with defaults is usually both.
- A schema registry (with Avro, Protobuf or JSON Schema) stores versions and enforces compatibility rules on every producer; the check below shows the idea in plain Python.
- Include an event type and version in each message (or rely on the registry's schema ID).
The code
SCHEMA_V1 = {"name": "OrderPlaced", "version": 1,
"fields": {"order_id": {"type": "string", "required": True},
"total_cents": {"type": "int", "required": True},
"currency": {"type": "string", "required": True}}}
def compatible(old, new):
'''Old consumers must understand new messages, and new consumers must understand old ones.'''
problems = []
for name, spec in old["fields"].items():
if name not in new["fields"]:
problems.append(f"removed field '{name}'")
elif new["fields"][name]["type"] != spec["type"]:
problems.append(f"changed type of '{name}'")
for name, spec in new["fields"].items():
if name not in old["fields"] and spec["required"] and "default" not in spec:
problems.append(f"new required field '{name}' without a default")
return problems
def with_fields(**changes):
fields = dict(SCHEMA_V1["fields"])
for name, spec in changes.items():
if spec is None:
fields.pop(name)
else:
fields[name] = spec
return {**SCHEMA_V1, "version": 2, "fields": fields}
proposals = {
"add optional coupon": with_fields(coupon={"type": "string", "required": False}),
"add required channel with default": with_fields(channel={"type": "string", "required": True, "default": "web"}),
"rename total_cents -> amount": with_fields(total_cents=None, amount={"type": "int", "required": True}),
"total_cents as string": with_fields(total_cents={"type": "string", "required": True}),
}
for change, schema in proposals.items():
problems = compatible(SCHEMA_V1, schema)
print(f"{change}: {'OK' if not problems else 'REJECTED - ' + '; '.join(problems)}")
Output
add optional coupon: OK add required channel with default: OK rename total_cents -> amount: REJECTED - removed field 'total_cents'; new required field 'amount' without a default total_cents as string: REJECTED - changed type of 'total_cents'
Run this on your computer — see “Get Started” for setup.
A rename is a remove plus an add: do it in steps (add the new field, migrate consumers, stop writing the old one) or publish a new event type.
Common mistakes
- Free-form JSON with no contract, so one producer change breaks downstream services silently.
- Renaming or retyping fields in place.
- Putting large payloads or PII-heavy data in events "just in case"; publish what consumers need.
Key points
- Message formats are contracts; version them.
- Only make compatible changes (add optional fields with defaults).
- Enforce compatibility automatically with a schema registry.
Further reading: Schema evolution and compatibility (Confluent) · Apache Avro specification
Practice questions
What's a backward-compatible schema change?
Answer
One that lets consumers using the new schema read data written with the old one, such as adding an optional field with a default.
Why is renaming a field a breaking change?
Answer
To readers it's a removed field plus a new one; consumers expecting the old name break. Migrate in steps or publish a new event type.
What does a schema registry add?
Answer
Central storage of schema versions and automatic compatibility checks when producers register or use a new schema.
Open in the app to mark lessons complete, take notes and track your progress.