System Design (HLD)Event Schema Evolution

Event Schema Evolution

LevelAdvanced

Duration90 mins

TopicEvent Schema Evolution

3 / 5

Forward Compatibility

The Producer Protection Principle

If backward compatibility protects consumers from producer changes, forward compatibility does the opposite—it protects producers from consumer changes. Forward compatibility ensures that old events (from old producers) can be read by new consumers.

Why does this matter? Consider a rolling deployment: you're upgrading consumers one by one while producers continue churning out events in the old format. Or imagine consumers that need new features immediately—they deploy ahead of producers to handle new fields gracefully when they eventually arrive.

Forward compatibility is the less-discussed sibling of backward compatibility, but it's equally important for truly decoupled, independently deployable systems.

What You Will Learn

By the end of this page, you will understand forward compatibility: why new consumers must handle old events, how to implement forward-compatible schemas, the relationship between forward and backward compatibility, and patterns for handling missing fields that consumers expect.

What is Forward Compatibility?

Forward compatibility means that consumers using a newer schema version can successfully process events produced with an older schema version. The "reader" code is newer than the "writer" code.

The mental model:

Imagine you deploy a new consumer version today that expects a currency field on every order event. But old producers (deployed last month) are still running and emitting orders without the currency field. For forward compatibility, your new consumer must:

Successfully deserialize old events (no parsing failures)
Handle missing expected fields (use defaults or fallbacks)
Not crash when new fields are absent
Maintain correct business logic despite incomplete data

Converting Mermaid diagram...

Key distinction from backward compatibility:

Aspect	Backward Compatibility	Forward Compatibility
Scenario	New producer → Old consumer	Old producer → New consumer
Who must adapt	Old consumer ignores new fields	New consumer handles missing fields
Breaking action	Remove/change existing fields	Add required fields without defaults
Primary benefit	Producer can evolve freely	Consumer can deploy ahead of producer

Deployment Flexibility

Forward compatibility is essential for rolling deployments and blue-green deployments. During the transition, you'll have old producers feeding new consumers (forward compatibility) AND new producers feeding old consumers (backward compatibility). You need both.

Why Forward Compatibility Matters

Forward compatibility might seem less critical than backward compatibility—after all, how often do consumers deploy before producers? More often than you'd think:

Common forward compatibility scenarios:

When Forward Compatibility is Needed

•Rolling deployments — Consumers upgrade one pod/instance at a time while producers keep running old code. New consumer instances receive events from old producer instances.
•Blue-green deployments — New consumer version runs alongside old version. Traffic gradually shifts. Both versions receive events from any producer version.
•Consumer-first features — A consumer team needs a new field for analytics. They deploy code that uses the field if present. Producer adds the field later.
•Multi-producer environments — Multiple microservices produce the same event type. They upgrade at different times. Consumers must handle the oldest producer's format.
•Event replay/reprocessing — Historical events from years ago are replayed through current consumer code. Current consumers see very old schemas.
•Third-party producers — External systems produce events on their schedule. You can't control when they upgrade.

The cost of forward incompatibility:

Without forward compatibility, consumer deploys are blocked until all producers upgrade. This creates:

Deployment coupling — Can't ship consumer improvements until producers are ready
Coordination overhead — Must synchronize releases across teams
Rollback risk — Rolling back producers requires rolling back consumers too
Feature delays — New consumer features wait on producer timelines

The Hidden Coupling

Lack of forward compatibility creates hidden coupling between teams. The consumer team's release schedule becomes dependent on the producer team's priorities. This is exactly the coupling microservices aim to eliminate.

Forward Compatible Schema Design

Forward compatibility requires consumers to handle missing fields gracefully. This has implications for how we design both schemas and consumer code.

Core principle: Every field a consumer accesses must either:

Be guaranteed present in all schema versions, OR
Have a safe default/fallback when absent

Forward Compatible Patterns

•Always define defaults for new fields
•Never add required fields to existing events
•Use optionals for all non-original fields
•Check field presence before access
•Provide fallback logic for missing data

Forward Breaking Patterns

•Adding required fields — Old events lack them
•Assuming field presence in new code
•Strict validation that rejects old events
•Required nested objects on existing events
•Non-nullable types for new fields

forward-compatible-schema.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
// FORWARD COMPATIBLE: All new fields optional with defaults
interface OrderCreatedV3 {
  // Original fields (from v1) — always present
  orderId: string;
  customerId: string;
  items: OrderItem[];
  totalAmount: number;
  
  // v2 additions — may be absent in v1 events
  currency?: string;            // Default: 'USD'
  shippingAddress?: Address;    // Default: null (use customer default)
  
  // v3 additions — may be absent in v1/v2 events
  loyaltyTier?: 'bronze' | 'silver' | 'gold';  // Default: 'bronze'
  estimatedDelivery?: string;   // Default: calculated from items
  promotionCodes?: string[];    // Default: []
}
 
// FORWARD INCOMPATIBLE: Adding required field
interface OrderCreatedV3Bad {
  orderId: string;
  customerId: string;
  items: OrderItem[];
  totalAmount: number;
  currency: string;  // BREAKING: Required field added in v3
                     // Old events (v1, v2) don't have this!
}
 
// JSON Schema version — note all new fields are NOT required
{
  "type": "object",
  "required": ["orderId", "customerId", "items", "totalAmount"],  // Original only
  "properties": {
    "orderId": { "type": "string" },
    "customerId": { "type": "string" },
    "items": { "type": "array" },
    "totalAmount": { "type": "number" },
    "currency": { "type": "string", "default": "USD" },
    "shippingAddress": { "$ref": "#/definitions/Address" },
    "loyaltyTier": { 
      "type": "string", 
      "enum": ["bronze", "silver", "gold"],
      "default": "bronze"
    },
    "estimatedDelivery": { "type": "string", "format": "date-time" },
    "promotionCodes": { "type": "array", "items": { "type": "string" }, "default": [] }
  }
}

The Required Fields Rule

Once an event type is in production, its required fields are locked. You can never add a new required field to an existing event type—only to new event types. This is the most common forward compatibility violation.

Handling Missing Fields in Consumers

The real work of forward compatibility happens in consumer code. New consumers must gracefully handle the absence of fields they want to use.

Missing field strategies:

Use schema-defined or code-defined defaults when fields are absent.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
// TypeScript: Defaults during destructuring
function processOrder(event: OrderCreatedV3) {
  const {
    orderId,
    customerId,
    totalAmount,
    // Defaults for potentially missing fields
    currency = 'USD',
    loyaltyTier = 'bronze',
    promotionCodes = [],
  } = event;
  
  // Safe to use — defaults applied
  const formattedTotal = formatCurrency(totalAmount, currency);
  const discount = calculateLoyaltyDiscount(totalAmount, loyaltyTier);
}
 
// Java: Optional with orElse
public void processOrder(OrderCreated event) {
    String currency = Optional.ofNullable(event.getCurrency())
                              .orElse("USD");
    
    List<String> promoCodes = Optional.ofNullable(event.getPromotionCodes())
                                      .orElse(Collections.emptyList());
}
 
// Python: dict.get with default
def process_order(event: dict):
    currency = event.get('currency', 'USD')
    loyalty_tier = event.get('loyaltyTier', 'bronze')
    promo_codes = event.get('promotionCodes', [])

Monitor Missing Field Rates

Track how often new consumers encounter events without expected fields. This metric shows migration progress and identifies producers that haven't upgraded. High missing-field rates might indicate a producer stuck on old version.

Full Compatibility

Full compatibility (or bidirectional compatibility) combines backward and forward compatibility. It's the gold standard for event schemas in microservices.

Full compatibility guarantees:

New producer events work with old consumers (backward)
Old producer events work with new consumers (forward)
Any producer can work with any consumer (within MAJOR version)

Converting Mermaid diagram...

Achieving full compatibility:

Full compatibility is more restrictive than either direction alone. The rules are cumulative:

Rule	Backward	Forward	Full
Don't remove fields	✓		✓
Don't change field types	✓		✓
Don't add required fields		✓	✓
New fields must have defaults		✓	✓
Consumers must ignore unknown	✓		✓
Consumers must handle missing		✓	✓

full-compatibility-rules.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
// FULLY COMPATIBLE evolution: v1 → v2 → v3
 
// v1 (Original)
interface OrderCreatedV1 {
  orderId: string;      // Required - original
  customerId: string;   // Required - original  
  totalAmount: number;  // Required - original
}
 
// v2 (Fully compatible with v1)
interface OrderCreatedV2 {
  orderId: string;
  customerId: string;
  totalAmount: number;
  currency?: string;              // ADDED: Optional with default
  shippingAddress?: Address;      // ADDED: Optional, nullable
}
 
// v3 (Fully compatible with v1 AND v2)
interface OrderCreatedV3 {
  orderId: string;
  customerId: string;
  totalAmount: number;
  currency?: string;
  shippingAddress?: Address;
  loyaltyTier?: LoyaltyTier;      // ADDED: Enum, optional
  promotionCodes?: string[];      // ADDED: Array, defaults to []
}
 
// Compatibility matrix:
// - v1 consumer reads v3 event: ✓ (ignores new fields)
// - v3 consumer reads v1 event: ✓ (uses defaults for missing)
// - v2 consumer reads v3 event: ✓ (ignores v3 additions)
// - v3 consumer reads v2 event: ✓ (uses defaults for v3 fields)
 
// Schema registry configuration
const schemaConfig = {
  subject: 'order-created',
  compatibilityMode: 'FULL',  // Enforce bidirectional compatibility
  // Alternatives:
  // 'BACKWARD' - only backward compatible
  // 'FORWARD' - only forward compatible
  // 'NONE' - no compatibility checks
};

When to Use Full Compatibility

Use full compatibility by default for all event schemas in microservices. The additional constraints (no required field additions) are minor compared to the deployment flexibility gained. Reserve looser compatibility modes for very specific scenarios where the tradeoff is justified.

Transitive Compatibility

Transitive compatibility extends compatibility guarantees across all historical versions, not just adjacent ones.

Non-transitive vs. transitive:

Non-transitive: Each version compatible with immediate predecessor/successor only
Transitive: Each version compatible with ALL previous/subsequent versions

Why transitive matters:

transitive-compatibility.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// NON-TRANSITIVE COMPATIBILITY (DANGEROUS)
// Each version compatible only with adjacent version
 
// v1: { orderId, totalAmount }
// v2: { orderId, totalAmount, currency? }  // Compatible with v1
// v3: { orderId, totalAmount, currency?, tax? }  // Compatible with v2
 
// But what if we skip v2 during upgrade?
// v1 producer → v3 consumer
// v3 consumer might ASSUME currency is commonly present
// because v2 established it. But v1 never had it!
 
// TRANSITIVE COMPATIBILITY (SAFE)
// Each version compatible with ALL historical versions
 
interface CompatibilityCheck {
  subject: string;
  version: number;
  checkAgainst: 'LATEST' | 'ALL';  // ALL = transitive
}
 
// Schema registry with transitive enforcement
// Before accepting v4, checks compatibility with v3, v2, AND v1
async function registerSchema(newSchema: Schema): Promise<void> {
  const allVersions = await registry.getAllVersions(newSchema.subject);
  
  for (const oldVersion of allVersions) {
    const oldSchema = await registry.getSchema(newSchema.subject, oldVersion);
    
    // Must be compatible with EVERY historical version
    if (!isCompatible(newSchema, oldSchema)) {
      throw new IncompatibleSchemaError(
        `v${newSchema.version} incompatible with v${oldVersion}`
      );
    }
  }
  
  await registry.register(newSchema);
}

Schema Registry Compatibility Modes
Mode	Checks Against	Use Case
BACKWARD	Latest version only	Consumers always current; producers may lag
BACKWARD_TRANSITIVE	All previous versions	Long-running consumers; event replay
FORWARD	Latest version only	Producers always current; consumers may lag
FORWARD_TRANSITIVE	All previous versions	Multiple producer versions in parallel
FULL	Latest version only	Independent deployment (common case)
FULL_TRANSITIVE	All previous versions	Maximum flexibility; event sourcing

Event Sourcing Requires Transitive

If you're doing event sourcing, transitive compatibility is mandatory. Event replay reads ALL historical events through current code. A consumer might process v1 events from years ago immediately followed by v5 events from today. Non-transitive compatibility will break replay.

Forward Compatibility Challenges

Forward compatibility presents unique challenges that don't exist with backward compatibility:

Challenge 1: Feature requirement mismatch

A new consumer feature requires data that old events don't have. You can't magically make old producers include new fields.

feature-requirement-mismatch.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
// New feature: Real-time fraud scoring
// REQUIRES: customerDeviceFingerprint, paymentMethodToken
 
async function scoreFraudRisk(event: OrderCreatedV3): Promise<FraudScore> {
  // New fields required for fraud scoring
  if (!event.deviceFingerprint || !event.paymentMethodToken) {
    // Old events lack this data — can't score accurately!
    return {
      score: 0.5,  // Neutral score
      confidence: 'LOW',
      reason: 'Insufficient data for accurate scoring',
    };
  }
  
  // Full scoring with v3 data
  return await fraudEngine.score({
    device: event.deviceFingerprint,
    payment: event.paymentMethodToken,
    customer: event.customerId,
    amount: event.totalAmount,
  });
}
 
// Solution: Graceful degradation with backfill
async function scoreFraudRiskWithBackfill(
  event: OrderCreatedV3
): Promise<FraudScore> {
  // Attempt to backfill missing data from other sources
  const device = event.deviceFingerprint ?? 
                 await sessionService.getLastDevice(event.customerId);
  
  const payment = event.paymentMethodToken ??
                  await paymentService.getDefaultMethod(event.customerId);
  
  if (!device || !payment) {
    // Still can't score — this is expected for old events
    return { score: 0.5, confidence: 'LOW', reason: 'Data unavailable' };
  }
  
  // Score with backfilled data (lower confidence)
  const result = await fraudEngine.score({ device, payment, ... });
  return { ...result, confidence: 'MEDIUM' };  // Lower than native data
}

Challenge 2: Semantic evolution

The meaning of fields changes over time. An old 'status' value might mean something different than a new 'status' value.

semantic-evolution.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// v1: status = 'PENDING' | 'ACTIVE' | 'CLOSED'
// v3: status = 'PENDING' | 'ACTIVE' | 'COMPLETED' | 'CANCELLED' | 'REFUNDED'
//     ('CLOSED' was split into 'COMPLETED', 'CANCELLED', 'REFUNDED')
 
function processAccountStatus(event: AccountUpdatedV3): void {
  switch (event.status) {
    case 'PENDING':
    case 'ACTIVE':
      // Same meaning across versions
      handleActiveFlow(event);
      break;
      
    case 'COMPLETED':
    case 'CANCELLED':
    case 'REFUNDED':
      // New granular statuses
      handleClosedFlow(event, event.status);
      break;
      
    case 'CLOSED':
      // LEGACY: v1 events use 'CLOSED'
      // We don't know if it was completed, cancelled, or refunded
      // Best effort: Treat as completed (most common case)
      console.warn('Legacy CLOSED status', { accountId: event.accountId });
      handleClosedFlow(event, 'COMPLETED');
      break;
      
    default:
      // Unknown status — forward compatibility
      handleUnknownStatus(event);
  }
}

Semantic Changes Need Documentation

When field semantics evolve, document the mapping between old and new meanings. Consumer code can then translate legacy values appropriately. This documentation should live with the schema definition, not buried in consumer code.

Testing Forward Compatibility

Testing forward compatibility requires samples from all historical schema versions. New consumer code must process events from every previous version successfully.

Testing approach:

forward-compatibility-tests.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
// Test fixture: Historical event samples
const historicalEventSamples = {
  'v1.0.0': [
    { orderId: 'v1-order-1', customerId: 'c1', totalAmount: 100 },
    { orderId: 'v1-order-2', customerId: 'c2', totalAmount: 250 },
  ],
  'v2.0.0': [
    { orderId: 'v2-order-1', customerId: 'c1', totalAmount: 100, currency: 'USD' },
    { orderId: 'v2-order-2', customerId: 'c2', totalAmount: 250, currency: 'EUR', 
      shippingAddress: { street: '123 Main', city: 'Berlin' } },
  ],
  'v3.0.0': [
    { orderId: 'v3-order-1', customerId: 'c1', totalAmount: 100, currency: 'USD',
      loyaltyTier: 'gold', promotionCodes: ['SAVE10'] },
  ],
};
 
describe('Forward Compatibility: New Consumer with Old Events', () => {
  const consumer = new OrderConsumerV3();  // Current consumer
  
  Object.entries(historicalEventSamples).forEach(([version, events]) => {
    describe(`Processing ${version} events`, () => {
      events.forEach((event, index) => {
        it(`handles ${version} event #${index}`, async () => {
          // Should not throw
          const result = await consumer.process(event);
          
          // Core processing works
          expect(result.orderId).toBe(event.orderId);
          expect(result.processed).toBe(true);
          
          // Defaults applied for missing fields
          expect(result.currency).toBe(event.currency ?? 'USD');
          expect(result.loyaltyTier).toBe(event.loyaltyTier ?? 'bronze');
        });
      });
    });
  });
  
  // Test edge cases
  describe('Missing field edge cases', () => {
    it('handles v1 event with no optional fields', async () => {
      const v1Event = { orderId: 'test', customerId: 'c', totalAmount: 50 };
      const result = await consumer.process(v1Event);
      
      // All defaults applied
      expect(result.currency).toBe('USD');
      expect(result.shippingAddress).toBeNull();
      expect(result.loyaltyTier).toBe('bronze');
      expect(result.promotionCodes).toEqual([]);
    });
    
    it('handles partial v2 event (some optionals present)', async () => {
      const partialV2 = { 
        orderId: 'test', customerId: 'c', totalAmount: 50,
        currency: 'GBP'  // Has currency, no shippingAddress
      };
      const result = await consumer.process(partialV2);
      
      // Mixed: explicit value + defaults
      expect(result.currency).toBe('GBP');
      expect(result.shippingAddress).toBeNull();
    });
  });
});
 
// Integration test: Replay historical events
describe('Event Replay Forward Compatibility', () => {
  it('replays entire event history without errors', async () => {
    const eventStore = new EventStore();
    const consumer = new OrderConsumerV3();
    
    // Fetch all historical events
    const allEvents = await eventStore.getAllEvents('order-created');
    
    let errorCount = 0;
    for (const event of allEvents) {
      try {
        await consumer.process(event);
      } catch (error) {
        errorCount++;
        console.error('Replay failed:', event.orderId, error);
      }
    }
    
    expect(errorCount).toBe(0);
  });
});

Maintain Historical Samples

Create and version-control sample events from each schema version. These samples are your forward compatibility test fixtures. Without them, you're testing against synthetic data that might not represent real historical quirks.

Migration Strategies

When evolving schemas, the order of producer and consumer deployment matters. Different strategies have different compatibility requirements:

Strategy 1: Consumer-first deployment

Consumer-First Deployment

•Step 1: Deploy new consumers that handle both old AND new event formats
•Step 2: Verify consumers correctly process old events (forward compatibility test)
•Step 3: Deploy new producers that emit new format
•Step 4: Verify consumers correctly process new events
•Rollback: If producers fail, consumers continue processing old format

Strategy 2: Producer-first deployment

Producer-First Deployment

•Step 1: Deploy new producers (new format is backward compatible)
•Step 2: Verify old consumers correctly process new events (backward compatibility)
•Step 3: Deploy new consumers that leverage new fields
•Step 4: Verify new consumers work with new events
•Rollback: If consumers fail, roll back consumers; producers keep running

deployment-coordination.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
// Deployment coordinator for schema migration
 
interface DeploymentPlan {
  schemaVersion: string;
  strategy: 'CONSUMER_FIRST' | 'PRODUCER_FIRST';
  steps: DeploymentStep[];
}
 
const consumerFirstPlan: DeploymentPlan = {
  schemaVersion: '3.0.0',
  strategy: 'CONSUMER_FIRST',
  steps: [
    {
      phase: 1,
      action: 'DEPLOY_CONSUMERS',
      services: ['analytics', 'shipping', 'notifications'],
      validation: 'Consumers process v2 events without errors',
      rollbackTrigger: 'Error rate > 0.1%',
    },
    {
      phase: 2,
      action: 'WAIT',
      duration: '24h',
      validation: 'Consumers stable in production',
    },
    {
      phase: 3,
      action: 'DEPLOY_PRODUCERS',
      services: ['order-service'],
      validation: 'Producers emit v3 events, consumers process successfully',
      rollbackTrigger: 'Consumer error rate > 0.1%',
    },
    {
      phase: 4,
      action: 'CLEANUP',
      tasks: ['Remove v2 support from producers', 'Update documentation'],
    },
  ],
};
 
// Feature flags for gradual rollout
const schemaFeatureFlags = {
  'emit-v3-fields': {
    enabled: true,
    percentage: 10,  // Start with 10% of events
    monitors: ['consumer-error-rate', 'processing-latency'],
    autoRollback: {
      errorRateThreshold: 0.5,
      latencyP99Threshold: 500,
    },
  },
};

Consumer-First is Usually Safer

Consumer-first deployment is generally safer because it validates forward compatibility before any new data is produced. If the new consumer fails with old events, you can fix it without affecting production data. Producer-first risks producing events that break consumers.

Summary: Forward Compatibility

Forward compatibility ensures new consumers can process old events. Let's consolidate the key takeaways:

Key Takeaways

•Forward compatibility enables consumer-ahead deployment — New consumers work before producers upgrade.
•Never add required fields to existing events — Old events won't have them; new consumers will fail.
•All new fields must be optional with defaults — Consumers must handle absence gracefully.
•Handle missing fields programmatically — Use defaults, computed fallbacks, or conditional logic.
•Full compatibility combines backward AND forward — Maximum deployment flexibility.
•Transitive compatibility spans all versions — Essential for event replay and event sourcing.
•Test with historical event samples — Verify new consumer code processes all old schema versions.
•Consumer-first deployment is often safer — Validates forward compatibility before producing new data.

What's next:

With forward and backward compatibility understood, the next page explores schema registries—the infrastructure that manages schemas, enforces compatibility, and provides schema discovery for producers and consumers.

Page Complete

You now understand forward compatibility and its role in enabling independent consumer deployment. You can design forward-compatible schemas, implement consumers that handle missing fields, and plan deployment strategies that leverage both forward and backward compatibility.

3 / 5

Loading learning content...

System Design (HLD)Event Schema Evolution

Event Schema Evolution

LevelAdvanced

Duration90 mins

TopicEvent Schema Evolution

3 / 5

Forward Compatibility

The Producer Protection Principle

Forward compatibility is the less-discussed sibling of backward compatibility, but it's equally important for truly decoupled, independently deployable systems.

What You Will Learn

What is Forward Compatibility?

Forward compatibility means that consumers using a newer schema version can successfully process events produced with an older schema version. The "reader" code is newer than the "writer" code.

The mental model:

Successfully deserialize old events (no parsing failures)
Handle missing expected fields (use defaults or fallbacks)
Not crash when new fields are absent
Maintain correct business logic despite incomplete data

Converting Mermaid diagram...

Key distinction from backward compatibility:

Aspect	Backward Compatibility	Forward Compatibility
Scenario	New producer → Old consumer	Old producer → New consumer
Who must adapt	Old consumer ignores new fields	New consumer handles missing fields
Breaking action	Remove/change existing fields	Add required fields without defaults
Primary benefit	Producer can evolve freely	Consumer can deploy ahead of producer

Deployment Flexibility

Why Forward Compatibility Matters

Forward compatibility might seem less critical than backward compatibility—after all, how often do consumers deploy before producers? More often than you'd think:

Common forward compatibility scenarios:

When Forward Compatibility is Needed

•Rolling deployments — Consumers upgrade one pod/instance at a time while producers keep running old code. New consumer instances receive events from old producer instances.
•Blue-green deployments — New consumer version runs alongside old version. Traffic gradually shifts. Both versions receive events from any producer version.
•Consumer-first features — A consumer team needs a new field for analytics. They deploy code that uses the field if present. Producer adds the field later.
•Multi-producer environments — Multiple microservices produce the same event type. They upgrade at different times. Consumers must handle the oldest producer's format.
•Event replay/reprocessing — Historical events from years ago are replayed through current consumer code. Current consumers see very old schemas.
•Third-party producers — External systems produce events on their schedule. You can't control when they upgrade.

The cost of forward incompatibility:

Without forward compatibility, consumer deploys are blocked until all producers upgrade. This creates:

Deployment coupling — Can't ship consumer improvements until producers are ready
Coordination overhead — Must synchronize releases across teams
Rollback risk — Rolling back producers requires rolling back consumers too
Feature delays — New consumer features wait on producer timelines

The Hidden Coupling

Forward Compatible Schema Design

Forward compatibility requires consumers to handle missing fields gracefully. This has implications for how we design both schemas and consumer code.

Core principle: Every field a consumer accesses must either:

Be guaranteed present in all schema versions, OR
Have a safe default/fallback when absent

Forward Compatible Patterns

•Always define defaults for new fields
•Never add required fields to existing events
•Use optionals for all non-original fields
•Check field presence before access
•Provide fallback logic for missing data

Forward Breaking Patterns

•Adding required fields — Old events lack them
•Assuming field presence in new code
•Strict validation that rejects old events
•Required nested objects on existing events
•Non-nullable types for new fields

forward-compatible-schema.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
// FORWARD COMPATIBLE: All new fields optional with defaults
interface OrderCreatedV3 {
  // Original fields (from v1) — always present
  orderId: string;
  customerId: string;
  items: OrderItem[];
  totalAmount: number;
  
  // v2 additions — may be absent in v1 events
  currency?: string;            // Default: 'USD'
  shippingAddress?: Address;    // Default: null (use customer default)
  
  // v3 additions — may be absent in v1/v2 events
  loyaltyTier?: 'bronze' | 'silver' | 'gold';  // Default: 'bronze'
  estimatedDelivery?: string;   // Default: calculated from items
  promotionCodes?: string[];    // Default: []
}
 
// FORWARD INCOMPATIBLE: Adding required field
interface OrderCreatedV3Bad {
  orderId: string;
  customerId: string;
  items: OrderItem[];
  totalAmount: number;
  currency: string;  // BREAKING: Required field added in v3
                     // Old events (v1, v2) don't have this!
}
 
// JSON Schema version — note all new fields are NOT required
{
  "type": "object",
  "required": ["orderId", "customerId", "items", "totalAmount"],  // Original only
  "properties": {
    "orderId": { "type": "string" },
    "customerId": { "type": "string" },
    "items": { "type": "array" },
    "totalAmount": { "type": "number" },
    "currency": { "type": "string", "default": "USD" },
    "shippingAddress": { "$ref": "#/definitions/Address" },
    "loyaltyTier": { 
      "type": "string", 
      "enum": ["bronze", "silver", "gold"],
      "default": "bronze"
    },
    "estimatedDelivery": { "type": "string", "format": "date-time" },
    "promotionCodes": { "type": "array", "items": { "type": "string" }, "default": [] }
  }
}

The Required Fields Rule

Handling Missing Fields in Consumers

The real work of forward compatibility happens in consumer code. New consumers must gracefully handle the absence of fields they want to use.

Missing field strategies:

Use schema-defined or code-defined defaults when fields are absent.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
// TypeScript: Defaults during destructuring
function processOrder(event: OrderCreatedV3) {
  const {
    orderId,
    customerId,
    totalAmount,
    // Defaults for potentially missing fields
    currency = 'USD',
    loyaltyTier = 'bronze',
    promotionCodes = [],
  } = event;
  
  // Safe to use — defaults applied
  const formattedTotal = formatCurrency(totalAmount, currency);
  const discount = calculateLoyaltyDiscount(totalAmount, loyaltyTier);
}
 
// Java: Optional with orElse
public void processOrder(OrderCreated event) {
    String currency = Optional.ofNullable(event.getCurrency())
                              .orElse("USD");
    
    List<String> promoCodes = Optional.ofNullable(event.getPromotionCodes())
                                      .orElse(Collections.emptyList());
}
 
// Python: dict.get with default
def process_order(event: dict):
    currency = event.get('currency', 'USD')
    loyalty_tier = event.get('loyaltyTier', 'bronze')
    promo_codes = event.get('promotionCodes', [])

Monitor Missing Field Rates

Full Compatibility

Full compatibility (or bidirectional compatibility) combines backward and forward compatibility. It's the gold standard for event schemas in microservices.

Full compatibility guarantees:

New producer events work with old consumers (backward)
Old producer events work with new consumers (forward)
Any producer can work with any consumer (within MAJOR version)

Converting Mermaid diagram...

Achieving full compatibility:

Full compatibility is more restrictive than either direction alone. The rules are cumulative:

Rule	Backward	Forward	Full
Don't remove fields	✓		✓
Don't change field types	✓		✓
Don't add required fields		✓	✓
New fields must have defaults		✓	✓
Consumers must ignore unknown	✓		✓
Consumers must handle missing		✓	✓

full-compatibility-rules.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
// FULLY COMPATIBLE evolution: v1 → v2 → v3
 
// v1 (Original)
interface OrderCreatedV1 {
  orderId: string;      // Required - original
  customerId: string;   // Required - original  
  totalAmount: number;  // Required - original
}
 
// v2 (Fully compatible with v1)
interface OrderCreatedV2 {
  orderId: string;
  customerId: string;
  totalAmount: number;
  currency?: string;              // ADDED: Optional with default
  shippingAddress?: Address;      // ADDED: Optional, nullable
}
 
// v3 (Fully compatible with v1 AND v2)
interface OrderCreatedV3 {
  orderId: string;
  customerId: string;
  totalAmount: number;
  currency?: string;
  shippingAddress?: Address;
  loyaltyTier?: LoyaltyTier;      // ADDED: Enum, optional
  promotionCodes?: string[];      // ADDED: Array, defaults to []
}
 
// Compatibility matrix:
// - v1 consumer reads v3 event: ✓ (ignores new fields)
// - v3 consumer reads v1 event: ✓ (uses defaults for missing)
// - v2 consumer reads v3 event: ✓ (ignores v3 additions)
// - v3 consumer reads v2 event: ✓ (uses defaults for v3 fields)
 
// Schema registry configuration
const schemaConfig = {
  subject: 'order-created',
  compatibilityMode: 'FULL',  // Enforce bidirectional compatibility
  // Alternatives:
  // 'BACKWARD' - only backward compatible
  // 'FORWARD' - only forward compatible
  // 'NONE' - no compatibility checks
};

When to Use Full Compatibility

Transitive Compatibility

Transitive compatibility extends compatibility guarantees across all historical versions, not just adjacent ones.

Non-transitive vs. transitive:

Non-transitive: Each version compatible with immediate predecessor/successor only
Transitive: Each version compatible with ALL previous/subsequent versions

Why transitive matters:

transitive-compatibility.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// NON-TRANSITIVE COMPATIBILITY (DANGEROUS)
// Each version compatible only with adjacent version
 
// v1: { orderId, totalAmount }
// v2: { orderId, totalAmount, currency? }  // Compatible with v1
// v3: { orderId, totalAmount, currency?, tax? }  // Compatible with v2
 
// But what if we skip v2 during upgrade?
// v1 producer → v3 consumer
// v3 consumer might ASSUME currency is commonly present
// because v2 established it. But v1 never had it!
 
// TRANSITIVE COMPATIBILITY (SAFE)
// Each version compatible with ALL historical versions
 
interface CompatibilityCheck {
  subject: string;
  version: number;
  checkAgainst: 'LATEST' | 'ALL';  // ALL = transitive
}
 
// Schema registry with transitive enforcement
// Before accepting v4, checks compatibility with v3, v2, AND v1
async function registerSchema(newSchema: Schema): Promise<void> {
  const allVersions = await registry.getAllVersions(newSchema.subject);
  
  for (const oldVersion of allVersions) {
    const oldSchema = await registry.getSchema(newSchema.subject, oldVersion);
    
    // Must be compatible with EVERY historical version
    if (!isCompatible(newSchema, oldSchema)) {
      throw new IncompatibleSchemaError(
        `v${newSchema.version} incompatible with v${oldVersion}`
      );
    }
  }
  
  await registry.register(newSchema);
}

Schema Registry Compatibility Modes
Mode	Checks Against	Use Case
BACKWARD	Latest version only	Consumers always current; producers may lag
BACKWARD_TRANSITIVE	All previous versions	Long-running consumers; event replay
FORWARD	Latest version only	Producers always current; consumers may lag
FORWARD_TRANSITIVE	All previous versions	Multiple producer versions in parallel
FULL	Latest version only	Independent deployment (common case)
FULL_TRANSITIVE	All previous versions	Maximum flexibility; event sourcing

Event Sourcing Requires Transitive

Forward Compatibility Challenges

Forward compatibility presents unique challenges that don't exist with backward compatibility:

Challenge 1: Feature requirement mismatch

A new consumer feature requires data that old events don't have. You can't magically make old producers include new fields.

feature-requirement-mismatch.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
// New feature: Real-time fraud scoring
// REQUIRES: customerDeviceFingerprint, paymentMethodToken
 
async function scoreFraudRisk(event: OrderCreatedV3): Promise<FraudScore> {
  // New fields required for fraud scoring
  if (!event.deviceFingerprint || !event.paymentMethodToken) {
    // Old events lack this data — can't score accurately!
    return {
      score: 0.5,  // Neutral score
      confidence: 'LOW',
      reason: 'Insufficient data for accurate scoring',
    };
  }
  
  // Full scoring with v3 data
  return await fraudEngine.score({
    device: event.deviceFingerprint,
    payment: event.paymentMethodToken,
    customer: event.customerId,
    amount: event.totalAmount,
  });
}
 
// Solution: Graceful degradation with backfill
async function scoreFraudRiskWithBackfill(
  event: OrderCreatedV3
): Promise<FraudScore> {
  // Attempt to backfill missing data from other sources
  const device = event.deviceFingerprint ?? 
                 await sessionService.getLastDevice(event.customerId);
  
  const payment = event.paymentMethodToken ??
                  await paymentService.getDefaultMethod(event.customerId);
  
  if (!device || !payment) {
    // Still can't score — this is expected for old events
    return { score: 0.5, confidence: 'LOW', reason: 'Data unavailable' };
  }
  
  // Score with backfilled data (lower confidence)
  const result = await fraudEngine.score({ device, payment, ... });
  return { ...result, confidence: 'MEDIUM' };  // Lower than native data
}

Challenge 2: Semantic evolution

The meaning of fields changes over time. An old 'status' value might mean something different than a new 'status' value.

semantic-evolution.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// v1: status = 'PENDING' | 'ACTIVE' | 'CLOSED'
// v3: status = 'PENDING' | 'ACTIVE' | 'COMPLETED' | 'CANCELLED' | 'REFUNDED'
//     ('CLOSED' was split into 'COMPLETED', 'CANCELLED', 'REFUNDED')
 
function processAccountStatus(event: AccountUpdatedV3): void {
  switch (event.status) {
    case 'PENDING':
    case 'ACTIVE':
      // Same meaning across versions
      handleActiveFlow(event);
      break;
      
    case 'COMPLETED':
    case 'CANCELLED':
    case 'REFUNDED':
      // New granular statuses
      handleClosedFlow(event, event.status);
      break;
      
    case 'CLOSED':
      // LEGACY: v1 events use 'CLOSED'
      // We don't know if it was completed, cancelled, or refunded
      // Best effort: Treat as completed (most common case)
      console.warn('Legacy CLOSED status', { accountId: event.accountId });
      handleClosedFlow(event, 'COMPLETED');
      break;
      
    default:
      // Unknown status — forward compatibility
      handleUnknownStatus(event);
  }
}

Semantic Changes Need Documentation

Testing Forward Compatibility

Testing forward compatibility requires samples from all historical schema versions. New consumer code must process events from every previous version successfully.

Testing approach:

forward-compatibility-tests.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
// Test fixture: Historical event samples
const historicalEventSamples = {
  'v1.0.0': [
    { orderId: 'v1-order-1', customerId: 'c1', totalAmount: 100 },
    { orderId: 'v1-order-2', customerId: 'c2', totalAmount: 250 },
  ],
  'v2.0.0': [
    { orderId: 'v2-order-1', customerId: 'c1', totalAmount: 100, currency: 'USD' },
    { orderId: 'v2-order-2', customerId: 'c2', totalAmount: 250, currency: 'EUR', 
      shippingAddress: { street: '123 Main', city: 'Berlin' } },
  ],
  'v3.0.0': [
    { orderId: 'v3-order-1', customerId: 'c1', totalAmount: 100, currency: 'USD',
      loyaltyTier: 'gold', promotionCodes: ['SAVE10'] },
  ],
};
 
describe('Forward Compatibility: New Consumer with Old Events', () => {
  const consumer = new OrderConsumerV3();  // Current consumer
  
  Object.entries(historicalEventSamples).forEach(([version, events]) => {
    describe(`Processing ${version} events`, () => {
      events.forEach((event, index) => {
        it(`handles ${version} event #${index}`, async () => {
          // Should not throw
          const result = await consumer.process(event);
          
          // Core processing works
          expect(result.orderId).toBe(event.orderId);
          expect(result.processed).toBe(true);
          
          // Defaults applied for missing fields
          expect(result.currency).toBe(event.currency ?? 'USD');
          expect(result.loyaltyTier).toBe(event.loyaltyTier ?? 'bronze');
        });
      });
    });
  });
  
  // Test edge cases
  describe('Missing field edge cases', () => {
    it('handles v1 event with no optional fields', async () => {
      const v1Event = { orderId: 'test', customerId: 'c', totalAmount: 50 };
      const result = await consumer.process(v1Event);
      
      // All defaults applied
      expect(result.currency).toBe('USD');
      expect(result.shippingAddress).toBeNull();
      expect(result.loyaltyTier).toBe('bronze');
      expect(result.promotionCodes).toEqual([]);
    });
    
    it('handles partial v2 event (some optionals present)', async () => {
      const partialV2 = { 
        orderId: 'test', customerId: 'c', totalAmount: 50,
        currency: 'GBP'  // Has currency, no shippingAddress
      };
      const result = await consumer.process(partialV2);
      
      // Mixed: explicit value + defaults
      expect(result.currency).toBe('GBP');
      expect(result.shippingAddress).toBeNull();
    });
  });
});
 
// Integration test: Replay historical events
describe('Event Replay Forward Compatibility', () => {
  it('replays entire event history without errors', async () => {
    const eventStore = new EventStore();
    const consumer = new OrderConsumerV3();
    
    // Fetch all historical events
    const allEvents = await eventStore.getAllEvents('order-created');
    
    let errorCount = 0;
    for (const event of allEvents) {
      try {
        await consumer.process(event);
      } catch (error) {
        errorCount++;
        console.error('Replay failed:', event.orderId, error);
      }
    }
    
    expect(errorCount).toBe(0);
  });
});

Maintain Historical Samples

Migration Strategies

When evolving schemas, the order of producer and consumer deployment matters. Different strategies have different compatibility requirements:

Strategy 1: Consumer-first deployment

Consumer-First Deployment

•Step 1: Deploy new consumers that handle both old AND new event formats
•Step 2: Verify consumers correctly process old events (forward compatibility test)
•Step 3: Deploy new producers that emit new format
•Step 4: Verify consumers correctly process new events
•Rollback: If producers fail, consumers continue processing old format

Strategy 2: Producer-first deployment

Producer-First Deployment

•Step 1: Deploy new producers (new format is backward compatible)
•Step 2: Verify old consumers correctly process new events (backward compatibility)
•Step 3: Deploy new consumers that leverage new fields
•Step 4: Verify new consumers work with new events
•Rollback: If consumers fail, roll back consumers; producers keep running

deployment-coordination.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
// Deployment coordinator for schema migration
 
interface DeploymentPlan {
  schemaVersion: string;
  strategy: 'CONSUMER_FIRST' | 'PRODUCER_FIRST';
  steps: DeploymentStep[];
}
 
const consumerFirstPlan: DeploymentPlan = {
  schemaVersion: '3.0.0',
  strategy: 'CONSUMER_FIRST',
  steps: [
    {
      phase: 1,
      action: 'DEPLOY_CONSUMERS',
      services: ['analytics', 'shipping', 'notifications'],
      validation: 'Consumers process v2 events without errors',
      rollbackTrigger: 'Error rate > 0.1%',
    },
    {
      phase: 2,
      action: 'WAIT',
      duration: '24h',
      validation: 'Consumers stable in production',
    },
    {
      phase: 3,
      action: 'DEPLOY_PRODUCERS',
      services: ['order-service'],
      validation: 'Producers emit v3 events, consumers process successfully',
      rollbackTrigger: 'Consumer error rate > 0.1%',
    },
    {
      phase: 4,
      action: 'CLEANUP',
      tasks: ['Remove v2 support from producers', 'Update documentation'],
    },
  ],
};
 
// Feature flags for gradual rollout
const schemaFeatureFlags = {
  'emit-v3-fields': {
    enabled: true,
    percentage: 10,  // Start with 10% of events
    monitors: ['consumer-error-rate', 'processing-latency'],
    autoRollback: {
      errorRateThreshold: 0.5,
      latencyP99Threshold: 500,
    },
  },
};

Consumer-First is Usually Safer

Summary: Forward Compatibility

Forward compatibility ensures new consumers can process old events. Let's consolidate the key takeaways:

Key Takeaways

•Forward compatibility enables consumer-ahead deployment — New consumers work before producers upgrade.
•Never add required fields to existing events — Old events won't have them; new consumers will fail.
•All new fields must be optional with defaults — Consumers must handle absence gracefully.
•Handle missing fields programmatically — Use defaults, computed fallbacks, or conditional logic.
•Full compatibility combines backward AND forward — Maximum deployment flexibility.
•Transitive compatibility spans all versions — Essential for event replay and event sourcing.
•Test with historical event samples — Verify new consumer code processes all old schema versions.
•Consumer-first deployment is often safer — Validates forward compatibility before producing new data.

What's next:

Page Complete

3 / 5