Change Data Capture (CDC)

Imagine you're running an e-commerce platform. Your inventory database holds the source of truth for product availability. But this data needs to flow to multiple places: your Elasticsearch index for search, your Redis cache for fast lookups, your analytics warehouse for reporting, and your recommendation engine for personalization.

How do you keep all these systems synchronized?

The naive approach—having your application write to every system—creates a tightly coupled architecture that's fragile, slow, and prone to inconsistencies. What if the cache write fails after the database write succeeds? What about systems that were added after the original code was written?

This fundamental challenge—propagating data changes reliably across distributed systems—is precisely what Change Data Capture solves.

Change Data Capture (CDC) is a set of software design patterns and technologies used to identify and capture changes made to data in a database, and then deliver those changes in real-time to downstream systems.

At its core, CDC answers a deceptively simple question:

What data changed, when did it change, and what did it change from and to?

This seemingly straightforward question becomes remarkably complex in distributed systems. Consider a simple UPDATE statement:

UPDATE orders SET status = 'shipped' WHERE order_id = 12345;

For a single database, this operation is atomic and complete. But in a distributed architecture, this change might need to:

• Update a search index so customers can filter by shipped orders
• Invalidate a cache entry to prevent stale data
• Trigger a notification to the customer
• Update a real-time dashboard for operations
• Append to an analytics pipeline for business intelligence
• Inform a recommendation engine about purchase completion

CDC captures this change once, at the source, and propagates it reliably to all interested consumers.

Formal Definition:

Change Data Capture is the process of:

1. Detecting changes (inserts, updates, deletes) to source data
2. Capturing the details of those changes, including the before-state, after-state, and metadata
3. Delivering change events to downstream consumers in a reliable, ordered, and often real-time manner
4. Tracking progress to ensure exactly-once or at-least-once delivery semantics

The sophistication of CDC lies not in any single step, but in performing all of these reliably, at scale, with minimal impact on the source system.

To appreciate why CDC has become crucial, we must understand the evolution of data integration:

The ETL Era (1990s-2000s)

Traditionally, organizations moved data between systems using Extract-Transform-Load (ETL) processes. These ran nightly or hourly, extracting full datasets or deltas based on timestamps, transforming them, and loading them into data warehouses.

This approach worked when:

• Business processes could tolerate hours or days of data latency
• Data volumes were manageable for full table scans
• Systems were relatively isolated

The Microservices Revolution (2010s)

As organizations decomposed monoliths into microservices, each with its own database, the challenge of data synchronization exploded. ETL couldn't keep up—services needed real-time awareness of changes in other services.

The Real-Time Expectation (2020s)

Today's users expect real-time experiences. A customer updating their shipping address expects search, recommendations, and customer service to reflect that change immediately—not in tomorrow's batch job.

Why CDC Won:

CDC emerged as the superior pattern because it uniquely combines:

• Real-time latency: Changes propagate in seconds or sub-seconds
• Low coupling: Source systems are unaware of consumers
• High reliability: Leverages the database's own durability guarantees
• Completeness: Captures every change, even those from direct SQL or other applications
• Minimal overhead: Reads logs rather than querying tables repeatedly

At a conceptual level, CDC operates as a bridge between the transactional world and the event-driven world. Here's how the data flows:

The Data Flow Explained:

1. Application writes to database: Your application performs normal CRUD operations—inserts, updates, deletes.

2. Database writes to transaction log: Before committing any change, the database records it in its transaction log (also called write-ahead log, redo log, or binlog depending on the database). This is fundamental to how databases ensure durability.

3. CDC connector reads the log: A CDC system connects to the database and reads the transaction log in order, exactly as the database would during recovery.

4. Changes become events: Each change is transformed into a structured event containing:

   • Operation type: INSERT, UPDATE, or DELETE
   • Before state: What the row looked like before (for updates/deletes)
   • After state: What the row looks like after (for inserts/updates)
   • Metadata: Timestamp, transaction ID, sequence number, table name, etc.

5. Events flow to consumers: These change events are published to a message broker or stream, where any number of consumers can process them independently.

Understanding CDC's advantages requires comparing it to traditional data synchronization approaches. Each alternative has significant drawbacks that CDC addresses:

Detailed Comparison with All Alternatives:

Production-grade CDC systems must exhibit several critical properties. Understanding these properties helps you evaluate CDC solutions and design robust pipelines:

Delivery Semantics Deep Dive:

Understanding delivery guarantees is crucial for designing correct consumers:

At-least-once is the practical standard for most CDC systems. Exactly-once requires the entire pipeline (source, CDC, broker, consumer) to support transactional semantics, which adds complexity and latency.

Designing for at-least-once:

// Idempotent consumer example
async function handleOrderUpdate(event: CDCEvent) {
    // Use the event's transaction ID or offset as idempotency key
    const dedupeKey = `${event.source.txId}:${event.source.offset}`;
    
    // Check if already processed
    const processed = await cache.get(dedupeKey);
    if (processed) return; // Skip duplicate
    
    // Process the change
    await updateSearchIndex(event.after);
    
    // Mark as processed (with TTL to avoid infinite growth)
    await cache.set(dedupeKey, true, { ttl: '7d' });
}

CDC has become a foundational building block in several dominant architectural patterns. Understanding where CDC fits helps you recognize opportunities to apply it:

CDC is powerful but not universal. Understanding when it shines—and when to use alternatives—is essential for good architecture decisions.

We've established the foundation for understanding Change Data Capture. Let's consolidate the key insights:

What's Next:

Now that you understand what CDC is and why it matters, we'll dive into how it actually works at the database level. The next page explores log-based CDC in detail—how databases write their transaction logs, how CDC systems read them, and the technical mechanics that make real-time change capture possible.