Backup and Disaster Recovery

In the realm of distributed systems engineering, few responsibilities carry greater weight than data protection. A single misconfigured delete operation, a ransomware attack, or a catastrophic hardware failure can erase years of accumulated business data in moments. The financial and reputational costs can be existential—companies have literally ceased to exist because they couldn't recover their data.

Backup strategies form the first line of defense in your data protection arsenal. But not all backups are created equal. The choice between full, incremental, and differential backup strategies involves deep trade-offs affecting recovery time, storage costs, backup windows, and operational complexity. Understanding these trade-offs is essential for any engineer designing systems that store data of consequence.

Before diving into specific strategies, we must establish a rigorous understanding of what backups actually accomplish and the constraints that govern their design.

The Purpose of Backups:

Backups serve multiple distinct purposes that often require different technical approaches:

1. Point-in-Time Recovery (PITR): Restoring data to a specific moment, often to undo human errors or logical corruption
2. Disaster Recovery (DR): Rebuilding entire systems after catastrophic failures
3. Compliance and Archival: Meeting regulatory requirements for data retention
4. Migration and Testing: Creating copies for system moves or safe testing environments

Each purpose imposes different requirements on backup frequency, retention period, recovery speed, and verification processes. A backup strategy optimized for compliance archival (rarely accessed, long retention) differs substantially from one optimized for rapid operational recovery.

A full backup captures the entire dataset at a point in time, creating a complete, self-contained copy that can restore the system independently without requiring any other backup sets.

Mechanics of Full Backups:

The full backup process involves:

1. Snapshot Initiation: Creating a consistent view of the data, often requiring application quiescing, database checkpoints, or storage-level snapshots
2. Data Transfer: Copying all data blocks to the backup destination, regardless of whether they've changed since the last backup
3. Metadata Recording: Storing backup metadata including timestamps, checksums, schema versions, and restoration procedures
4. Verification: Confirming backup completeness and integrity through checksum validation

When Full Backups Excel:

Full backups are the optimal choice when:

• Dataset size is small relative to backup window and storage budget: If you can complete a full backup within available time and afford the storage, the simplicity advantage is significant
• Change rate is high: If 50%+ of data changes between backups, incremental strategies lose their efficiency advantage
• Recovery speed is critical: When RTO requirements are aggressive, eliminating chain reconstruction overhead is valuable
• Operational simplicity is prioritized: Full backups require less expertise to manage and restore correctly
• Compliance requires independent copies: Some regulations require self-contained backup copies without dependencies

An incremental backup captures only the data that has changed since the last backup of any type—whether that was a full backup or another incremental backup. This dramatically reduces storage requirements and backup duration at the cost of increased recovery complexity.

Mechanics of Incremental Backups:

Change Detection Methods:

The efficiency of incremental backups depends critically on how changes are detected:

1. Archive Bit / File Metadata: Track file modification timestamps or archive attributes. Fast but coarse—any file modification triggers full file backup even if only one byte changed.

2. Block-Level Change Tracking (CBT): Monitor changes at the storage block level. Highly efficient for virtual machines and databases but requires storage system or hypervisor support.

3. Database Transaction Logs: For databases, backup transaction logs since the last backup. Provides exact change capture but requires database-specific integration.

4. Content-Based Chunking: Use content-defined chunking algorithms (like Rabin fingerprinting) to identify changed data segments. Used by deduplication systems for sub-file level change detection.

5. Filesystem Journaling: Read filesystem journal entries to identify changed files without scanning the entire filesystem. Efficient for large filesystems with sparse changes.

A differential backup captures all data that has changed since the last full backup, regardless of any intervening differential or incremental backups. This creates a hybrid approach with better recovery characteristics than incrementals while still reducing storage compared to full backups.

The Differential Difference:

The critical distinction is the reference point for change detection:

• Incremental: Changes since last backup of any type
• Differential: Changes since last full backup only

This seemingly small difference has profound implications for both storage efficiency and recovery complexity.

The Growth Pattern:

Unlike incremental backups where each backup is roughly the same size (assuming consistent change rates), differential backups exhibit cumulative growth:

By Saturday, the differential is 23 GB while incrementals total 23 GB across all individual backups. The storage totals are similar, but the distribution differs—and critically, the recovery process differs dramatically.

Selecting the right backup strategy requires evaluating your specific constraints and priorities. Let's analyze a concrete scenario to illustrate the decision process.

Scenario: You're designing backup architecture for a 50 TB database supporting an e-commerce platform. Daily change rate averages 3% (1.5 TB), backup window is 6 hours, and you need 30-day retention.

Analysis:

1. Full Only: Physically impossible. 50 TB in 6 hours requires 2.3 GB/s sustained—beyond typical enterprise capabilities.

2. Incremental: Most storage-efficient (95 TB), but recovery is problematic. Restoring to day 30 requires applying 29 incrementals sequentially, taking ~20 hours and requiring all 29 chain links to be intact.

3. Differential: Balanced approach. Recovery to any day takes ~9 hours (full + one differential). Storage is higher than incremental but substantially less than full-only.

The Hybrid Reality:

In practice, most enterprise systems use hybrid strategies:

Moving from strategy to implementation requires addressing several critical technical challenges:

1. Consistency and Application Awareness:

File-level backups may capture inconsistent state if applications are actively writing. Database backups require special handling:

2. Backup Catalog and Metadata Management:

Backup systems must maintain detailed catalogs tracking:

• What was backed up (files, blocks, database objects)
• When it was backed up (timestamps, sequence numbers)
• Where it's stored (media location, retention tier)
• Chain dependencies (parent backups for incrementals/differentials)
• Verification status (checksums, test restore results)

Corruption or loss of the backup catalog can render backup data unrecoverable even if the data itself is intact.

3. Parallelization and Performance:

We've conducted a comprehensive analysis of the three fundamental backup strategies. Let's consolidate the key insights:

What's Next:

Backup strategies answer 'how do we create copies of data?' The next page addresses the equally critical question: 'how much data can we afford to lose, and how quickly must we recover?' We'll explore Recovery Point Objective (RPO) and Recovery Time Objective (RTO)—the metrics that drive backup policy decisions.