Throughout this module, we've focused on getting data from applications to persistent storage correctly—write-through for durability, write-back for performance, ordered writes for consistency. But there's a deeper question we haven't addressed: How do we know the data on disk is still correct after it's written?

Data can become corrupted silently:

• Bit rot: Random bit flips in storage media over time
• Firmware bugs: Controller writes to wrong location
• Cosmic rays: High-energy particles flip bits in DRAM or storage
• Phantom writes: Write acknowledged but not performed
• Misdirected writes: Write lands at wrong address
• Torn writes: Partial writes due to power loss
• Read errors: Correct on disk, corrupt during read

The terrifying reality: a study by CERN found that 1 in 10^7 to 10^8 bytes in large storage systems contains silent errors that go undetected by disk-level ECC. At petabyte scale, this means thousands of corrupt bytes per system.

This page explores how systems detect, prevent, and recover from data corruption—the strategies that enable us to trust long-term data storage.

To protect data integrity, we must first understand the failure modes we're protecting against. Each mode has different characteristics and requires different countermeasures.

Corruption Categories

The Silent Corruption Problem

Most of these failures are silent—the system doesn't know corruption occurred until corrupted data is read. By then, it may be too late:

Timeline of silent corruption:

Day 1:   Write block B containing important data
         Disk returns success
         
Day 5:   Bit flip occurs in block B on media
         No read, no detection
         
Day 30:  Make backup
         Backup contains corrupted block B!
         
Day 60:  Try to read block B
         ERROR or wrong data returned
         Backups are also corrupt
         
         Data is lost

The key insight: detection must happen at read time, not just write time. And ideally, detection should happen during routine scrubbing, before the data is actually needed.

Corruption Rates in Practice

Research from NetApp, Google, CERN, and others provides real-world corruption statistics:

SATA drives: 1 in 10^14 to 10^15 bits URE (Unrecoverable Read Error)
             ~1-10 URE per 10TB read over device lifetime
             
Enterprise SAS: 1 in 10^15 to 10^16 bits
             ~0.1-1 URE per 10TB

Silent corruption (past ECC): 1 in 10^7 to 10^8 bytes
             CERN study: 128 undetected errors per PB/year

DRAM bit flips: 1-5% of DIMMs per year have errors
             Google study: much higher than expected

These numbers may seem small, but at scale they're significant. A 100 PB storage system might expect 10,000+ silent corruption events per year.

Checksums are the primary mechanism for detecting data corruption. Understanding their properties enables choosing the right checksum for each use case.

What a Checksum Provides

A checksum is a fixed-size value computed from data that changes (with high probability) if the data changes. It's a fingerprint of the data.

Checksum(data) → fixed-size value

Properties:
  - Deterministic: Same data always produces same checksum
  - Fixed output size: Regardless of input size
  - Sensitivity: Any change to input changes output
  - Distribution: Outputs uniformly distributed (for good checksums)
  
What it CAN detect:
  - Random bit flips
  - Truncation
  - Insertions/deletions
  - Block transposition (with block-addressed checksums)
  
What it CANNOT prevent:
  - Intentional tampering (need cryptographic hash)
  - Errors that happen to produce matching checksum (rare but possible)

Common Checksum Algorithms

CRC-32C: The Modern Standard

CRC-32C (Castagnoli polynomial) is widely used in storage because:

1. Hardware acceleration: Intel SSE4.2 provides CRC-32C instruction
2. Better error detection: Catches more error patterns than standard CRC-32
3. Adequate for bit-flip detection: 32 bits detects most random errors

#include <x86intrin.h>

// Hardware-accelerated CRC-32C using SSE4.2
uint32_t crc32c_hardware(const void *data, size_t len) {
    const uint8_t *p = data;
    uint32_t crc = 0xFFFFFFFF;
    
    // Process 8 bytes at a time
    while (len >= 8) {
        crc = _mm_crc32_u64(crc, *(const uint64_t *)p);
        p += 8;
        len -= 8;
    }
    
    // Process remaining bytes
    while (len--) {
        crc = _mm_crc32_u8(crc, *p++);
    }
    
    return crc ^ 0xFFFFFFFF;
}

// Performance: 20-40 GB/s on modern CPUs
// Compare to disk speed: 3-7 GB/s for NVMe
// Checksum is NOT the bottleneck

Collision Probability

For a 32-bit checksum, the probability of two random inputs having the same checksum is 2^-32 ≈ 2.3 × 10^-10. This seems small, but:

At 1 billion (10^9) blocks:
  Expected collisions ≈ 10^9 × 10^9 × 2.3 × 10^-10 / 2 ≈ 115

For larger systems, 64-bit checksums provide:
  Collision probability: 2^-64 ≈ 5.4 × 10^-20
  
  At 1 trillion (10^12) blocks:
  Expected collisions ≈ 10^12 × 10^12 × 5.4 × 10^-20 / 2 ≈ 0.027

This is why ZFS uses 256-bit checksums—collision probability is negligible even at extreme scale.

Device-level checksums (like sector ECC) protect only one segment of the data path. True data integrity requires end-to-end protection: verifying that what the application reads is exactly what it wrote.

The Data Path Vulnerability Points

Application Buffer (DRAM)
        │
        │ ← Bus errors, DMA errors, DRAM bit flips
        ▼
Kernel Page Cache (DRAM)
        │
        │ ← Same DRAM risks, plus kernel memory corruption
        ▼
Filesystem / Block Layer
        │
        │ ← Software bugs, misdirected I/O
        ▼
Device Driver
        │
        │ ← Driver bugs, incorrect command formatting
        ▼
HBA/Controller (often has DRAM)
        │
        │ ← Controller firmware bugs, controller DRAM errors
        ▼
Network (if SAN/iSCSI/NFS)
        │
        │ ← Packet corruption, routing errors
        ▼
Device Controller (has DRAM, cache)
        │
        │ ← Firmware bugs, cache corruption
        ▼
Persistent Media
        │
        │ ← Bit rot, media degradation
        ▼
[Read path goes back up with same risks]

Each layer might have checksums, but layer-specific checksums don't catch cross-layer errors. A misdirected write (correct checksum, wrong location) isn't detected by media ECC.

End-to-End Verification Architecture

True end-to-end integrity means:

1. Compute checksum at write time (in application or filesystem)
2. Store checksum with or near data
3. Verify checksum at read time (before returning to application)
4. Include metadata in checksum (physical address, version, etc.)

Application writes block B at logical address L:

1. Compute: checksum = CRC32C(L || B)
   Including L in checksum detects misdirected writes
   
2. Write: [L][B][checksum] → storage

3. Read at address L:
   - Fetch [B][checksum]
   - Compute: expected = CRC32C(L || B)
   - Compare: if (checksum != expected) → CORRUPTION!
   
4. If corruption detected:
   - Attempt read from replica/parity
   - Log error for analysis
   - Mark block as suspect

ZFS: The Gold Standard for End-to-End Integrity

ZFS pioneered comprehensive end-to-end integrity in general-purpose filesystems:

ZFS Integrity Features:

1. Block-level checksums
   - 256-bit checksum for every block
   - Checksum stored in PARENT block's pointer
   - Cannot have checksum-storage mismatch
   
2. Merkle tree verification
   - Root block checksum verifies children
   - Children verify grandchildren
   - Down to leaf data blocks
   - Corruption anywhere is detected at root
   
3. Self-healing with redundancy
   - On checksum mismatch, read from mirror/parity
   - If redundant copy is good, repair bad copy
   - Silent self-healing, no data loss
   
4. Scrubbing
   - Background process reads all blocks
   - Verifies checksums
   - Repairs from redundancy if possible
   - Reports errors for blocks without redundancy

The parent pointer design is crucial: if the checksum were stored with the data, a misdirected write could overwrite both data and checksum. By storing the checksum in the parent, a misdirected write is immediately detected when the parent's pointer checksum fails.

Different filesystems provide different levels of integrity protection. Understanding these differences is crucial for selecting the right filesystem for data-critical applications.

Feature Comparison

*ext4 metadata checksums available with metadata_csum feature (default in newer versions)

ext4 Integrity Strategy

ext4 provides minimal built-in integrity, relying on external tools:

ext4 Approach:
  - Journal for metadata consistency (not integrity)
  - Optional metadata checksums (mkfs.ext4 -O metadata_csum)
  - No data checksums
  - Relies on RAID for data protection
  - e2fsck for offline repair
  
When to use ext4:
  - Compatibility requirements
  - Hardware RAID providing protection
  - Applications do their own checksumming
  - Performance over safety trade-off

Btrfs Integrity Strategy

Btrfs provides comprehensive checksumming with self-healing:

Btrfs Approach:
  - CRC32C checksums for all data and metadata
  - Checksums stored in separate checksum tree
  - COW ensures atomic updates
  - Self-healing with RAID1/5/6
  - Online scrubbing
  
Commands:
  btrfs scrub start /mount/point    # Start scrub
  btrfs scrub status /mount/point   # Check progress
  btrfs device stats /mount/point   # View error counters
  
  # Example output
  [/dev/sda].corruption_errs: 2
  [/dev/sda].read_errs: 0
  [/dev/sda].write_errs: 0
  [/dev/sdb].corruption_errs: 0  # Mirror copy was good

ZFS Integrity Architecture

ZFS has the most comprehensive integrity system:

Even with checksumming filesystems, applications may need additional integrity measures for:

• Portability across filesystems
• Multi-system replication
• Long-term archive verification
• Regulatory compliance

Pattern 1: Record-Level Checksums

import hashlib
import struct

class ChecksummedRecord:
    HEADER_SIZE = 36  # 4 bytes length + 32 bytes SHA-256
    
    def __init__(self, data):
        self.data = data
        self.checksum = hashlib.sha256(data).digest()
    
    def serialize(self):
        length = len(self.data)
        return struct.pack(">I", length) + self.checksum + self.data
    
    @classmethod
    def deserialize(cls, raw):
        length = struct.unpack(">I", raw[:4])[0]
        stored_checksum = raw[4:36]
        data = raw[36:36+length]
        
        computed_checksum = hashlib.sha256(data).digest()
        
        if stored_checksum != computed_checksum:
            raise IntegrityError(
                f"Checksum mismatch: expected {stored_checksum.hex()}, "
                f"got {computed_checksum.hex()}"
            )
        
        record = cls.__new__(cls)
        record.data = data
        record.checksum = stored_checksum
        return record

Pattern 2: Content-Addressed Storage

Deduplicate and verify in one step:

Pattern 3: Merkle Trees for Efficient Verification

For large datasets, verify incrementally:

Full verification: Hash 1TB = ~30 seconds
                  Must read entire dataset

Merkle verification:
  Level 0: 256MB chunks, hash each → 4096 hashes
  Level 1: Groups of 16-chunk hashes → 256 hashes  
  Level 2: Groups of 16 → 16 hashes
  Level 3: Root hash → 1 hash
  
  Total: ~32KB of hashes for 1TB data
  
To verify ONE chunk changed:
  - Hash the chunk (32 bytes to update)
  - Recompute parent path to root (4 levels, ~4ms)
  - Compare root hash
  
  Only need to read 256MB + parent path, not 1TB

This is exactly how Git verifies repository integrity—any change anywhere is detected by checking the root commit hash.

Detection is only half the battle. When corruption is found, systems need recovery strategies.

Recovery Hierarchy

Order of recovery attempts (most to least desirable):

1. Self-heal from redundancy
   - Read from mirror/parity
   - Repair corrupted copy
   - User never knows
   
2. Replay from transaction log
   - WAL contains original data
   - Replay recreates correct state
   - Applies to crash recovery too
   
3. Restore from backup
   - Most recent backup
   - Check backup isn't also corrupt!
   - May lose recent changes
   
4. Reconstruct from replicas
   - Distributed system: fetch from other nodes
   - Eventually consistent: latest wins
   
5. Partial recovery
   - Salvage what's readable
   - Mark bad blocks
   - User decides how to proceed
   
6. Accept data loss
   - Log the loss
   - Alert operations
   - Move on

Self-Healing Implementation

Backup Verification

Backups themselves can be corrupted. Always verify:

# PostgreSQL: Verify backup integrity
pg_verifybackup /path/to/backup

# tar: Verify archive checksums (if available)
tar --verify -f backup.tar

# restic: Verify backup integrity
restic -r /backup/repo check --read-data

# ZFS: Send with checksum verification
zfs send -c pool/dataset | zfs receive -F backup_pool/dataset
# -c sends compressed, preserves checksums

Production systems need continuous monitoring of data integrity metrics. Problems caught early are much easier to resolve.

Key Metrics to Monitor

Alerting Thresholds

Prometheus Metrics Example

# Prometheus metrics for integrity monitoring
- name: storage_checksum_errors_total
  help: Total number of checksum verification failures
  type: counter
  labels:
    - device
    - pool

- name: storage_silent_repairs_total
  help: Corruptions repaired silently from redundancy
  type: counter
  labels:
    - device
    - pool

- name: storage_last_scrub_timestamp
  help: Unix timestamp of last completed scrub
  type: gauge
  labels:
    - pool

- name: storage_last_scrub_errors
  help: Errors found in most recent scrub
  type: gauge
  labels:
    - pool
    - error_type

Data integrity is the foundation upon which all other storage guarantees rest. Without confidence that stored data remains correct, durability and performance optimizations are meaningless. This page has covered the complete spectrum of integrity concerns, from corruption taxonomy to production monitoring.

Module Complete: Write Strategies

We've now comprehensively covered the spectrum of write strategies:

1. Write-Through: Immediate durability, maximum safety, lowest performance
2. Write-Back: Deferred durability, maximum performance, risk of data loss
3. Delayed Write: Application-controlled sync points, balanced trade-off
4. Ordered Writes: Controlled write ordering for crash-consistent structures
5. Data Integrity: End-to-end verification ensuring stored data remains correct

Together, these strategies form the toolkit for building storage systems that meet any combination of performance, durability, and reliability requirements. The art of systems engineering is choosing the right combination for each use case—and now you have the deep understanding needed to make those choices correctly.