Recovery Requirements

Understanding failure types is only the first step. The real question is: What must a recovery system deliver? When the database restarts after a crash, or when a transaction is rolled back after an error, what guarantees should users expect?

Recovery requirements define the contract between the database system and its users. They specify what 'correct recovery' means, what resources are needed to achieve it, and what trade-offs exist between recovery capabilities and system performance.

This page examines recovery requirements comprehensively—the fundamental guarantees, the ACID connection, measurable objectives like RPO and RTO, and the architectural requirements that enable recovery. Understanding these requirements is essential for both designing recovery systems and making informed decisions about database deployment.

A correct recovery system must provide two fundamental guarantees after any failure:

The Two Cardinal Guarantees:

These guarantees map directly to two ACID properties:

• Durability (D): Once committed, data persists through failures
• Atomicity (A): Uncommitted transactions are 'all or nothing'—and after failure, they're nothing

Formal Statement of Recovery Correctness:

Let DB_crash be the database state at the moment of failure, and DB_recovered be the database state after recovery completes. Recovery is correct if and only if:

DB_recovered = ApplyCommitted(DB_initial) 
             = The state that would result from perfectly executing
               all committed transactions on the initial database,
               with no trace of uncommitted transactions

This is a powerful requirement: the recovered database must look exactly as if all committed transactions completed normally and all uncommitted transactions never started.

Why Both Guarantees Matter:

Violating either guarantee has severe consequences:

Beyond correctness, recovery systems are measured by two key performance objectives:

Recovery Point Objective (RPO):

RPO specifies the maximum acceptable data loss measured in time. It answers: 'How much data can we afford to lose?'

• RPO = 0: No data loss tolerable. Requires synchronous replication or similar mechanisms.
• RPO = 15 minutes: Up to 15 minutes of transactions can be lost. Requires log backups every 15 minutes.
• RPO = 24 hours: Up to one day of data can be lost. Daily backups suffice.

Recovery Time Objective (RTO):

RTO specifies the maximum acceptable downtime. It answers: 'How long can we be offline?'

• RTO = 0 (or near-zero): Requires automatic failover to a standby.
• RTO = 1 hour: Manual recovery with prepared procedures is acceptable.
• RTO = 24 hours: Extended recovery processes are acceptable.

The RPO/RTO Matrix:

Different failure types have different RPO/RTO characteristics:

Calculating RPO Requirements:

RPO_needed = (Cost of lost transactions) × (Probability of failure) / (Cost of protection)

Simplified: What's the maximum you'd pay to recover one hour of work?
           If that exceeds the cost of hourly log backups, implement them.

Calculating RTO Requirements:

RTO_needed = Maximum acceptable downtime before unacceptable business impact

Consider:
- Revenue loss per hour of downtime
- SLA penalties
- Customer impact and churn risk
- Regulatory requirements
- Operational dependencies

Recovery doesn't happen by magic—it requires specific resources that must be maintained and protected. The availability and quality of these resources directly determines recovery capability.

Essential Recovery Resources:

Resource Protection Requirements:

Recovery resources themselves must be protected:

1. Log Protection:

2. Backup Protection:

3. Checkpoint Protection:

Checkpoints are transient (they define a point in time) but the checkpoint process must be reliable:

• Checkpoint must complete atomically
• Checkpoint record in log must be durable
• Dirty pages must be reliably flushed

Recovery operations themselves must satisfy certain properties to be reliable. These properties ensure that recovery works correctly even under adverse conditions:

Property 1: Idempotence

Recovery operations must be idempotent—applying them multiple times produces the same result as applying them once.

Why this matters: Recovery might be interrupted (another crash during recovery). When recovery restarts, it may re-execute operations it already performed. Idempotence guarantees this doesn't cause problems.

How it's achieved: Log Sequence Numbers (LSNs) track which operations have been applied. Before applying a redo operation to a page, compare the page's LSN to the log record's LSN. If the page LSN is already >= record LSN, the operation was already applied—skip it.

Property 2: Determinism

Recovery must be deterministic—given the same log and data files, recovery always produces the same result.

Why this matters:

• Testing and verification require predictable behavior
• Debugging recovery issues requires reproducibility
• Parallel recovery must coordinate deterministically

How it's achieved:

• Log records contain all information needed to reproduce operations
• No dependence on external state (time, random values) during recovery
• Recovery order is determined by log order, not timing

Property 3: Consistency Preservation

Recovery must maintain database consistency at all times:

• Internal page structure must be valid
• Indexes must match table data
• Constraints must be satisfied
• Cross-page references must be valid

How it's achieved:

• Operations are logged at a level that maintains consistency
• Physiological logging logs actions at page level with logical descriptions
• Page-level locks protect consistency during recovery

Recovery systems must balance recovery capability against normal operation performance. Stronger recovery guarantees often impose overhead during normal operation.

Normal Operation Overhead:

Recovery mechanisms impose costs during normal operation:

Recovery Operation Performance:

Recovery itself should be as fast as possible:

Factors Affecting Recovery Time:

1. Log Volume: More log records = longer redo/undo

   • Mitigation: Frequent checkpoints reduce log to process

2. Random I/O: Reading scattered pages for redo

   • Mitigation: SSDs provide fast random access; parallel recovery

3. Undo Transactions: Active transactions at crash time

   • Mitigation: Keep transactions short; limit concurrent transactions

4. Page Reads: Each dirty page must be read before redo

   • Mitigation: Pre-fetch pages during analysis phase

5. Sequential Processing: Traditional recovery is single-threaded

   • Mitigation: Modern systems use parallel recovery

Recovery system design involves fundamental trade-offs. Understanding these trade-offs is essential for making informed decisions about database configuration and deployment:

Trade-off 1: Durability vs. Performance

Trade-off 2: Recovery Time vs. Operation Overhead

Trade-off 3: Space vs. Recoverability

Trade-off 4: Simplicity vs. Flexibility

Simpler recovery mechanisms are easier to implement correctly and faster to execute, but less flexible:

A recovery system that has never been tested is not trustworthy. Verification and testing are essential requirements for any production recovery system:

Continuous Verification:

Disaster Recovery Testing:

Full disaster recovery tests should be conducted periodically:

1. Tabletop Exercises:

• Walk through recovery procedures on paper
• Identify gaps in documentation
• Ensure personnel know their roles
• No actual systems affected

2. Technical Dry Runs:

• Restore to non-production environment
• Verify data integrity
• Measure actual recovery time
• Test application connectivity

3. Full Failover Tests:

• Actually fail over to standby
• Run production workload on standby
• Verify all functionality works
• Practice failback procedure

4. Chaos Engineering:

• Randomly inject failures in production
• Verify systems recover automatically
• Find weaknesses before real failures
• Netflix's Chaos Monkey is the famous example