Log Concept

Imagine a scenario that haunts every database administrator: a power outage strikes at the worst possible moment—right in the middle of a critical banking transaction transferring $50,000 between accounts. The money has been deducted from one account, but before it could be credited to the other, everything went dark. When the system restarts, how does the database know what happened? How does it recover to a consistent state without losing the customer's money or, worse, duplicating it?

The answer lies in one of the most elegant and essential mechanisms in database systems: the transaction log.

The transaction log is the unsung hero of database reliability. While indexes speed up queries and buffer pools optimize performance, it's the log that makes the fundamental promise of databases possible—that once data is committed, it will survive any failure, and if a transaction is incomplete, it will be as if it never happened.

A transaction log (also called a write-ahead log, redo log, or simply log) is a sequential, append-only file that records every modification made to the database. Before any change is applied to the actual data pages on disk, a record of that change is first written to the log. This seemingly simple principle—log first, then modify—is the foundation of all database recovery mechanisms.

Think of the transaction log as a meticulous diary that the database keeps. Every single operation—inserting a row, updating a value, deleting a record—is documented in this diary before it happens. If anything goes wrong, the database can read this diary to understand exactly what happened and determine how to restore consistency.

Key Characteristics of Transaction Logs:

• Sequential Writing: Log records are written in strict chronological order, one after another. This sequential nature makes logging extremely fast—much faster than random disk I/O for data pages.

• Append-Only Structure: New records are always added at the end. Existing records are never modified or overwritten (except during log truncation/archival). This immutability is crucial for reliability.

• Durable Storage: Logs are written to stable storage (disk, SSD, or other persistent media) and often to multiple locations. The log must survive the same failures it's designed to recover from.

• Sufficient Information: Each log record contains enough information to either redo the operation (for completed transactions) or undo it (for incomplete transactions).

To understand why transaction logs are absolutely essential, we need to consider the fundamental challenge that databases face: the disparity between volatile and persistent memory.

Databases perform modifications in memory (in the buffer pool) for performance reasons. Writing every change immediately to disk would be prohibitively slow. However, memory is volatile—it loses its contents when power is lost. This creates a dilemma:

• For performance: Keep modified pages in memory as long as possible
• For durability: Ensure committed data survives failures

Without a transaction log, these goals would be mutually exclusive. The log resolves this tension elegantly.

A transaction log is organized as a sequence of log records, each documenting a specific action performed by a transaction. Understanding the structure of these records is fundamental to understanding how recovery works.

Each log record contains identifying information that allows the recovery system to process it correctly. While specific implementations vary, conceptually every log record includes these core elements:

The LSN: The Log's Primary Key

The Log Sequence Number (LSN) deserves special attention. It serves multiple critical purposes:

1. Unique Identification: Every log record has a unique LSN that never repeats.

2. Ordering: LSNs are assigned in the order operations occur, providing a total ordering of all database modifications.

3. Page Tracking: Each data page stores the LSN of the last log record that modified it. This allows the recovery system to quickly determine if a page needs to be updated during redo.

4. Recovery Coordination: The LSN serves as a reference point for determining what work needs to be done during crash recovery.

The LSN is typically implemented as a combination of log file number and offset within that file, making it both unique and ordered.

One of the most powerful ways to understand the transaction log is to view it as a complete timeline of database history. Unlike data files, which only show the current state, the log preserves the sequence of events that led to that state.

Consider this analogy: If the database's data files are like a photograph—showing how things look right now—then the transaction log is like a video recording—capturing every moment of action that brought us to the present.

This timeline perspective reveals why the log is so valuable:

Replaying History

Because the log is a complete record of changes, you can theoretically:

1. Start from an empty database and replay the entire log to arrive at the current state
2. Start from any backup and replay the log from that point forward
3. Stop at any point in the log to see what the database looked like at that moment

This capability is fundamental to point-in-time recovery (PITR), where administrators can restore a database to any specific moment—for example, "right before that accidental DELETE was executed."

The Timeline in Practice:

When a crash occurs, the timeline nature of the log enables precise recovery:

1. Each committed transaction's changes are present in order
2. Each uncommitted transaction's partial changes are also recorded
3. The recovery process replays (redoes) committed work
4. Then it reverses (undoes) uncommitted work
5. Result: Database returns to a consistent state

The transaction log's power derives from a strict protocol known as Write-Ahead Logging (WAL). This protocol establishes the fundamental guarantee that makes recovery possible:

The WAL Rule: Before any data modification is written to the database files on disk, the log record describing that modification must first be written to stable storage.

This rule is absolute—there are no exceptions. Every database system that provides durability guarantees enforces this rule rigorously.

The Flow of a Modification:

1. Transaction Modifies Data: A transaction updates a row in the buffer pool (in memory)

2. Log Record Created: A log record is created describing the change (what table, what page, old value, new value)

3. Log Record Written to Disk: The log record is written to the log buffer and then flushed to stable storage

4. Data Page Eventually Written: At some later point, the modified data page is written from the buffer pool to the data files on disk

The critical sequence is: Log record to disk → Data page to disk. Never the reverse.

Why This Order Matters:

The Commit Protocol:

WAL has a specific requirement for commits:

When a transaction commits, its commit log record must be written to stable storage before the commit is acknowledged to the application.

This ensures that once an application receives confirmation that a transaction committed, the database guarantees that the transaction's effects will survive any subsequent failure. This is the durability promise in action.

Let's trace through a complete transaction to see how the log captures its activity. Consider a funds transfer from Account A to Account B:

The Log Records Generated:

As this transaction executes, the database generates a sequence of log records. Here's what the log would conceptually contain:

What the Log Enables:

• If crash occurs after LSN 001: Transaction was just starting. No changes to undo.

• If crash occurs after LSN 002: First update was logged. During recovery, this uncommitted change will be undone using the before image (balance restored to $1000).

• If crash occurs after LSN 003: Both updates logged but no commit. During recovery, both changes will be undone (A → $1000, B → $2000).

• If crash occurs after LSN 004: Commit is on stable storage. During recovery, if data pages weren't written yet, the changes will be redone using the after images. The commitment survives.

This is the magic of log-based recovery: regardless of when the crash happens, the database can always reach a consistent state where all committed transactions are fully applied and all uncommitted transactions are fully reversed.

Given the complexity that logging adds, you might wonder: are there alternatives? Why not just write every change directly to disk? Or keep backup copies? Let's examine why logging emerged as the dominant approach:

Why WAL Wins:

1. Sequential I/O: Logging uses sequential writes, which are dramatically faster than the random writes needed for direct data page updates. Modern storage still shows 10-100x performance differences.

2. Minimal Commit Latency: A commit only requires writing a small log record, not potentially dozens of scattered data pages.

3. Buffer Pool Optimization: Data pages can remain in the buffer pool and be written opportunistically, potentially batching multiple modifications to the same page.

4. Complete History: The log provides capabilities beyond recovery—replication, auditing, point-in-time restore—that other approaches cannot match.

5. Proven at Scale: WAL has been tested and refined over 40+ years in systems handling billions of transactions. It works.