When you call mmap() on a 10GB file, something remarkable happens—or rather, doesn't happen. The kernel creates data structures describing the mapping, returns a pointer, and your application resumes execution in microseconds. No disk I/O. No memory allocation. No data copying. A 10GB file is "loaded" in the time it takes to read a few memory locations.

This seemingly impossible feat is achieved through lazy loading (also called demand paging when applied to virtual memory)—a fundamental principle that pervades modern operating system design. Rather than doing work upfront, the system defers work until the last possible moment: when you actually try to use the data.

void *map = mmap(NULL, 10737418240, PROT_READ, MAP_PRIVATE, fd, 0);  // 10GB
// Returns instantly! No 10GB of RAM allocated, no disk reads

char first_byte = ((char *)map)[0];  // NOW the kernel does work:
// - Allocates a physical page
// - Reads 4KB from disk
// - Maps it into your address space
// - Returns the byte

This lazy approach isn't just an optimization—it's the fundamental mechanism that makes virtual memory practical. Without it, computers couldn't run programs larger than physical RAM, couldn't share libraries efficiently between processes, and couldn't provide the illusion of abundant memory to applications.

A page fault is a CPU exception that occurs when a program accesses a virtual address that isn't currently mapped to physical memory. Far from being an error (despite the name), page faults are the mechanism that enables lazy loading.

The CPU's Perspective:

When your program executes an instruction that accesses memory, the CPU's Memory Management Unit (MMU) translates the virtual address to a physical address using the page table. If the page table entry indicates "not present," the CPU raises a page fault exception:

1. Exception raised: CPU saves the instruction pointer and transfers control to the kernel's page fault handler
2. Faulting address captured: The virtual address that caused the fault is stored in a special register (CR2 on x86)
3. Handler invoked: The kernel's page fault handler analyzes the situation
4. Resolution: The kernel determines the appropriate response:
   • Load data from disk and map the page
   • Allocate a new zero page
   • Deliver SIGSEGV (if the access is invalid)
5. Resume: Control returns to user space; the CPU retries the faulting instruction

Types of Page Faults:

Page faults come in three categories, with dramatically different performance implications:

Minor faults are the fast path. They occur when:

• The page exists in the page cache (another process mapped it, or you're re-faulting a reclaimed page)
• A forked process first touches a copy-on-write page that doesn't need copying yet
• The page was unmapped from your address space but still exists in memory

Major faults are the slow path. They require actual disk I/O:

• Initial access to a file page not in the page cache
• Page was reclaimed from the page cache and must be re-read
• Swap-in operations for anonymous pages

For memory-mapped files, the goal is to maximize minor faults and minimize major faults by keeping your working set in the page cache.

When a page fault occurs, the kernel must quickly determine what to do. The page fault handler is one of the most performance-critical paths in the kernel—it's invoked millions of times per second on a busy system.

Linux's do_page_fault() Function:

On Linux, the main entry point is architecture-specific (e.g., do_page_fault() on x86), which delegates to the generic handle_mm_fault(). Here's the conceptual flow:

// Simplified pseudocode
void handle_page_fault(unsigned long address, unsigned int error_code) {
    struct mm_struct *mm = current->mm;  // Process's memory descriptor
    
    // Step 1: Find the VMA containing this address
    struct vm_area_struct *vma = find_vma(mm, address);
    
    if (!vma || address < vma->vm_start) {
        // No mapping exists at this address
        // -> Deliver SIGSEGV
        return do_segfault(address);
    }
    
    // Step 2: Check access permissions
    if ((error_code & PF_WRITE) && !(vma->vm_flags & VM_WRITE)) {
        // Write to read-only region
        // -> Deliver SIGSEGV
        return do_segfault(address);
    }
    
    // Step 3: Handle the fault based on VMA type
    return handle_mm_fault(vma, address, error_code);
}

VMA-Specific Fault Handling:

Each VMA has a pointer to a vm_operations_struct containing fault handlers appropriate for its type:

File-Backed Fault Handling (filemap_fault):

When you fault on a memory-mapped file, the kernel's filemap_fault() function handles it:

1. Calculate file position: offset = (faulting_address - vma->vm_start) + vma->vm_pgoff * PAGE_SIZE

2. Search page cache: Look for the page at (file_inode, offset) in the page cache

3. If found (minor fault):

   • Lock the page
   • Install page table entry mapping virtual \u2192; physical
   • Return with page now accessible

4. If not found (major fault):

   • Allocate new page from page allocator
   • Add page to page cache (indexed by inode + offset)
   • Initiate disk I/O to read the page
   • Wait for I/O completion (process sleeps)
   • Install page table entry
   • Return

Performance Implications:

The path through the page fault handler critically affects performance:

• VMA lookup: Uses a red-black tree—O(log n) where n is number of VMAs. Processes with many mappings pay more.
• Page cache lookup: Hash table lookup—effectively O(1)
• Page allocation: Usually fast from per-CPU free lists, but can trigger reclaim if memory is low
• Disk I/O: The dominant cost when it occurs—nothing else compares

Loading pages one at a time, each requiring a separate disk I/O, would be disastrously slow for sequential access. The kernel employs read-ahead to predict future page accesses and load them proactively.

The Read-Ahead Mechanism:

When the kernel detects sequential access patterns, it loads pages ahead of where you're currently accessing:

Access Pattern Detection:
  Access page 0 → Load pages 0-3    (initial window: 4 pages)
  Access page 1 → Already cached    (read-ahead working!)
  Access page 2 → Already cached
  Access page 3 → Load pages 4-11   (window doubled: 8 pages)
  Access page 4 → Already cached
  ...
  Access page 11 → Load pages 12-27 (window grows: 16 pages)

The read-ahead window grows exponentially up to a maximum (often 128-256 KB by default), enabling very high sequential throughput.

Read-Ahead State Machine:

The kernel maintains read-ahead state per file mapping:

Synchronous vs. Asynchronous Read-Ahead:

• Synchronous read-ahead: Triggered when you access a page not in cache. The kernel loads a batch of pages, and your process waits for the I/O to complete.

• Asynchronous read-ahead: Triggered when you access a page within the "async" portion of the current window (typically the last quarter). The kernel initiates I/O for the next window without blocking your access—the pages you're accessing are already cached.

|--------- Current Window ----------|------ Next Window ------|  
|-- Already Accessed --|-- Async ---|                          
                        ^  
                        | Access here triggers async read-ahead  
                          for next window. You don't block;  
                          current page is already cached.  

How mmap() Interacts with Read-Ahead:

With mmap(), read-ahead is triggered by page faults, not system calls. The kernel can't always detect sequential patterns as easily because:

1. Page faults occur at page granularity (every 4KB), not byte granularity
2. The faulting address pattern might not clearly indicate sequential intent

To help the kernel, you can use madvise() to declare your access pattern:

#include <sys/mman.h>

// Hint: I'll access sequentially - please read ahead aggressively
madvise(mapped_addr, length, MADV_SEQUENTIAL);

// Hint: I'll access randomly - don't bother with read-ahead
madvise(mapped_addr, length, MADV_RANDOM);

// Hint: I'll need this soon - start loading now
madvise(mapped_addr, length, MADV_WILLNEED);

// Hint: I'm done with this - feel free to reclaim
madvise(mapped_addr, length, MADV_DONTNEED);

The working set is the set of pages actually being used by your program at a given time. Understanding working set dynamics is crucial for optimizing memory-mapped file performance.

Definition:

Formally, the working set at time t with window w is the set of pages accessed in the time interval [t-w, t]. In practice, we care about:

• How many pages are actively used?
• Do these pages fit in available RAM?
• What happens when they don't?

The Page Cache and Memory Pressure:

File-backed mmap() pages reside in the kernel's page cache. Under memory pressure:

1. Clean pages (unchanged from disk) can simply be discarded—they can be re-read from disk if needed
2. Dirty pages (modified) must be written to disk before reclamation
3. The kernel uses LRU-like algorithms to choose which pages to reclaim

                 ┌─────────────────────────────────────┐
                 │          Physical Memory            │
                 │  ┌───────────────────────────────┐  │
                 │  │       Page Cache              │  │
Memory-mapped ──►│  │   (File-backed pages)         │  │
files            │  │   [Clean: reclaimable]        │  │
                 │  │   [Dirty: must sync first]    │  │
                 │  └───────────────────────────────┘  │
                 │  ┌───────────────────────────────┐  │
Anonymous       ►│  │     Anonymous Pages           │  │
mappings         │  │   (Heap, stack, MAP_ANON)     │  │
mallocated       │  │   [May be swapped out]        │  │
memory           │  └───────────────────────────────┘  │
                 └─────────────────────────────────────┘

Measuring Working Set and Page Faults:

On Linux, monitor page faults using:

# Per-process stats via /proc
cat /proc/<pid>/stat   # Field 10: minor faults, Field 12: major faults

# Real-time monitoring
watch -n 1 'grep pgfault /proc/vmstat'

# Tool-based monitoring
perf stat -e page-faults ./your_program

# Detailed via sar
sar -B 1  # Page-in/out, fault rates

Strategies for Working Set Optimization:

Sometimes lazy loading isn't what you want. If you know you'll access every page of a mapping, loading them lazily incurs page fault overhead for each page. In such cases, you can request eager loading using MAP_POPULATE.

What MAP_POPULATE Does:

void *map = mmap(NULL, size, PROT_READ, 
                 MAP_PRIVATE | MAP_POPULATE,  // <-- Force pre-loading
                 fd, 0);

With MAP_POPULATE:

• The mmap() call takes longer (it does the I/O upfront)
• When mmap() returns, all pages are already in memory
• Subsequent accesses incur zero page faults

Performance Profile Comparison:

When to Use MAP_POPULATE:

1. You'll definitely access the entire file (e.g., loading a configuration file, reading a serialized data structure)
2. Predictable latency is critical (e.g., real-time systems where page fault latencies are unacceptable)
3. The file fits in RAM (otherwise you'll trigger memory pressure during mmap())
4. Startup time is acceptable (you're trading startup latency for runtime predictability)

Alternative: madvise() with MADV_WILLNEED:

For more control, you can combine lazy mmap() with explicit pre-faulting:

void *map = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);

// Pre-fault asynchronously - returns immediately, I/O happens in background
madvise(map, size, MADV_WILLNEED);

// Do other initialization work while pages load...
initialize_other_subsystems();

// By the time you access the mapping, pages are likely cached
process(map, size);

This approach combines lazy mmap() with proactive loading, giving you:

• Fast mmap() return (no blocking)
• Background I/O during other initialization
• Reduced page faults when you access the data

When memory-mapped file performance is poor, page faults are often the culprit. Here's how to investigate:

High-Level Monitoring:

# System-wide page fault statistics
vmstat 1
# Watch columns: si/so (swap), bi/bo (block I/O)
# High bi values during your workload indicate major faults

# Per-process page faults
pidstat -r 1 -p <pid>
# minflt/s: minor faults/sec (cheap)
# majflt/s: major faults/sec (expensive - these hurt!)

# Detailed breakdown
sar -B 1
# pgpgin/s, pgpgout/s: pages read from/written to disk
# fault/s, majflt/s: fault rates

Detailed Analysis with perf:

# Record page fault events
perf record -e page-faults,major-faults,minor-faults ./your_program

# Analyze results
perf report

# See which code paths trigger the most faults
perf annotate --symbol=your_function

Understanding mincore():

The mincore() system call lets you query which pages of a mapping are currently in memory:

#include <sys/mman.h>
#include <unistd.h>

void check_page_residency(void *map, size_t length) {
    long page_size = sysconf(_SC_PAGESIZE);
    size_t num_pages = (length + page_size - 1) / page_size;
    
    unsigned char *vec = malloc(num_pages);
    if (mincore(map, length, vec) == -1) {
        perror("mincore");
        return;
    }
    
    size_t resident = 0;
    for (size_t i = 0; i < num_pages; i++) {
        if (vec[i] & 1) resident++;
    }
    
    printf("Resident: %zu / %zu pages (%.1f%%)\n",
           resident, num_pages, 100.0 * resident / num_pages);
    
    free(vec);
}

Common Performance Problems:

When you combine MAP_PRIVATE with read-write access, lazy loading interacts with copy-on-write (COW) semantics:

Initial State:

• Pages are not present in your page table
• First read triggers major fault → page loaded from file, mapped read-only

On First Write:

• Write triggers protection fault (page is marked read-only)
• Kernel creates a private copy of the page
• New copy is mapped read-write into your address space
• Your write proceeds on the private copy
• Original file is unchanged

Timeline: MAP_PRIVATE with lazy loading

mmap() returned     Read page 0       Write to page 0
     |                   |                   |
     v                   v                   v
  [Empty]           [Read-only]         [Read-write]
                    from page cache     private copy
                                        original unchanged

This Has Performance Implications:

For read-heavy workloads on MAP_PRIVATE mappings:

• Initial reads fault pages in from the file
• Subsequent reads hit the same pages (now in your page table)
• Very efficient—pages are shared with the page cache

For write-heavy workloads on MAP_PRIVATE mappings:

• Each first write to a page triggers TWO faults:
  1. Major fault to load the page (if not already loaded)
  2. COW fault to create a private copy
• Private copies consume additional RAM (no longer shared)
• Write patterns that touch many pages \u2192 double the fault overhead

Optimization: Write-Only Access Pattern:

If you're going to overwrite entire pages (not read-modify-write), consider:

// For pure overwrites, MAP_ANONYMOUS avoids reading the file at all
void *workspace = mmap(NULL, size, PROT_READ | PROT_WRITE,
                       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

// Pages are zero-filled on first access (minor fault only)
// No file I/O involved

This is faster than mapping a file with MAP_PRIVATE when you don't need the original content.

We've comprehensively explored lazy loading—the fundamental mechanism that makes memory-mapped files efficient and enables programs to work with datasets larger than physical memory.

What's Next:

With lazy loading understood, we'll explore shared mappings—how multiple processes can map the same file into their address spaces and see each other's changes. This is the foundation of shared-memory IPC, memory-mapped databases, and collaborative file access patterns.