Logical Vs Physical Addresses - Learning Module

Loading content...

0/227

Physical Address Space

Where Data Actually Lives

While programs operate in the comfortable abstraction of logical addresses, real data must reside somewhere tangible—in actual electronic circuits that store and retrieve binary information. This is the domain of physical addresses: the true locations in hardware where every byte of data ultimately exists.

Understanding physical address space is essential because it represents the fundamental constraint against which all memory management operates. Logical addresses might promise gigabytes or terabytes of space, but physical memory is finite, expensive, and shared among all running processes. The operating system's memory subsystem is fundamentally a resource manager, and the resource being managed is physical address space.

What You Will Learn

By the end of this page, you will understand what physical addresses are, how physical memory is organized in modern systems, the hardware constraints that make physical memory management necessary, how physical addressing differs fundamentally from logical addressing, and the challenges that arise from this duality.

Definition of Physical Address

A physical address is a hardware address that identifies a specific location in the computer's main memory (RAM) or in memory-mapped hardware devices. Physical addresses are what the memory bus ultimately uses to access actual storage cells.

Formally:

A physical address is an identifier that specifies an exact location in the physical memory hardware, used by the memory controller to select specific memory cells for read or write operations.

The collection of all valid physical addresses in a system constitutes the physical address space.

Key Properties of Physical Addresses

•Hardware-Determined: Physical address space size is determined by the memory controller and system bus width, not by the CPU's logical addressing capabilities. A system may have 48-bit logical addresses but only 36-bit physical addresses.
•Finite and Bounded: Unlike potentially vast logical address spaces, physical address space is limited by installed RAM and configured memory-mapped regions. You cannot have more physical addresses than there is hardware to back them.
•Shared Resource: A single physical address space is shared by all processes, the operating system kernel, and memory-mapped devices. This sharing necessitates protection mechanisms.
•Uniform Addressing: Each physical address refers to a single byte (on byte-addressable systems). Address 0x1000 refers to the same memory cell regardless of which process's code causes it to be accessed.
•Direct Hardware Access: Physical addresses are what appear on the memory bus. They are the 'real' addresses that memory chips respond to.

Physical vs. Logical Address Space Sizes:

A key insight is that physical and logical address spaces are sized independently:

System Type	Logical Address Bits	Physical Address Bits	Logical Space	Physical Space (Typical)
32-bit x86	32 bits	32-36 bits	4 GB	512 MB - 4 GB
x86-64 (standard)	48 bits	46-52 bits	256 TB	8 GB - 1 TB
x86-64 (5-level)	57 bits	52 bits	128 PB	Up to 4 PB
ARM64	48/52 bits	48-52 bits	256 TB+	Varies

Notice that logical space typically exceeds physical space by orders of magnitude—this is the foundation of virtual memory.

Physical Address ≠ DRAM Address

Physical addresses are system-level addresses. The memory controller further translates physical addresses into DRAM-specific signals: channel, rank, bank, row, and column selects. This second level of translation is transparent to software but significantly affects memory performance—understanding it is crucial for performance-critical systems programming.

Physical Memory Organization

Physical memory in a modern system is not a simple linear array of bytes. It's a complex hierarchical system with multiple components, each affecting how physical addresses map to actual storage.

Converting Mermaid diagram...

DRAM Internal Structure:

Within each DIMM (Dual In-line Memory Module), memory is organized hierarchically:

Ranks: One or two independent sets of DRAM chips that share addressing but can operate semi-independently.
Banks: Each rank contains multiple banks (typically 8-16 in DDR4/DDR5). Banks can be accessed in parallel to improve throughput.
Rows (Pages): Each bank contains thousands of rows. Accessing a different row requires a precharge/activate cycle—the primary source of DRAM latency.
Columns: Within an active row, different columns can be accessed relatively quickly.

The memory controller translates physical addresses into the specific {channel, rank, bank, row, column} combination.

physical_address_decoding.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
/*
 * Physical Address to DRAM Location Mapping
 * 
 * This is a simplified example of how a physical address might be
 * decoded into DRAM addressing signals. Real implementations vary
 * based on memory controller design and interleaving policies.
 */
 
#include <stdint.h>
#include <stdio.h>
 
// Example: 16 GB system, 2 channels, 2 ranks per channel, 8 banks per rank
// DDR4-style addressing
 
typedef struct {
    uint8_t channel;    // 0-1 (1 bit)
    uint8_t rank;       // 0-1 (1 bit)
    uint8_t bank_group; // 0-3 (2 bits)
    uint8_t bank;       // 0-3 (2 bits within group)
    uint16_t row;       // 0-32767 (15 bits)
    uint8_t column;     // 0-1023 (10 bits, but lower 3 for burst)
} DRAMAddress;
 
/*
 * Example physical address layout (48-bit physical address):
 * 
 * Bits:  47    ...    17 | 16-15 | 14-13 | 12-11 | 10-7 | 6     | 5-3  | 2-0
 * Field: Row (15+ bits)  | Bank  | BankGr| Column| Col  |Channel| Rank | Byte
 * 
 * Note: Real systems interleave differently for performance.
 * Channel bits might be in lower positions for better parallelism.
 */
 
DRAMAddress decode_physical_address(uint64_t phys_addr) {
    DRAMAddress dram;
    
    // Byte offset within cache line (typically ignored by DRAM, handled by cache)
    // uint8_t byte_offset = phys_addr & 0x7;
    
    // Column address (cache line granularity)
    dram.column = (phys_addr >> 6) & 0x3FF;  // Bits 6-15 (simplified)
    
    // Channel interleaving (often at cache line level for parallelism)
    dram.channel = (phys_addr >> 6) & 0x1;   // Bit 6 (example)
    
    // Rank selection
    dram.rank = (phys_addr >> 16) & 0x1;     // Bit 16 (example)
    
    // Bank group and bank (DDR4/DDR5)
    dram.bank_group = (phys_addr >> 17) & 0x3;
    dram.bank = (phys_addr >> 19) & 0x3;
    
    // Row address (high-order bits)
    dram.row = (phys_addr >> 21) & 0x7FFF;   // Bits 21-35
    
    return dram;
}
 
void print_dram_address(uint64_t phys_addr) {
    DRAMAddress dram = decode_physical_address(phys_addr);
    
    printf("Physical Address: 0x%012llx\n", (unsigned long long)phys_addr);
    printf("  Channel:    %d\n", dram.channel);
    printf("  Rank:       %d\n", dram.rank);
    printf("  Bank Group: %d\n", dram.bank_group);
    printf("  Bank:       %d\n", dram.bank);
    printf("  Row:        %d\n", dram.row);
    printf("  Column:     %d\n", dram.column);
}
 
/*
 * Performance Implications:
 * 
 * - Accesses to the same row (row buffer hit): ~10-15 ns
 * - Accesses to different row in same bank (row buffer miss): ~50-60 ns
 * - Accesses to different banks (parallel): Lower latency
 * 
 * Memory allocators and OS page placement can optimize for these patterns.
 */

NUMA: Non-Uniform Memory Access

In multi-socket systems, physical memory is distributed across NUMA nodes. Each CPU socket has its own memory controller and 'local' RAM. Accessing remote memory (attached to another socket) has higher latency—often 50-100% more. The OS tries to allocate physical memory local to the CPU that will use it, adding another dimension to physical address management.

Physical Address Space Layout

The physical address space is not entirely dedicated to RAM. Various regions are reserved for specific purposes, creating a complex map that the operating system must navigate.

physical_address_map.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Typical x86-64 Physical Address Space Layout:
 
Physical Address        | Size          | Description
------------------------|---------------|----------------------------------
0x00000000 - 0x0009FFFF | 640 KB        | Conventional Memory (legacy)
0x000A0000 - 0x000FFFFF | 384 KB        | Legacy ROM, VGA memory (hole)
0x00100000 - 0x7FFFFFFF | ~2 GB         | Main RAM (below 4 GB mark)
0x80000000 - 0xDFFFFFFF | ~1.5 GB       | RAM continues (varies)
0xE0000000 - 0xFFFFFFFF | ~512 MB       | Memory-mapped PCI devices, APIC,
                        |               | firmware (BIOS/UEFI), ROM
------------------- 4 GB boundary ("MMIO hole") -------------------
0x100000000 - 0x????????| Variable      | RAM above 4 GB (main memory)
0x???????? - High       | Variable      | Additional RAM, PCIe config space
 
Key Points:
- The region 0xE0000000-0xFFFFFFFF is NOT RAM; accessing these
  addresses reaches hardware devices, not DRAM
  
- The "MMIO hole" means a 32-bit system with 4 GB RAM doesn't
  actually have 4 GB usable—some is lost to device mappings
  
- Memory above 4 GB requires 64-bit addressing (PAE or x86-64)
 
- The kernel receives this map from BIOS/UEFI (E820 map on x86)
  and must work around the gaps

Physical Address Space Regions
Region Type	Purpose	Accessible By	Characteristics
Conventional RAM	Program code and data	CPU via memory controller	Fast, volatile, cached
Memory-Mapped I/O	Device registers	CPU to device controllers	Not cached, side effects on access
Firmware ROM	BIOS/UEFI code	CPU at boot, read-only	Non-volatile, may be shadowed
Reserved	Hardware-specific	Varies	Cannot be used for RAM
PCI Config Space	Device configuration	OS via special access	Small (256B/device), structured
APIC Registers	Interrupt controller	Kernel for IRQ management	Memory-mapped at fixed address

The E820 Memory Map:

At boot time, the BIOS or UEFI firmware provides the operating system with a memory map (the E820 map on x86 systems) describing which physical address ranges are:

Usable RAM: Can be used for normal memory allocation
Reserved: Hardware-reserved, do not use
ACPI Data: Firmware tables the OS needs to preserve
ACPI NVS: Non-volatile storage used by firmware
Unusable: Defective or otherwise unavailable

The operating system must respect these designations. Using reserved memory can cause hardware malfunctions or crashes.

read_physical_memory_map.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
/*
 * Reading Physical Memory Information on Linux
 * 
 * The kernel exposes physical memory information through /proc and /sys
 */
 
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
 
void print_iomem() {
    // /proc/iomem shows physical address space allocation
    printf("=== Physical Address Space Map (/proc/iomem) ===\n\n");
    
    FILE *f = fopen("/proc/iomem", "r");
    if (!f) {
        perror("Cannot open /proc/iomem (requires root)");
        return;
    }
    
    char line[256];
    int count = 0;
    while (fgets(line, sizeof(line), f) && count < 30) {
        printf("%s", line);
        count++;
    }
    if (count == 30) printf("... (truncated)\n");
    fclose(f);
}
 
void print_memory_info() {
    printf("\n=== Memory Statistics (/proc/meminfo) ===\n\n");
    
    FILE *f = fopen("/proc/meminfo", "r");
    if (!f) {
        perror("Cannot open /proc/meminfo");
        return;
    }
    
    char line[256];
    int count = 0;
    while (fgets(line, sizeof(line), f) && count < 10) {
        printf("%s", line);
        count++;
    }
    fclose(f);
}
 
/*
 * Sample /proc/iomem output:
 * 
 * 00000000-00000fff : Reserved
 * 00001000-0009fbff : System RAM
 * 0009fc00-0009ffff : Reserved
 * 000a0000-000bffff : PCI Bus 0000:00
 * 000c0000-000c7fff : Video ROM
 * 000e0000-000fffff : Reserved
 *   000f0000-000fffff : System ROM
 * 00100000-7bfeffff : System RAM
 *   01000000-0199b443 : Kernel code
 *   0199b444-01eb1b3f : Kernel data
 *   02068000-020fafff : Kernel bss
 * 7bf00000-7bffffff : RAM buffer
 * 80000000-90ffffff : PCI Bus 0000:00
 * e0000000-efffffff : PCI MMCONFIG 0
 * f0000000-f7ffffff : PCI Bus 0000:00
 *   f0000000-f7ffffff : 0000:00:02.0
 * fed00000-fed003ff : HPET 0
 * fee00000-fee00fff : Local APIC
 * 100000000-27fffffff : System RAM
 * 
 * Notice: System RAM regions are scattered, with holes for devices
 */
 
int main() {
    print_iomem();
    print_memory_info();
    return 0;
}

The 640 KB Barrier Legacy

The strange layout of the low physical addresses (640 KB usable, then 384 KB of I/O) is a legacy from the original IBM PC's design in 1981. Despite being irrelevant for modern computing, compatibility requirements have preserved this layout for over 40 years. UEFI systems can optionally avoid some of these constraints, but legacy BIOS boot still requires it.

Physical Memory as a Shared Resource

Unlike logical address spaces—where each process has its own private space—physical address space is a single, shared resource. This sharing is both a challenge and an opportunity.

Challenges of Sharing

•Protection Required: Without hardware protection, any process could read/write any physical location—including kernel memory or other processes' data.
•Contention for Resources: Multiple processes compete for finite physical memory. The OS must fairly and efficiently allocate this scarce resource.
•Fragmentation Risk: As processes allocate and free memory, physical address space can become fragmented, making large contiguous allocations impossible.
•Coordination Complexity: The OS must track every physical page: who owns it, what's in it, whether it can be reclaimed.

Opportunities of Sharing

•Memory Deduplication: Identical physical pages (e.g., common libraries) need only one physical copy, mapped into multiple address spaces.
•Efficient Communication: Shared memory regions in physical space enable zero-copy inter-process communication.
•Copy-on-Write: fork() can share physical pages between parent and child until a write occurs, saving both memory and time.
•Resource Overcommitment: Virtual memory allows promising more logical memory than physically exists, with physical pages allocated on demand.

Physical Page as the Unit of Management:

Operating systems manage physical memory in fixed-size units called frames (when discussing physical memory) or pages (when discussing logical memory). Typically these are 4 KB, though modern systems support larger pages (2 MB, 1 GB) for specific use cases.

The frame size represents a tradeoff:

Smaller frames: Less internal fragmentation, finer granularity, but more metadata overhead and larger page tables
Larger frames: Less overhead, better TLB coverage, but more internal fragmentation and coarser protection granularity

4 KB has been the standard page size since the 1970s and remains dominant, though the performance benefits of huge pages are increasingly recognized.

physical_frame_management.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
/*
 * Conceptual Physical Frame Management
 * 
 * The OS maintains data structures to track every physical frame:
 * - Who owns it (kernel, which process, or free)
 * - Reference count (how many mappings point to it)
 * - Status flags (dirty, accessed, locked, etc.)
 */
 
#include <stdint.h>
#include <stdbool.h>
 
#define PAGE_SIZE 4096
#define MAX_PHYSICAL_PAGES (16ULL * 1024 * 1024 * 1024 / PAGE_SIZE)  // 16 GB
 
typedef enum {
    FRAME_FREE,           // Available for allocation
    FRAME_KERNEL,         // Used by kernel code/data
    FRAME_USER,           // Allocated to user process
    FRAME_RESERVED,       // Hardware reserved, unusable
    FRAME_PAGE_TABLE,     // Used for page tables
    FRAME_DMA,            // Reserved for device DMA
} FrameState;
 
typedef struct {
    FrameState state;
    uint32_t owner_pid;       // Process ID (if USER)
    uint16_t ref_count;       // Number of mappings to this frame
    uint16_t flags;           // Dirty, accessed, pinned, etc.
} FrameDescriptor;
 
// The page frame database - one entry per physical frame
FrameDescriptor frame_db[MAX_PHYSICAL_PAGES];
 
// Free list for O(1) allocation
uint64_t* free_list;
uint64_t free_count;
 
/*
 * Allocate a physical frame
 * Returns the physical frame number (PFN), or -1 if OOM
 */
int64_t allocate_frame(uint32_t pid) {
    if (free_count == 0) {
        // Out of physical memory!
        // Options: reclaim from caches, swap, or OOM kill
        return -1;
    }
    
    // Pop from free list
    uint64_t pfn = free_list[--free_count];
    
    // Initialize frame descriptor
    frame_db[pfn].state = FRAME_USER;
    frame_db[pfn].owner_pid = pid;
    frame_db[pfn].ref_count = 1;
    frame_db[pfn].flags = 0;
    
    // Zero the frame for security (don't leak data between processes)
    // memset(physical_to_virtual(pfn * PAGE_SIZE), 0, PAGE_SIZE);
    
    return pfn;
}
 
/*
 * Free a physical frame back to the pool
 */
void free_frame(uint64_t pfn) {
    if (--frame_db[pfn].ref_count == 0) {
        frame_db[pfn].state = FRAME_FREE;
        frame_db[pfn].owner_pid = 0;
        free_list[free_count++] = pfn;
    }
    // If ref_count > 0, the frame is still mapped elsewhere (shared memory)
}
 
/*
 * Physical Frame Number (PFN) to Physical Address conversion
 * 
 * Physical Address = PFN × PAGE_SIZE
 * PFN = Physical Address / PAGE_SIZE
 */
uint64_t pfn_to_physical(uint64_t pfn) {
    return pfn * PAGE_SIZE;
}
 
uint64_t physical_to_pfn(uint64_t physical) {
    return physical / PAGE_SIZE;
}
 
/*
 * In Linux, this data structure is 'struct page' (the page frame database)
 * and is one of the most critical kernel data structures.
 * 
 * With 16 GB RAM and 4 KB pages: 4 million frames to track
 * At ~64 bytes per 'struct page': ~256 MB just for frame metadata!
 */

The Frame Table Overhead

The frame table (page frame database) consumes significant memory—roughly 1.5% of total RAM in Linux. For a 256 GB server, that's about 4 GB just to track physical frames. This overhead is unavoidable: the OS must know the status of every physical frame to manage memory correctly.

Physical vs. Logical: The Critical Comparison

Understanding the fundamental differences between physical and logical address spaces is essential for grasping how memory management works.

Physical vs. Logical Address Space Comparison
Aspect	Logical Address Space	Physical Address Space
Origin	Generated by CPU during execution	Actual hardware memory locations
Per-Process?	Yes, each process has its own	No, single shared space system-wide
Size	Determined by CPU architecture (32/64-bit)	Determined by installed RAM + MMIO
Continuity	Appears contiguous to process	May be fragmented with holes
Protection	Achieved through translation tables	Hardware prevents unauthorized access
Visibility	Seen by running programs	Hidden from applications
Lifespan	Exists while process runs	Persists across process lifetimes
Address 0	Valid (often first page of text segment)	Typically low memory region (not usable by apps)
Sparse Usage	Common—large unmapped gaps	Every byte is accounted for
Translation Required	Yes, for every memory access	No, this IS the final address

The Binding Problem:

The fundamental challenge of memory management is binding: connecting logical addresses (what programs use) to physical addresses (where data actually is). This binding can happen at different times:

Compile Time: Absolute addresses are hardcoded. Only works if physical load address is known in advance. Obsolete for general-purpose computing.
Load Time: Addresses are adjusted when the program is loaded into memory. The program must stay at that physical location for its lifetime.
Execution Time: Addresses are translated dynamically during execution. This is the modern approach, enabling relocation, sharing, and virtual memory.

Modern systems use execution-time binding exclusively, with hardware (the MMU) performing translation on every memory access.

Converting Mermaid diagram...

Why This Matters for Programmers

When debugging, the addresses you see in gdb or print statements are logical addresses—they won't match what you'd see examining physical RAM. When analyzing performance, understanding physical layout (cache lines, NUMA nodes, page placement) requires thinking beyond the logical abstraction. Both perspectives are necessary for systems programming.

Physical Memory Allocation Challenges

Managing physical memory presents unique challenges that don't exist in the logical address realm. These challenges drive many design decisions in operating system memory subsystems.

Physical Memory Allocation Challenges

•External Fragmentation: Over time, free physical memory becomes scattered in small, non-contiguous chunks. While logical address space can be contiguous (pages map to anywhere), some allocations (DMA buffers, huge pages) need contiguous physical memory.
•Zone Constraints: Some physical addresses have special requirements. The first 16 MB may be needed for legacy ISA DMA. The first 4 GB is addressable by 32-bit devices. Physical memory is divided into zones with different allocation rules.
•NUMA Locality: Allocating physical memory from the wrong NUMA node can double memory access latency. The allocator must track which CPUs are 'close' to which physical memory.
•Cache Coloring: Physical address bits affect cache placement. Poor allocation can cause cache conflicts. Some high-performance systems explicitly manage physical allocation for cache efficiency.
•Memory Pressure: When physical memory runs low, the OS must reclaim frames from less important uses—caches, inactive processes, or swapping to disk. These decisions affect system responsiveness.
•Hot-Add/Remove: Modern servers support adding or removing RAM while running. The OS must handle physical address space changing dynamically.

Zone-Based Allocation:

Linux divides physical memory into zones to handle these constraints:

Zone	Physical Address Range	Purpose
ZONE_DMA	0-16 MB	Legacy ISA DMA devices
ZONE_DMA32	0-4 GB	32-bit device DMA
ZONE_NORMAL	Above 4 GB	General purpose allocation
ZONE_MOVABLE	High memory	Can be migrated for memory hotplug
ZONE_DEVICE	Varies	Persistent memory, device memory

When allocating physical memory, the kernel prefers ZONE_NORMAL but falls back to other zones if necessary.

zone_allocation.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
/*
 * Physical Memory Zones (Linux-style)
 * 
 * Different physical address ranges have different capabilities.
 * The allocator must respect these constraints.
 */
 
#include <stdint.h>
#include <stdbool.h>
 
typedef enum {
    ZONE_DMA,       // 0 - 16 MB: Legacy ISA DMA
    ZONE_DMA32,     // 0 - 4 GB: 32-bit device addressing
    ZONE_NORMAL,    // All memory: General purpose
    ZONE_MOVABLE,   // Hot-pluggable memory
    ZONE_COUNT
} MemoryZone;
 
typedef struct {
    uint64_t start_pfn;      // First page frame number in zone
    uint64_t end_pfn;        // Last page frame number in zone
    uint64_t free_pages;     // Count of free pages
    uint64_t min_watermark;  // Minimum free pages before reclaim
    uint64_t high_watermark; // Target free pages after reclaim
    // ... free lists, per-cpu caches, etc.
} Zone;
 
Zone zones[ZONE_COUNT];
 
/*
 * Allocation flags determine which zones can satisfy a request
 */
typedef enum {
    GFP_DMA     = (1 << 0),   // Must be in DMA zone
    GFP_DMA32   = (1 << 1),   // Must be below 4 GB
    GFP_NORMAL  = (1 << 2),   // Any zone is acceptable
    GFP_MOVABLE = (1 << 3),   // Prefer movable zone
    GFP_NOIO    = (1 << 4),   // Don't start I/O to reclaim
    GFP_NOWAIT  = (1 << 5),   // Don't wait, fail if unavailable
} GFPFlags;
 
/*
 * Zone fallback order
 * 
 * When ZONE_NORMAL is empty, we might try DMA32, then DMA.
 * This is a policy decision with performance implications.
 */
MemoryZone zone_fallback[ZONE_COUNT][ZONE_COUNT] = {
    [ZONE_DMA]     = { ZONE_DMA },                           // DMA: no fallback
    [ZONE_DMA32]   = { ZONE_DMA32, ZONE_DMA },               // DMA32: fall to DMA
    [ZONE_NORMAL]  = { ZONE_NORMAL, ZONE_DMA32, ZONE_DMA },  // Normal: fall through
    [ZONE_MOVABLE] = { ZONE_MOVABLE, ZONE_NORMAL, ZONE_DMA32, ZONE_DMA },
};
 
/*
 * Allocate a page from appropriate zone
 */
uint64_t alloc_page(GFPFlags flags) {
    MemoryZone preferred;
    
    // Determine preferred zone from flags
    if (flags & GFP_DMA)
        preferred = ZONE_DMA;
    else if (flags & GFP_DMA32)
        preferred = ZONE_DMA32;
    else if (flags & GFP_MOVABLE)
        preferred = ZONE_MOVABLE;
    else
        preferred = ZONE_NORMAL;
    
    // Try zones in fallback order
    for (int i = 0; i < ZONE_COUNT; i++) {
        MemoryZone zone = zone_fallback[preferred][i];
        if (zones[zone].free_pages > zones[zone].min_watermark) {
            // Allocate from this zone
            zones[zone].free_pages--;
            // return pfn from zone's free list
        }
    }
    
    // All zones exhausted - trigger reclaim or OOM
    // ...
    
    return (uint64_t)-1;  // Allocation failed
}

The Physical Contiguity Problem

As the system runs, physical memory becomes fragmented. Allocating large contiguous physical regions (e.g., for huge pages or DMA buffers) can fail even with plenty of total free memory. This is why the kernel periodically compacts memory—moving pages to create larger contiguous regions—at the cost of performance during compaction.

Accessing Physical Addresses From Software

In a properly designed system, user-space programs cannot directly access physical addresses—all their memory accesses go through logical-to-physical translation. However, the kernel and certain specialized programs need physical address access. Understanding how this works illuminates the physical/logical boundary.

Kernel Direct Mapping:

The kernel typically maintains a direct mapping of all physical memory into its logical address space. On Linux x86-64, physical address X is accessible at logical address PAGE_OFFSET + X (where PAGE_OFFSET is typically 0xffff888000000000). This allows the kernel to access any physical frame by simply computing the corresponding virtual address:

void* physical_to_virtual(uint64_t physical_addr) {
    return (void*)(physical_addr + PAGE_OFFSET);
}

This direct mapping exists only in kernel space. User processes cannot access these addresses—the page tables don't include mappings for kernel regions in user mode.

physical_access_methods.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
/*
 * Methods for Accessing Physical Memory
 * 
 * Different contexts require different approaches
 */
 
/* ============================================
 * METHOD 1: Kernel Direct Mapping (Linux Kernel)
 * ============================================
 * The kernel has all physical memory mapped into its address space.
 * Converting is trivial:
 */
 
// In Linux kernel code:
#define PAGE_OFFSET 0xffff888000000000UL
 
void* phys_to_virt(uint64_t phys_addr) {
    return (void*)(phys_addr + PAGE_OFFSET);
}
 
uint64_t virt_to_phys(void* virt_addr) {
    return (uint64_t)virt_addr - PAGE_OFFSET;
}
 
/* ============================================
 * METHOD 2: /dev/mem (User Space, Root Only)
 * ============================================
 * Character device providing raw physical memory access.
 * Extremely dangerous; disabled by default on most systems.
 */
 
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
 
void* map_physical_memory(uint64_t phys_addr, size_t length) {
    int fd = open("/dev/mem", O_RDWR | O_SYNC);
    if (fd < 0) {
        // Probably /dev/mem is restricted (CONFIG_STRICT_DEVMEM)
        return NULL;
    }
    
    // Map physical memory into our address space
    void* mapped = mmap(NULL, 
                        length, 
                        PROT_READ | PROT_WRITE, 
                        MAP_SHARED, 
                        fd, 
                        phys_addr);
    
    close(fd);
    return (mapped == MAP_FAILED) ? NULL : mapped;
}
 
/* ============================================
 * METHOD 3: /proc/[pid]/pagemap (User Space, Root)
 * ============================================
 * Read-only view of virtual-to-physical mappings.
 * Used for debugging and performance analysis.
 */
 
#include <stdio.h>
#include <stdint.h>
 
uint64_t virtual_to_physical(void* virtual_addr) {
    // Parse /proc/self/pagemap to find physical page frame number
    FILE* pagemap = fopen("/proc/self/pagemap", "rb");
    if (!pagemap) return 0;
    
    uint64_t vpn = (uint64_t)virtual_addr / 4096;  // Virtual page number
    uint64_t offset = vpn * sizeof(uint64_t);
    
    fseek(pagemap, offset, SEEK_SET);
    
    uint64_t entry;
    if (fread(&entry, sizeof(entry), 1, pagemap) != 1) {
        fclose(pagemap);
        return 0;
    }
    fclose(pagemap);
    
    // Check if page is present
    if (!(entry & (1ULL << 63))) {
        return 0;  // Page not in memory
    }
    
    // Extract page frame number (bits 0-54)
    uint64_t pfn = entry & ((1ULL << 55) - 1);
    
    // Physical address = PFN * page_size + offset_within_page
    return (pfn * 4096) + ((uint64_t)virtual_addr % 4096);
}
 
/* ============================================
 * METHOD 4: MMIO Mapping (Kernel Device Drivers)
 * ============================================
 * Map device registers (not RAM) into kernel address space
 */
 
// In a kernel driver:
void __iomem* map_device_registers(uint64_t phys_addr, size_t size) {
    // ioremap creates a mapping to memory-mapped I/O region
    // The returned pointer looks like a normal pointer but:
    // - May be uncached (no caching of device registers!)
    // - Accesses have ordering guarantees appropriate for I/O
    // return ioremap(phys_addr, size);
    return NULL;  // Placeholder
}
 
/*
 * Security Note:
 * 
 * Direct physical memory access bypasses all protections:
 * - Process isolation
 * - Kernel/user separation
 * - Memory protection
 * 
 * That's why /dev/mem is restricted, and only the kernel
 * should normally access physical addresses directly.
 */

When Physical Addresses Matter

Physical addresses become visible in specific contexts: device driver development (DMA setup), performance analysis (cache behavior, NUMA effects), virtualization (guest-to-host translation), and memory forensics. In most application programming, physical addresses are correctly abstracted away. Knowing when to think physically vs. logically is a key systems programming skill.

Summary: Physical Address Space

We've explored the physical address space—the actual hardware memory that underlies all computing. This understanding is essential for grasping why address translation, memory protection, and virtual memory exist.

Key Takeaways

•Physical addresses identify actual hardware locations—the real memory cells where data resides, accessed through the memory bus.
•Physical address space is finite and shared—unlike logical spaces, there's one physical address space for the entire system, with size determined by installed hardware.
•Physical memory has complex internal organization—channels, ranks, banks, rows, and columns affect performance in ways invisible to logical addressing.
•The physical address space is fragmented—with holes for MMIO devices, legacy regions, and hardware reservations that the OS must navigate.
•Physical memory management is zone-based—different address ranges have different capabilities (DMA, 32-bit addressability) requiring careful allocation policies.
•Challenges include fragmentation, NUMA, and zone constraints—making physical allocation far more complex than simple logical allocation.
•Direct physical access is restricted—only the kernel normally touches physical addresses, with user space accessing memory through the logical abstraction.

What's Next:

We now understand both sides of the addressing duality: logical (what programs see) and physical (what hardware provides). The next page explores address translation—the mechanism that bridges these two worlds, mapping every logical address to a physical location in real time.

Page Complete

You now understand physical address space as the finite, shared hardware resource underlying all memory operations. This knowledge prepares you to understand address translation: how the MMU and page tables convert the logical addresses processes use into the physical addresses that access actual RAM. The bridge between these two worlds is the heart of memory management.

Physical Address Space

Where Data Actually Lives

What You Will Learn

Definition of Physical Address

Formally:

A physical address is an identifier that specifies an exact location in the physical memory hardware, used by the memory controller to select specific memory cells for read or write operations.

The collection of all valid physical addresses in a system constitutes the physical address space.

Key Properties of Physical Addresses

•Hardware-Determined: Physical address space size is determined by the memory controller and system bus width, not by the CPU's logical addressing capabilities. A system may have 48-bit logical addresses but only 36-bit physical addresses.
•Finite and Bounded: Unlike potentially vast logical address spaces, physical address space is limited by installed RAM and configured memory-mapped regions. You cannot have more physical addresses than there is hardware to back them.
•Shared Resource: A single physical address space is shared by all processes, the operating system kernel, and memory-mapped devices. This sharing necessitates protection mechanisms.
•Uniform Addressing: Each physical address refers to a single byte (on byte-addressable systems). Address 0x1000 refers to the same memory cell regardless of which process's code causes it to be accessed.
•Direct Hardware Access: Physical addresses are what appear on the memory bus. They are the 'real' addresses that memory chips respond to.

Physical vs. Logical Address Space Sizes:

A key insight is that physical and logical address spaces are sized independently:

System Type	Logical Address Bits	Physical Address Bits	Logical Space	Physical Space (Typical)
32-bit x86	32 bits	32-36 bits	4 GB	512 MB - 4 GB
x86-64 (standard)	48 bits	46-52 bits	256 TB	8 GB - 1 TB
x86-64 (5-level)	57 bits	52 bits	128 PB	Up to 4 PB
ARM64	48/52 bits	48-52 bits	256 TB+	Varies

Notice that logical space typically exceeds physical space by orders of magnitude—this is the foundation of virtual memory.

Physical Address ≠ DRAM Address

Physical Memory Organization

Physical memory in a modern system is not a simple linear array of bytes. It's a complex hierarchical system with multiple components, each affecting how physical addresses map to actual storage.

Converting Mermaid diagram...

DRAM Internal Structure:

Within each DIMM (Dual In-line Memory Module), memory is organized hierarchically:

Ranks: One or two independent sets of DRAM chips that share addressing but can operate semi-independently.
Banks: Each rank contains multiple banks (typically 8-16 in DDR4/DDR5). Banks can be accessed in parallel to improve throughput.
Rows (Pages): Each bank contains thousands of rows. Accessing a different row requires a precharge/activate cycle—the primary source of DRAM latency.
Columns: Within an active row, different columns can be accessed relatively quickly.

The memory controller translates physical addresses into the specific {channel, rank, bank, row, column} combination.

physical_address_decoding.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
/*
 * Physical Address to DRAM Location Mapping
 * 
 * This is a simplified example of how a physical address might be
 * decoded into DRAM addressing signals. Real implementations vary
 * based on memory controller design and interleaving policies.
 */
 
#include <stdint.h>
#include <stdio.h>
 
// Example: 16 GB system, 2 channels, 2 ranks per channel, 8 banks per rank
// DDR4-style addressing
 
typedef struct {
    uint8_t channel;    // 0-1 (1 bit)
    uint8_t rank;       // 0-1 (1 bit)
    uint8_t bank_group; // 0-3 (2 bits)
    uint8_t bank;       // 0-3 (2 bits within group)
    uint16_t row;       // 0-32767 (15 bits)
    uint8_t column;     // 0-1023 (10 bits, but lower 3 for burst)
} DRAMAddress;
 
/*
 * Example physical address layout (48-bit physical address):
 * 
 * Bits:  47    ...    17 | 16-15 | 14-13 | 12-11 | 10-7 | 6     | 5-3  | 2-0
 * Field: Row (15+ bits)  | Bank  | BankGr| Column| Col  |Channel| Rank | Byte
 * 
 * Note: Real systems interleave differently for performance.
 * Channel bits might be in lower positions for better parallelism.
 */
 
DRAMAddress decode_physical_address(uint64_t phys_addr) {
    DRAMAddress dram;
    
    // Byte offset within cache line (typically ignored by DRAM, handled by cache)
    // uint8_t byte_offset = phys_addr & 0x7;
    
    // Column address (cache line granularity)
    dram.column = (phys_addr >> 6) & 0x3FF;  // Bits 6-15 (simplified)
    
    // Channel interleaving (often at cache line level for parallelism)
    dram.channel = (phys_addr >> 6) & 0x1;   // Bit 6 (example)
    
    // Rank selection
    dram.rank = (phys_addr >> 16) & 0x1;     // Bit 16 (example)
    
    // Bank group and bank (DDR4/DDR5)
    dram.bank_group = (phys_addr >> 17) & 0x3;
    dram.bank = (phys_addr >> 19) & 0x3;
    
    // Row address (high-order bits)
    dram.row = (phys_addr >> 21) & 0x7FFF;   // Bits 21-35
    
    return dram;
}
 
void print_dram_address(uint64_t phys_addr) {
    DRAMAddress dram = decode_physical_address(phys_addr);
    
    printf("Physical Address: 0x%012llx\n", (unsigned long long)phys_addr);
    printf("  Channel:    %d\n", dram.channel);
    printf("  Rank:       %d\n", dram.rank);
    printf("  Bank Group: %d\n", dram.bank_group);
    printf("  Bank:       %d\n", dram.bank);
    printf("  Row:        %d\n", dram.row);
    printf("  Column:     %d\n", dram.column);
}
 
/*
 * Performance Implications:
 * 
 * - Accesses to the same row (row buffer hit): ~10-15 ns
 * - Accesses to different row in same bank (row buffer miss): ~50-60 ns
 * - Accesses to different banks (parallel): Lower latency
 * 
 * Memory allocators and OS page placement can optimize for these patterns.
 */

NUMA: Non-Uniform Memory Access

Physical Address Space Layout

The physical address space is not entirely dedicated to RAM. Various regions are reserved for specific purposes, creating a complex map that the operating system must navigate.

physical_address_map.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Typical x86-64 Physical Address Space Layout:
 
Physical Address        | Size          | Description
------------------------|---------------|----------------------------------
0x00000000 - 0x0009FFFF | 640 KB        | Conventional Memory (legacy)
0x000A0000 - 0x000FFFFF | 384 KB        | Legacy ROM, VGA memory (hole)
0x00100000 - 0x7FFFFFFF | ~2 GB         | Main RAM (below 4 GB mark)
0x80000000 - 0xDFFFFFFF | ~1.5 GB       | RAM continues (varies)
0xE0000000 - 0xFFFFFFFF | ~512 MB       | Memory-mapped PCI devices, APIC,
                        |               | firmware (BIOS/UEFI), ROM
------------------- 4 GB boundary ("MMIO hole") -------------------
0x100000000 - 0x????????| Variable      | RAM above 4 GB (main memory)
0x???????? - High       | Variable      | Additional RAM, PCIe config space
 
Key Points:
- The region 0xE0000000-0xFFFFFFFF is NOT RAM; accessing these
  addresses reaches hardware devices, not DRAM
  
- The "MMIO hole" means a 32-bit system with 4 GB RAM doesn't
  actually have 4 GB usable—some is lost to device mappings
  
- Memory above 4 GB requires 64-bit addressing (PAE or x86-64)
 
- The kernel receives this map from BIOS/UEFI (E820 map on x86)
  and must work around the gaps

Physical Address Space Regions
Region Type	Purpose	Accessible By	Characteristics
Conventional RAM	Program code and data	CPU via memory controller	Fast, volatile, cached
Memory-Mapped I/O	Device registers	CPU to device controllers	Not cached, side effects on access
Firmware ROM	BIOS/UEFI code	CPU at boot, read-only	Non-volatile, may be shadowed
Reserved	Hardware-specific	Varies	Cannot be used for RAM
PCI Config Space	Device configuration	OS via special access	Small (256B/device), structured
APIC Registers	Interrupt controller	Kernel for IRQ management	Memory-mapped at fixed address

The E820 Memory Map:

At boot time, the BIOS or UEFI firmware provides the operating system with a memory map (the E820 map on x86 systems) describing which physical address ranges are:

Usable RAM: Can be used for normal memory allocation
Reserved: Hardware-reserved, do not use
ACPI Data: Firmware tables the OS needs to preserve
ACPI NVS: Non-volatile storage used by firmware
Unusable: Defective or otherwise unavailable

The operating system must respect these designations. Using reserved memory can cause hardware malfunctions or crashes.

read_physical_memory_map.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
/*
 * Reading Physical Memory Information on Linux
 * 
 * The kernel exposes physical memory information through /proc and /sys
 */
 
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
 
void print_iomem() {
    // /proc/iomem shows physical address space allocation
    printf("=== Physical Address Space Map (/proc/iomem) ===\n\n");
    
    FILE *f = fopen("/proc/iomem", "r");
    if (!f) {
        perror("Cannot open /proc/iomem (requires root)");
        return;
    }
    
    char line[256];
    int count = 0;
    while (fgets(line, sizeof(line), f) && count < 30) {
        printf("%s", line);
        count++;
    }
    if (count == 30) printf("... (truncated)\n");
    fclose(f);
}
 
void print_memory_info() {
    printf("\n=== Memory Statistics (/proc/meminfo) ===\n\n");
    
    FILE *f = fopen("/proc/meminfo", "r");
    if (!f) {
        perror("Cannot open /proc/meminfo");
        return;
    }
    
    char line[256];
    int count = 0;
    while (fgets(line, sizeof(line), f) && count < 10) {
        printf("%s", line);
        count++;
    }
    fclose(f);
}
 
/*
 * Sample /proc/iomem output:
 * 
 * 00000000-00000fff : Reserved
 * 00001000-0009fbff : System RAM
 * 0009fc00-0009ffff : Reserved
 * 000a0000-000bffff : PCI Bus 0000:00
 * 000c0000-000c7fff : Video ROM
 * 000e0000-000fffff : Reserved
 *   000f0000-000fffff : System ROM
 * 00100000-7bfeffff : System RAM
 *   01000000-0199b443 : Kernel code
 *   0199b444-01eb1b3f : Kernel data
 *   02068000-020fafff : Kernel bss
 * 7bf00000-7bffffff : RAM buffer
 * 80000000-90ffffff : PCI Bus 0000:00
 * e0000000-efffffff : PCI MMCONFIG 0
 * f0000000-f7ffffff : PCI Bus 0000:00
 *   f0000000-f7ffffff : 0000:00:02.0
 * fed00000-fed003ff : HPET 0
 * fee00000-fee00fff : Local APIC
 * 100000000-27fffffff : System RAM
 * 
 * Notice: System RAM regions are scattered, with holes for devices
 */
 
int main() {
    print_iomem();
    print_memory_info();
    return 0;
}

The 640 KB Barrier Legacy

Physical Memory as a Shared Resource

Unlike logical address spaces—where each process has its own private space—physical address space is a single, shared resource. This sharing is both a challenge and an opportunity.

Challenges of Sharing

•Protection Required: Without hardware protection, any process could read/write any physical location—including kernel memory or other processes' data.
•Contention for Resources: Multiple processes compete for finite physical memory. The OS must fairly and efficiently allocate this scarce resource.
•Fragmentation Risk: As processes allocate and free memory, physical address space can become fragmented, making large contiguous allocations impossible.
•Coordination Complexity: The OS must track every physical page: who owns it, what's in it, whether it can be reclaimed.

Opportunities of Sharing

•Memory Deduplication: Identical physical pages (e.g., common libraries) need only one physical copy, mapped into multiple address spaces.
•Efficient Communication: Shared memory regions in physical space enable zero-copy inter-process communication.
•Copy-on-Write: fork() can share physical pages between parent and child until a write occurs, saving both memory and time.
•Resource Overcommitment: Virtual memory allows promising more logical memory than physically exists, with physical pages allocated on demand.

Physical Page as the Unit of Management:

The frame size represents a tradeoff:

Smaller frames: Less internal fragmentation, finer granularity, but more metadata overhead and larger page tables
Larger frames: Less overhead, better TLB coverage, but more internal fragmentation and coarser protection granularity

4 KB has been the standard page size since the 1970s and remains dominant, though the performance benefits of huge pages are increasingly recognized.

physical_frame_management.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
/*
 * Conceptual Physical Frame Management
 * 
 * The OS maintains data structures to track every physical frame:
 * - Who owns it (kernel, which process, or free)
 * - Reference count (how many mappings point to it)
 * - Status flags (dirty, accessed, locked, etc.)
 */
 
#include <stdint.h>
#include <stdbool.h>
 
#define PAGE_SIZE 4096
#define MAX_PHYSICAL_PAGES (16ULL * 1024 * 1024 * 1024 / PAGE_SIZE)  // 16 GB
 
typedef enum {
    FRAME_FREE,           // Available for allocation
    FRAME_KERNEL,         // Used by kernel code/data
    FRAME_USER,           // Allocated to user process
    FRAME_RESERVED,       // Hardware reserved, unusable
    FRAME_PAGE_TABLE,     // Used for page tables
    FRAME_DMA,            // Reserved for device DMA
} FrameState;
 
typedef struct {
    FrameState state;
    uint32_t owner_pid;       // Process ID (if USER)
    uint16_t ref_count;       // Number of mappings to this frame
    uint16_t flags;           // Dirty, accessed, pinned, etc.
} FrameDescriptor;
 
// The page frame database - one entry per physical frame
FrameDescriptor frame_db[MAX_PHYSICAL_PAGES];
 
// Free list for O(1) allocation
uint64_t* free_list;
uint64_t free_count;
 
/*
 * Allocate a physical frame
 * Returns the physical frame number (PFN), or -1 if OOM
 */
int64_t allocate_frame(uint32_t pid) {
    if (free_count == 0) {
        // Out of physical memory!
        // Options: reclaim from caches, swap, or OOM kill
        return -1;
    }
    
    // Pop from free list
    uint64_t pfn = free_list[--free_count];
    
    // Initialize frame descriptor
    frame_db[pfn].state = FRAME_USER;
    frame_db[pfn].owner_pid = pid;
    frame_db[pfn].ref_count = 1;
    frame_db[pfn].flags = 0;
    
    // Zero the frame for security (don't leak data between processes)
    // memset(physical_to_virtual(pfn * PAGE_SIZE), 0, PAGE_SIZE);
    
    return pfn;
}
 
/*
 * Free a physical frame back to the pool
 */
void free_frame(uint64_t pfn) {
    if (--frame_db[pfn].ref_count == 0) {
        frame_db[pfn].state = FRAME_FREE;
        frame_db[pfn].owner_pid = 0;
        free_list[free_count++] = pfn;
    }
    // If ref_count > 0, the frame is still mapped elsewhere (shared memory)
}
 
/*
 * Physical Frame Number (PFN) to Physical Address conversion
 * 
 * Physical Address = PFN × PAGE_SIZE
 * PFN = Physical Address / PAGE_SIZE
 */
uint64_t pfn_to_physical(uint64_t pfn) {
    return pfn * PAGE_SIZE;
}
 
uint64_t physical_to_pfn(uint64_t physical) {
    return physical / PAGE_SIZE;
}
 
/*
 * In Linux, this data structure is 'struct page' (the page frame database)
 * and is one of the most critical kernel data structures.
 * 
 * With 16 GB RAM and 4 KB pages: 4 million frames to track
 * At ~64 bytes per 'struct page': ~256 MB just for frame metadata!
 */

The Frame Table Overhead

Physical vs. Logical: The Critical Comparison

Understanding the fundamental differences between physical and logical address spaces is essential for grasping how memory management works.

Physical vs. Logical Address Space Comparison
Aspect	Logical Address Space	Physical Address Space
Origin	Generated by CPU during execution	Actual hardware memory locations
Per-Process?	Yes, each process has its own	No, single shared space system-wide
Size	Determined by CPU architecture (32/64-bit)	Determined by installed RAM + MMIO
Continuity	Appears contiguous to process	May be fragmented with holes
Protection	Achieved through translation tables	Hardware prevents unauthorized access
Visibility	Seen by running programs	Hidden from applications
Lifespan	Exists while process runs	Persists across process lifetimes
Address 0	Valid (often first page of text segment)	Typically low memory region (not usable by apps)
Sparse Usage	Common—large unmapped gaps	Every byte is accounted for
Translation Required	Yes, for every memory access	No, this IS the final address

The Binding Problem:

Compile Time: Absolute addresses are hardcoded. Only works if physical load address is known in advance. Obsolete for general-purpose computing.
Load Time: Addresses are adjusted when the program is loaded into memory. The program must stay at that physical location for its lifetime.
Execution Time: Addresses are translated dynamically during execution. This is the modern approach, enabling relocation, sharing, and virtual memory.

Modern systems use execution-time binding exclusively, with hardware (the MMU) performing translation on every memory access.

Converting Mermaid diagram...

Why This Matters for Programmers

Physical Memory Allocation Challenges

Managing physical memory presents unique challenges that don't exist in the logical address realm. These challenges drive many design decisions in operating system memory subsystems.

Physical Memory Allocation Challenges

•External Fragmentation: Over time, free physical memory becomes scattered in small, non-contiguous chunks. While logical address space can be contiguous (pages map to anywhere), some allocations (DMA buffers, huge pages) need contiguous physical memory.
•Zone Constraints: Some physical addresses have special requirements. The first 16 MB may be needed for legacy ISA DMA. The first 4 GB is addressable by 32-bit devices. Physical memory is divided into zones with different allocation rules.
•NUMA Locality: Allocating physical memory from the wrong NUMA node can double memory access latency. The allocator must track which CPUs are 'close' to which physical memory.
•Cache Coloring: Physical address bits affect cache placement. Poor allocation can cause cache conflicts. Some high-performance systems explicitly manage physical allocation for cache efficiency.
•Memory Pressure: When physical memory runs low, the OS must reclaim frames from less important uses—caches, inactive processes, or swapping to disk. These decisions affect system responsiveness.
•Hot-Add/Remove: Modern servers support adding or removing RAM while running. The OS must handle physical address space changing dynamically.

Zone-Based Allocation:

Linux divides physical memory into zones to handle these constraints:

Zone	Physical Address Range	Purpose
ZONE_DMA	0-16 MB	Legacy ISA DMA devices
ZONE_DMA32	0-4 GB	32-bit device DMA
ZONE_NORMAL	Above 4 GB	General purpose allocation
ZONE_MOVABLE	High memory	Can be migrated for memory hotplug
ZONE_DEVICE	Varies	Persistent memory, device memory

When allocating physical memory, the kernel prefers ZONE_NORMAL but falls back to other zones if necessary.

zone_allocation.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
/*
 * Physical Memory Zones (Linux-style)
 * 
 * Different physical address ranges have different capabilities.
 * The allocator must respect these constraints.
 */
 
#include <stdint.h>
#include <stdbool.h>
 
typedef enum {
    ZONE_DMA,       // 0 - 16 MB: Legacy ISA DMA
    ZONE_DMA32,     // 0 - 4 GB: 32-bit device addressing
    ZONE_NORMAL,    // All memory: General purpose
    ZONE_MOVABLE,   // Hot-pluggable memory
    ZONE_COUNT
} MemoryZone;
 
typedef struct {
    uint64_t start_pfn;      // First page frame number in zone
    uint64_t end_pfn;        // Last page frame number in zone
    uint64_t free_pages;     // Count of free pages
    uint64_t min_watermark;  // Minimum free pages before reclaim
    uint64_t high_watermark; // Target free pages after reclaim
    // ... free lists, per-cpu caches, etc.
} Zone;
 
Zone zones[ZONE_COUNT];
 
/*
 * Allocation flags determine which zones can satisfy a request
 */
typedef enum {
    GFP_DMA     = (1 << 0),   // Must be in DMA zone
    GFP_DMA32   = (1 << 1),   // Must be below 4 GB
    GFP_NORMAL  = (1 << 2),   // Any zone is acceptable
    GFP_MOVABLE = (1 << 3),   // Prefer movable zone
    GFP_NOIO    = (1 << 4),   // Don't start I/O to reclaim
    GFP_NOWAIT  = (1 << 5),   // Don't wait, fail if unavailable
} GFPFlags;
 
/*
 * Zone fallback order
 * 
 * When ZONE_NORMAL is empty, we might try DMA32, then DMA.
 * This is a policy decision with performance implications.
 */
MemoryZone zone_fallback[ZONE_COUNT][ZONE_COUNT] = {
    [ZONE_DMA]     = { ZONE_DMA },                           // DMA: no fallback
    [ZONE_DMA32]   = { ZONE_DMA32, ZONE_DMA },               // DMA32: fall to DMA
    [ZONE_NORMAL]  = { ZONE_NORMAL, ZONE_DMA32, ZONE_DMA },  // Normal: fall through
    [ZONE_MOVABLE] = { ZONE_MOVABLE, ZONE_NORMAL, ZONE_DMA32, ZONE_DMA },
};
 
/*
 * Allocate a page from appropriate zone
 */
uint64_t alloc_page(GFPFlags flags) {
    MemoryZone preferred;
    
    // Determine preferred zone from flags
    if (flags & GFP_DMA)
        preferred = ZONE_DMA;
    else if (flags & GFP_DMA32)
        preferred = ZONE_DMA32;
    else if (flags & GFP_MOVABLE)
        preferred = ZONE_MOVABLE;
    else
        preferred = ZONE_NORMAL;
    
    // Try zones in fallback order
    for (int i = 0; i < ZONE_COUNT; i++) {
        MemoryZone zone = zone_fallback[preferred][i];
        if (zones[zone].free_pages > zones[zone].min_watermark) {
            // Allocate from this zone
            zones[zone].free_pages--;
            // return pfn from zone's free list
        }
    }
    
    // All zones exhausted - trigger reclaim or OOM
    // ...
    
    return (uint64_t)-1;  // Allocation failed
}

The Physical Contiguity Problem

Accessing Physical Addresses From Software

Kernel Direct Mapping:

void* physical_to_virtual(uint64_t physical_addr) {
    return (void*)(physical_addr + PAGE_OFFSET);
}

This direct mapping exists only in kernel space. User processes cannot access these addresses—the page tables don't include mappings for kernel regions in user mode.

physical_access_methods.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
/*
 * Methods for Accessing Physical Memory
 * 
 * Different contexts require different approaches
 */
 
/* ============================================
 * METHOD 1: Kernel Direct Mapping (Linux Kernel)
 * ============================================
 * The kernel has all physical memory mapped into its address space.
 * Converting is trivial:
 */
 
// In Linux kernel code:
#define PAGE_OFFSET 0xffff888000000000UL
 
void* phys_to_virt(uint64_t phys_addr) {
    return (void*)(phys_addr + PAGE_OFFSET);
}
 
uint64_t virt_to_phys(void* virt_addr) {
    return (uint64_t)virt_addr - PAGE_OFFSET;
}
 
/* ============================================
 * METHOD 2: /dev/mem (User Space, Root Only)
 * ============================================
 * Character device providing raw physical memory access.
 * Extremely dangerous; disabled by default on most systems.
 */
 
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
 
void* map_physical_memory(uint64_t phys_addr, size_t length) {
    int fd = open("/dev/mem", O_RDWR | O_SYNC);
    if (fd < 0) {
        // Probably /dev/mem is restricted (CONFIG_STRICT_DEVMEM)
        return NULL;
    }
    
    // Map physical memory into our address space
    void* mapped = mmap(NULL, 
                        length, 
                        PROT_READ | PROT_WRITE, 
                        MAP_SHARED, 
                        fd, 
                        phys_addr);
    
    close(fd);
    return (mapped == MAP_FAILED) ? NULL : mapped;
}
 
/* ============================================
 * METHOD 3: /proc/[pid]/pagemap (User Space, Root)
 * ============================================
 * Read-only view of virtual-to-physical mappings.
 * Used for debugging and performance analysis.
 */
 
#include <stdio.h>
#include <stdint.h>
 
uint64_t virtual_to_physical(void* virtual_addr) {
    // Parse /proc/self/pagemap to find physical page frame number
    FILE* pagemap = fopen("/proc/self/pagemap", "rb");
    if (!pagemap) return 0;
    
    uint64_t vpn = (uint64_t)virtual_addr / 4096;  // Virtual page number
    uint64_t offset = vpn * sizeof(uint64_t);
    
    fseek(pagemap, offset, SEEK_SET);
    
    uint64_t entry;
    if (fread(&entry, sizeof(entry), 1, pagemap) != 1) {
        fclose(pagemap);
        return 0;
    }
    fclose(pagemap);
    
    // Check if page is present
    if (!(entry & (1ULL << 63))) {
        return 0;  // Page not in memory
    }
    
    // Extract page frame number (bits 0-54)
    uint64_t pfn = entry & ((1ULL << 55) - 1);
    
    // Physical address = PFN * page_size + offset_within_page
    return (pfn * 4096) + ((uint64_t)virtual_addr % 4096);
}
 
/* ============================================
 * METHOD 4: MMIO Mapping (Kernel Device Drivers)
 * ============================================
 * Map device registers (not RAM) into kernel address space
 */
 
// In a kernel driver:
void __iomem* map_device_registers(uint64_t phys_addr, size_t size) {
    // ioremap creates a mapping to memory-mapped I/O region
    // The returned pointer looks like a normal pointer but:
    // - May be uncached (no caching of device registers!)
    // - Accesses have ordering guarantees appropriate for I/O
    // return ioremap(phys_addr, size);
    return NULL;  // Placeholder
}
 
/*
 * Security Note:
 * 
 * Direct physical memory access bypasses all protections:
 * - Process isolation
 * - Kernel/user separation
 * - Memory protection
 * 
 * That's why /dev/mem is restricted, and only the kernel
 * should normally access physical addresses directly.
 */

When Physical Addresses Matter

Summary: Physical Address Space

Key Takeaways

•Physical addresses identify actual hardware locations—the real memory cells where data resides, accessed through the memory bus.
•Physical address space is finite and shared—unlike logical spaces, there's one physical address space for the entire system, with size determined by installed hardware.
•Physical memory has complex internal organization—channels, ranks, banks, rows, and columns affect performance in ways invisible to logical addressing.
•The physical address space is fragmented—with holes for MMIO devices, legacy regions, and hardware reservations that the OS must navigate.
•Physical memory management is zone-based—different address ranges have different capabilities (DMA, 32-bit addressability) requiring careful allocation policies.
•Challenges include fragmentation, NUMA, and zone constraints—making physical allocation far more complex than simple logical allocation.
•Direct physical access is restricted—only the kernel normally touches physical addresses, with user space accessing memory through the logical abstraction.

What's Next:

Page Complete