Buffer Overflows — Memory Debugging:… | OneNoughtOne

0/107

Operating SystemsMemory Debugging

Memory Debugging: Detecting and Resolving Memory Errors

LevelAdvanced

Duration120 mins

TopicMemory Debugging

2 / 5

Buffer Overflows

When Writes Escape Their Bounds

On November 2, 1988, a graduate student at Cornell University unleashed what would become known as the Morris Worm—the first major computer worm to spread across the nascent Internet. Among its attack vectors was a buffer overflow in the Unix fingerd daemon. By sending a carefully crafted 536-character string where only 512 characters were expected, Robert Morris's worm could overwrite the stack, hijack program control, and spread to the next machine. The Internet ground to a halt; an estimated 6,000 computers (10% of the entire Internet at the time) were infected. Three decades later, buffer overflows remain among the most severe vulnerability classes. Despite countless mitigations—stack canaries, ASLR, DEP, CFI—attackers continue to find ways to exploit programs that write beyond buffer boundaries. The 2021 Pegasus spyware used a zero-click buffer overflow in Apple's iMessage to compromise iPhones without any user interaction. This page provides a comprehensive exploration of buffer overflows: how they occur, why they're exploitable, and how modern systems defend against them. Understanding buffer overflows is essential for anyone working with systems software or security.

What You Will Learn

By the end of this page, you will understand: (1) The memory layout that makes buffer overflows dangerous, (2) Stack-based overflow mechanics including return address corruption, (3) Heap-based overflow mechanics and metadata corruption, (4) Integer overflows as buffer overflow enablers, (5) Modern mitigation techniques and their limitations, and (6) Secure coding practices that prevent overflows.

Memory Layout and Why Overflows Are Dangerous

To understand buffer overflows, we must first understand how programs lay out data in memory. The danger of overflows stems from the proximity of buffers to critical control data. Process Memory Layout (x86-64):

memory_layout.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
High Addresses (e.g., 0x7FFFFFFFFFFF)
┌─────────────────────────────────────┐
│           Kernel Space              │  ← Not accessible from user mode
├─────────────────────────────────────┤
│             Stack                   │  ← Grows downward (toward lower addresses)
│   ┌─────────────────────────────┐   │
│   │ Return address    [8 bytes] │   │  ← CRITICAL: If overwritten, controls execution
│   │ Saved base pointer [8 bytes]│   │
│   │ Local variables             │   │  ← Buffers typically here
│   │ (char buffer[64], int x...)│   │
│   └─────────────────────────────┘   │
│              ↓                      │
│         (grows down)                │
│                                     │
├─────────────────────────────────────┤
│                                     │
│          Memory-Mapped              │  ← Shared libraries, mmap() regions
│            Region                   │
│                                     │
├─────────────────────────────────────┤
│              ↑                      │
│         (grows up)                  │
│   ┌─────────────────────────────┐   │
│   │ Heap                        │   │
│   │ malloc'd data, metadata     │   │  ← Another overflow target
│   └─────────────────────────────┘   │
│             Heap                    │
├─────────────────────────────────────┤
│      Uninitialized Data (BSS)       │  ← Global/static uninitialized variables
├─────────────────────────────────────┤
│      Initialized Data (.data)       │  ← Global/static initialized variables
├─────────────────────────────────────┤
│         Text (Code) Segment         │  ← Executable instructions (read-only)
└─────────────────────────────────────┘
Low Addresses (e.g., 0x400000)

The Critical Insight: On the stack, local variables (including buffers) are adjacent to control information (return addresses, saved registers) . When you write past the end of a buffer, you don't hit empty space—you overwrite whatever happens to be next in memory. If a buffer is near the return address, overflowing it can redirect execution. Why Bounds Aren't Checked: C and C++ prioritize performance over safety. Array accesses compile to simple pointer arithmetic with no bounds checking:

bounds_checking.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// What you write:
char buffer[64];
buffer[70] = 'X';  // Clearly out of bounds
 
// What the compiler generates (x86-64):
// mov    BYTE PTR [rbp-64+70], 'X'
// 
// No check, no error - just writes to whatever is at that address
// In this case: 6 bytes past buffer, possibly into saved registers
 
// The philosophy:
// "Programmers are responsible for correct code"
// "Runtime checks are too expensive for systems programming"
// "Trust the programmer"
 
// The consequence:
// Millions of security vulnerabilities over 50 years

Buffer Overflow Definition: A buffer overflow occurs when a program writes data beyond the allocated bounds of a buffer, corrupting adjacent memory that may contain other data, control structures, or executable code. Buffer overflows are classified by location:

Buffer Overflow Classification
Type	Location	Typical Target	Exploitation Difficulty
Stack Buffer Overflow	Stack (local variables)	Return address, saved registers	Classic, well-understood
Heap Buffer Overflow	Heap (malloc'd memory)	Heap metadata, adjacent objects	Requires heap layout knowledge
BSS/Data Overflow	Global/static data	Function pointers, configuration data	Program-specific
Off-by-One	Any (single byte overwrite)	Terminators, pointers	Subtle but exploitable

Why This Still Matters

Despite being one of the oldest vulnerability classes, buffer overflows continue to be discovered in modern software. In 2023 alone, critical buffer overflows were found in Chrome, Firefox, Windows, Linux, iOS, and Android. Any code handling untrusted input in C/C++ is a potential overflow waiting to be discovered.

Stack-Based Buffer Overflows: The Classic Attack

Stack-based buffer overflows are the 'classic' form, extensively documented since the 1996 paper "Smashing the Stack for Fun and Profit" by Aleph One. Understanding them is foundational to understanding all overflow attacks. Stack Frame Structure: When a function is called, the processor creates a stack frame containing:

stack_frame.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Function call: caller() calls vulnerable()
 
High Addresses
┌──────────────────────────────────────┐
│   caller()'s stack frame             │
├──────────────────────────────────────┤
│   Arguments to vulnerable()          │  ← Pushed by caller
├──────────────────────────────────────┤
│   Return Address [8 bytes]           │  ← Pushed by CALL instruction
│   (where to resume after return)     │     TARGET #1 for exploitation
├──────────────────────────────────────┤
│   Saved RBP [8 bytes]                │  ← Pushed by function prologue  
│   (caller's base pointer)            │     TARGET #2 for exploitation
├──────────────────────────────────────┤  ← Current RBP points here
│   Local variable: int x              │  [rbp - 4]
├──────────────────────────────────────┤
│   Local variable: char buffer[64]    │  [rbp - 68] to [rbp - 5]
│   buffer[0] ... buffer[63]           │     OVERFLOW STARTS HERE
├──────────────────────────────────────┤  ← Current RSP (top of stack)
Low Addresses
 
Overflow Direction (if writing past buffer[63]):
→ buffer[64] corrupts 'int x'
→ buffer[68] starts corrupting saved RBP
→ buffer[76] starts corrupting return address
→ return address now points to attacker-controlled location

Vulnerable Code Example:

vulnerable_function.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
// Classic vulnerable function
#include <stdio.h>
#include <string.h>
 
void vulnerable(char* user_input) {
    char buffer[64];
    
    // DANGEROUS: strcpy doesn't check bounds
    // If user_input > 64 bytes, overflow occurs
    strcpy(buffer, user_input);
    
    printf("You entered: %s
", buffer);
}
 
int main(int argc, char* argv[]) {
    if (argc < 2) {
        printf("Usage: %s <input>
", argv[0]);
        return 1;
    }
    
    vulnerable(argv[1]);
    
    printf("Program completed normally.
");
    return 0;
}
 
/* What happens with 64 'A's:
   buffer = "AAAA...AAAA" (64 As) - fits exactly
   Output: "You entered: AAA..."
   Program completes normally
   
   What happens with 80 'A's:
   buffer = "AAAA...AAAA" (64 As)
   int x gets corrupted (if present)
   saved RBP = 0x4141414141414141 ("AAAAAAAA")
   return address = 0x4141414141414141
   When function returns, jumps to 0x4141414141414141
   SEGFAULT (or worse - exploitation if address is valid)
*/

Exploitation Goals: Overwriting the return address allows the attacker to redirect execution. Common exploitation techniques: 1. Return-to-Shellcode (Classic, now rare): Place executable code ("shellcode") in the buffer Overwrite return address to point at the buffer When function returns, shellcode executes Mitigated by: DEP/NX (non-executable stack) 2. Return-to-libc: Don't inject code; reuse existing code Overwrite return address to point at libc function (e.g., system()) Arrange stack so function receives controlled arguments Mitigated by: ASLR (randomizes libc location) 3. Return-Oriented Programming (ROP): Chain together small code snippets ("gadgets") ending in ret Each gadget performs one small operation Together, they form complete attack logic Mitigated by: CFI, Shadow Stacks (partially)

The Evolution of Exploitation

Each mitigation spawned new techniques. DEP stopped shellcode \to attackers used return-to-libc. ASLR randomized addresses \to attackers found information leaks. Stack canaries detected overwrites \to attackers found ways to leak or bypass them. This cat-and-mouse game continues today.

Heap-Based Buffer Overflows: Corrupting the Allocator

Heap overflows target dynamically allocated memory. Unlike stack overflows (which target return addresses), heap overflows typically corrupt heap metadata or adjacent heap objects. Heap Memory Layout:

heap_layout.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Heap Memory (simplified view of two adjacent allocations)
 
┌───────────────────────────────────────────────────────────┐
│   Chunk A Metadata (e.g., size, prev_size, flags)        │  ← Allocator's bookkeeping
├───────────────────────────────────────────────────────────┤
│   Chunk A User Data                                       │  ← What malloc() returns
│   buffer = malloc(64);                                    │
│   [         64 bytes of user data                    ]    │
│                                                           │  ← OVERFLOW GOES HERE
├───────────────────────────────────────────────────────────┤
│   Chunk B Metadata (e.g., size, prev_size, flags)        │  ← Corrupted by overflow!
├───────────────────────────────────────────────────────────┤
│   Chunk B User Data                                       │
│   victim = malloc(128);                                   │
│   [This might contain function pointers, objects...]     │  ← Also corruptible
└───────────────────────────────────────────────────────────┘
 
Overflow from Chunk A corrupts:
1. Chunk B's metadata → Confuses allocator → Arbitrary write primitive
2. Chunk B's user data → Corrupts application data → Depends on what's there

Heap Metadata Attacks: Heap allocators (like glibc's ptmalloc) use metadata chunks to manage free lists. Corrupting this metadata can trick the allocator into writing attacker-controlled data to attacker-controlled locations:

heap_exploitation.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
// Simplified heap overflow exploitation concepts
 
// Heap chunk structure (glibc ptmalloc2 simplified)
// When chunk is FREE:
struct malloc_chunk {
    size_t prev_size;  // Size of previous chunk (if free)  
    size_t size;       // Size of this chunk + flags in low bits
    
    // These only exist in FREE chunks:
    struct malloc_chunk* fd;  // Forward pointer in free list
    struct malloc_chunk* bk;  // Backward pointer in free list
};
 
// CLASSIC ATTACK: Unsafe unlink
// 
// When a free chunk is removed from the free list, allocator does:
//     fd->bk = bk;    // Write bk to location fd+offset
//     bk->fd = fd;    // Write fd to location bk+offset
//
// If attacker controls fd and bk (via overflow), they get:
//     Write arbitrary value (bk) to arbitrary address (fd+offset)
//     This is called "write-what-where" or "arbitrary write"
//
// Example exploitation:
//     fd = &got_entry - offset  (GOT = Global Offset Table)
//     bk = &shellcode
//     When chunk unlinked: GOT entry points to shellcode
//     Next call to that function → shellcode runs
 
// Modern allocators have many protections against this:
// - Corrupted size checks
// - Safe unlinking (verify fd->bk == p && bk->fd == p)
// - Randomized metadata locations
// - Guard pages between heap regions

Heap Spray Attacks: When precise heap layout control is difficult, attackers may use "heap spraying"—allocating many copies of the same data to increase the probability that a corrupted pointer lands in attacker-controlled memory:

heap_spray.js
JavaScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// Heap spray concept (for understanding, not actual exploitation)
 
// In browser exploits, JavaScript could spray the heap:
var spray = [];
var payload = "PAYLOAD_PLACEHOLDER";  // shellcode goes here
 
// Allocate hundreds of MB filled with our payload  
for (var i = 0; i < 200; i++) {
    // Create 1MB string filled with NOP sleds + shellcode
    var block = "";
    for (var j = 0; j < 1024 * 1024 / payload.length; j++) {
        block += payload;
    }
    spray.push(block);
}
 
// Now the heap looks like:
// [spray block][spray block][spray block]...[spray block]
// 
// If attacker can corrupt a pointer and get execution at
// ANY address in this huge sprayed region, they win.
// The "NOP sled" technique slides execution forward until
// it hits actual shellcode.
 
// Modern mitigations:
// - Heap isolation (different allocators for different object types)
// - Guard pages (unmapped pages between allocations)
// - Memory tagging (ARM MTE)
// - Reduced JavaScript precision for spray detection

Heap Exploitation is Complex

Unlike stack overflows (which have consistent structure), heap exploits require deep understanding of specific allocator implementations. Modern allocators like jemalloc, tcmalloc, and hardened glibc variants include extensive protections. However, the complexity also means bugs in allocator code itself can be catastrophic.

Integer Overflows: The Gateway to Buffer Overflows

Many buffer overflows don't stem from direct write operations exceeding bounds. Instead, they originate from integer overflows that cause incorrect buffer size calculations. Integer overflows are among the most subtle and dangerous vulnerability classes. How Integer Overflow Leads to Buffer Overflow:

integer_overflow.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
// Integer overflow leading to heap buffer overflow
 
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
 
// Vulnerable image processing function
int process_image(unsigned int width, unsigned int height, 
                  unsigned int bytes_per_pixel, 
                  unsigned char* image_data) {
    
    // Calculate buffer size for image
    // BUG: Integer overflow!
    unsigned int size = width * height * bytes_per_pixel;
    
    // Example attack values:
    // width = 0x10000 (65536)
    // height = 0x10000 (65536)  
    // bytes_per_pixel = 4
    // 
    // Expected size: 65536 * 65536 * 4 = 17,179,869,184 (17 GB)
    // Actual size after 32-bit overflow: 0 (or small value)
    
    // Allocate undersized buffer
    unsigned char* buffer = malloc(size);  // Allocates 0 or tiny amount
    if (!buffer) return -1;
    
    // Copy image data into undersized buffer
    // This writes GIGABYTES past the allocated buffer
    memcpy(buffer, image_data, width * height * bytes_per_pixel);
    
    // Massive heap overflow!
    
    free(buffer);
    return 0;
}
 
// FIXED VERSION: Check for overflow before allocation
int process_image_safe(size_t width, size_t height, 
                       size_t bytes_per_pixel,
                       unsigned char* image_data) {
    
    // Check for multiplication overflow
    size_t size;
    
    // Method 1: Manual check
    if (width > SIZE_MAX / height) {
        return -1;  // Overflow would occur
    }
    size_t area = width * height;
    
    if (area > SIZE_MAX / bytes_per_pixel) {
        return -1;  // Overflow would occur
    }
    size = area * bytes_per_pixel;
    
    // Method 2: Use compiler built-ins (GCC/Clang)
    if (__builtin_mul_overflow(width, height, &size)) {
        return -1;  // Overflow detected
    }
    if (__builtin_mul_overflow(size, bytes_per_pixel, &size)) {
        return -1;  // Overflow detected
    }
    
    // Safe to allocate
    unsigned char* buffer = malloc(size);
    // ...
}

Common Integer Overflow Patterns:

Integer Overflow Vulnerability Patterns
Pattern	Example	Result	Fix
Allocation size calculation	`malloc(n * size)`	Small allocation, large copy	Check for overflow before multiply
Length + 1 for null terminator	`malloc(strlen(s) + 1)` with len=SIZE_MAX	Wrap to 0, tiny allocation	Validate string length first
Signed/unsigned mismatch	`int len = -1; malloc(len)`	Huge allocation (len as unsigned)	Use consistent types; check values
Truncation on assignment	`uint16_t short_len = long_len`	Loses high bits, small value	Validate before narrowing cast
Subtraction underflow	`unsigned remaining = total - used`	If used > total, massive value	Check ordering before subtraction

Real-World Example: CVE-2004-0597 (libpng) The PNG image library had an integer overflow in handling image dimensions:

libpng_overflow.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
// Simplified version of CVE-2004-0597
 
// PNG file specifies dimensions
// Attacker provides: height = 0x40000000
//                    row_bytes = 3 (RGB, 1 pixel per row)
//                    (Actually width * pixel_size, calculated elsewhere)
 
size_t height = png_read_dimension();  // 0x40000000 (1 GB rows)
size_t row_bytes = calculate_row_bytes();  // Let's say 4
 
// The vulnerable calculation:
size_t buffer_size = height * row_bytes;  
 
// 0x40000000 * 4 = 0x100000000 (4 GB)
// But on 32-bit: 0x100000000 mod 2^32 = 0
// buffer_size = 0!
 
unsigned char* buffer = malloc(buffer_size);  // malloc(0) → small buffer
 
// Read entire image (billions of bytes) into zero-byte buffer...
// Catastrophic heap overflow
 
// The fix: Validate dimensions against sane limits
// and check for arithmetic overflow

Integer Safety Strategies

Use size_t for sizes/counts on 64-bit systems (larger range). 2. Check for overflow BEFORE operations using safe math functions. 3. Validate all external inputs against reasonable bounds. 4. Use compiler built-ins: __builtin_add_overflow, __builtin_mul_overflow. 5. Consider safe integer libraries for critical code.

Modern Mitigation Techniques

Modern systems employ multiple overlapping defenses against buffer overflows. Understanding these mitigations—and their limitations—is essential for both defenders and for understanding why some overflows remain exploitable.

Stack Protection Mechanisms

•Stack Canaries (Stack Protector) — A random value placed between local variables and the return address. Before returning, the function checks if the canary is intact. If corrupted, the program aborts. Bypass: information leak to learn canary value, or overwrite only what's above the canary.
•ASLR (Address Space Layout Randomization) — Randomizes the base addresses of stack, heap, libraries, and executable. Attackers can't hardcode addresses. Bypass: information leak to discover addresses, or heap spray to hit probability.
•DEP/NX (Data Execution Prevention) — Marks data regions (stack, heap) as non-executable. Injected shellcode can't run. Bypass: ROP (Return-Oriented Programming) reuses existing executable code.
•Shadow Stack — Maintains a second copy of return addresses in protected memory. On return, compares stack return address with shadow. Much harder to bypass than canaries.
•Control Flow Integrity (CFI) — Validates that indirect jumps/calls go to valid targets. Limits which code can be reached via corrupted pointers. Bypass: find allowed targets that are still useful to attacker.

Enabling Mitigations
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# Compiler flags for mitigations (GCC/Clang)
 
# Stack canaries (enabled by default in many distros)
gcc -fstack-protector-strong   # Protect functions with arrays or pointers
gcc -fstack-protector-all      # Protect all functions (more overhead)
 
# Position Independent Executable (enables ASLR for main executable)
gcc -fPIE -pie program.c -o program
 
# Full RELRO (Read-Only Relocations) - protects GOT
gcc -Wl,-z,relro,-z,now program.c
 
# FORTIFY_SOURCE - compile-time and run-time buffer overflow checks
gcc -D_FORTIFY_SOURCE=2 -O2 program.c
 
# Stack clash protection (prevents stack/heap collision)
gcc -fstack-clash-protection program.c
 
# Control-Flow Integrity (requires newer compilers)
clang -fsanitize=cfi -flto -fvisibility=hidden program.c
 
# Check binary mitigations
checksec --file=./program
# Outputs: RELRO, Stack Canary, NX, PIE, etc.
 
# Check system-wide ASLR
cat /proc/sys/kernel/randomize_va_space
# 0 = disabled, 1 = stack/mmap, 2 = full (including brk)

Hardware Protections:

Hardware-Based Memory Safety Features
Technology	Platform	Mechanism	Status
Intel CET (Shadow Stack)	Intel 11th gen+	Hardware shadow stack for return addresses	Deployed in Windows 11, Linux
ARM PAC (Pointer Auth)	ARMv8.3+	Cryptographic signatures on pointers	Deployed in iOS, macOS, some Android
ARM BTI (Branch Target ID)	ARMv8.5+	Validates indirect branch targets	Deployed in iOS, macOS
ARM MTE (Memory Tagging)	ARMv8.5+	16 color tags for memory regions	Android 12+ optional, Linux support
Intel MPX	Skylake+ (deprecated)	Hardware bounds checking	Removed from GCC, discontinued
CHERI	Research/Arm Morello	Capability-based pointers	Research/evaluation

Mitigations Are Not Perfect

Every mitigation has been bypassed in specific circumstances. ASLR entropy is sometimes brute-forceable. Canaries can be leaked. CFI implementations have coverage gaps. Defense in depth—layering multiple protections—remains essential. The goal is to make exploitation prohibitively difficult, not impossible.

Secure Coding: Preventing Buffer Overflows at the Source

While mitigations make exploitation harder, the best defense is preventing overflows from existing in the first place. Secure coding practices eliminate entire vulnerability classes. Rule 1: Never Use Unsafe Functions

Dangerous Functions and Their Safe Alternatives
Dangerous	Why	Safe Alternative	Notes
`gets()`	No length limit at all	`fgets(buf, size, stdin)`	gets() was removed from C11
`strcpy()`	No length limit	`strncpy()` or `strlcpy()`	strlcpy guarantees null termination
`strcat()`	No length limit	`strncat()` or `strlcat()`	Calculate remaining space carefully
`sprintf()`	No length limit	`snprintf()`	Always pass buffer size
`scanf("%s", buf)`	No length limit	`scanf("%63s", buf)` or fgets	Specify max width in format
`vsprintf()`	No length limit	`vsnprintf()`	For variadic formatting

safe_string_handling.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
// Safe string handling examples
 
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
 
#define BUFFER_SIZE 256
 
// WRONG: Using dangerous functions
void dangerous_copy(const char* input) {
    char buffer[BUFFER_SIZE];
    strcpy(buffer, input);  // OVERFLOW if input > 255 chars
}
 
// RIGHT: Using safe functions with explicit lengths
void safe_copy(const char* input) {
    char buffer[BUFFER_SIZE];
    
    // Option 1: strncpy (note: may not null-terminate!)
    strncpy(buffer, input, sizeof(buffer) - 1);
    buffer[sizeof(buffer) - 1] = '\0';  // Ensure termination
    
    // Option 2: snprintf (always null-terminates)
    snprintf(buffer, sizeof(buffer), "%s", input);
    
    // Option 3: Check manually and reject
    size_t input_len = strlen(input);
    if (input_len >= sizeof(buffer)) {
        fprintf(stderr, "Input too long
");
        return;
    }
    memcpy(buffer, input, input_len + 1);  // +1 for null
}
 
// PATTERN: Safe concatenation
char* safe_concat(const char* s1, const char* s2) {
    size_t len1 = strlen(s1);
    size_t len2 = strlen(s2);
    
    // Check for overflow in size calculation
    if (len1 > SIZE_MAX - len2 - 1) {
        return NULL;  // Would overflow
    }
    
    size_t total = len1 + len2 + 1;
    char* result = malloc(total);
    if (!result) return NULL;
    
    memcpy(result, s1, len1);
    memcpy(result + len1, s2, len2 + 1);  // +1 for null
    
    return result;
}
 
// PATTERN: Safe reading from file
int read_line_safely(FILE* f, char* buffer, size_t buffer_size) {
    if (fgets(buffer, buffer_size, f) == NULL) {
        return -1;  // EOF or error
    }
    
    // Remove trailing newline if present
    size_t len = strlen(buffer);
    if (len > 0 && buffer[len - 1] == '
') {
        buffer[len - 1] = '\0';
    }
    
    return 0;
}

Rule 2: Validate All External Input Treat ALL external data as potentially malicious:

input_validation.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
// Input validation patterns
 
// Validate numeric ranges
int parse_port(const char* str) {
    char* end;
    errno = 0;
    long val = strtol(str, &end, 10);
    
    // Check for conversion errors
    if (errno != 0 || end == str || *end != '\0') {
        return -1;  // Invalid
    }
    
    // Check range
    if (val < 1 || val > 65535) {
        return -1;  // Out of valid port range
    }
    
    return (int)val;
}
 
// Validate file paths (basic example)
int is_safe_path(const char* path) {
    // Reject if contains path traversal
    if (strstr(path, "..") != NULL) {
        return 0;  // Potential directory traversal
    }
    
    // Reject if starts with /
    if (path[0] == '/') {
        return 0;  // Absolute path
    }
    
    // Validate characters (example: alphanumeric + underscore only)
    for (const char* p = path; *p; p++) {
        if (!isalnum(*p) && *p != '_' && *p != '.' && *p != '/') {
            return 0;  // Invalid character
        }
    }
    
    return 1;  // Appears safe
}
 
// Validate array indices
void access_array_safely(int* array, size_t array_size, size_t index) {
    if (index >= array_size) {
        fprintf(stderr, "Index %zu out of bounds (size %zu)
", 
                index, array_size);
        abort();  // Or handle gracefully
    }
    
    // Safe to access
    int value = array[index];
}

Rule 3: Use Memory-Safe Abstractions When Possible

Memory-Safe Alternatives

•Use std::string and std::vector in C++ — Automatic bounds checking and memory management. Eliminate entire classes of overflows.
•Consider Rust for new systems code — Memory safety enforced at compile time. No buffer overflows possible in safe Rust.
•Use safe string libraries in C — Libraries like glib's GString or BSD's strlcpy/strlcat family provide safer operations.
•Wrap unsafe operations — Create safe wrapper functions that always validate bounds, then use only the wrappers.
•Enable sanitizers in testing — AddressSanitizer catches overflows at runtime during development and testing.

Defense in Depth for Code

Layer secure coding practices: Use safe functions AND validate input AND enable compiler mitigations AND run sanitizers during testing. Each layer catches what others miss. No single technique is sufficient alone.

Detecting Buffer Overflows in Practice

Buffer overflows often don't crash immediately—they may corrupt data subtly or crash much later when corrupted data is used. Detection requires specialized tools. Compile-Time Detection:

compile_time_detection.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// _FORTIFY_SOURCE catches some overflows at compile time
 
#define _FORTIFY_SOURCE 2
#include <string.h>
#include <stdio.h>
 
void fixed_size_overflow(void) {
    char buffer[10];
    
    // With _FORTIFY_SOURCE=2, compiler can detect this:
    strcpy(buffer, "This string is way too long for the buffer!");
    // WARNING: call to __builtin___strcpy_chk will always overflow
    
    // Runtime check inserted for dynamic cases:
    char* input = get_input();  // Compiler doesn't know length
    strcpy(buffer, input);      // Runtime abort if overflow detected
}
 
// Compile with:
// gcc -D_FORTIFY_SOURCE=2 -O2 -Wall program.c
 
// More static analysis options:
// gcc -fanalyzer program.c        // GCC 10+ static analyzer
// clang --analyze program.c       // Clang static analyzer
// scan-build make                 // Analyze entire build

Runtime Detection with Sanitizers: AddressSanitizer (ASan) is the gold standard for detecting buffer overflows at runtime. It instruments every memory access to check validity:

asan_example.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
// Example: Detecting heap buffer overflow with ASan
 
#include <stdlib.h>
#include <string.h>
 
int main() {
    char* buffer = malloc(10);
    
    // Write past the end of allocation
    buffer[10] = 'X';  // Off-by-one overflow!
    
    free(buffer);
    return 0;
}
 
/*
Compile with AddressSanitizer:
$ gcc -fsanitize=address -g overflow.c -o overflow
 
Run:
$ ./overflow
 
=================================================================
==12345==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000001a at pc 0x... bp 0x... sp 0x...
WRITE of size 1 at 0x60200000001a thread T0
    #0 0x... in main overflow.c:9
    #1 0x... in __libc_start_main
    #2 0x... in _start
 
0x60200000001a is located 0 bytes to the right of 10-byte region [0x602000000010,0x60200000001a)
allocated by thread T0 here:
    #0 0x... in malloc
    #1 0x... in main overflow.c:6
 
SUMMARY: AddressSanitizer: heap-buffer-overflow overflow.c:9 in main
*/

Fuzzing to Discover Overflows: Fuzz testing generates random or mutated inputs to trigger unexpected behavior. Combined with sanitizers, it effectively discovers overflows:

fuzz_testing.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# Fuzzing with AFL++ and AddressSanitizer
 
# Build with AFL++ instrumentation + ASan
AFL_USE_ASAN=1 afl-clang-fast -o program_fuzz program.c
 
# Create seed inputs
mkdir seeds
echo "normal input" > seeds/seed1
 
# Run fuzzer
afl-fuzz -i seeds -o findings ./program_fuzz
 
# AFL++ will:
# 1. Mutate inputs systematically
# 2. Track code coverage to find new paths
# 3. ASan catches any overflows triggered
# 4. Report crashes with reproducer inputs
 
# For libraries, use libFuzzer (built into LLVM):
clang -fsanitize=fuzzer,address -o fuzz_target target.c
 
# target.c contains:
# int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
#     your_function_to_test(data, size);
#     return 0;
# }
 
./fuzz_target corpus/ -max_len=1000

Continuous Fuzzing

Major projects like Chrome, Firefox, and the Linux kernel run continuous fuzzing. Google's OSS-Fuzz provides free fuzzing infrastructure for open-source projects. New overflows are regularly discovered in heavily-reviewed code—fuzzing finds what code review misses.

Operating SystemsMemory Debugging

Memory Debugging: Detecting and Resolving Memory Errors

LevelAdvanced

Duration120 mins

TopicMemory Debugging

2 / 5

Buffer Overflows

When Writes Escape Their Bounds

On November 2, 1988, a graduate student at Cornell University unleashed what would become known as the Morris Worm—the first major computer worm to spread across the nascent Internet. Among its attack vectors was a buffer overflow in the Unix fingerd daemon. By sending a carefully crafted 536-character string where only 512 characters were expected, Robert Morris's worm could overwrite the stack, hijack program control, and spread to the next machine. The Internet ground to a halt; an estimated 6,000 computers (10% of the entire Internet at the time) were infected. Three decades later, buffer overflows remain among the most severe vulnerability classes. Despite countless mitigations—stack canaries, ASLR, DEP, CFI—attackers continue to find ways to exploit programs that write beyond buffer boundaries. The 2021 Pegasus spyware used a zero-click buffer overflow in Apple's iMessage to compromise iPhones without any user interaction. This page provides a comprehensive exploration of buffer overflows: how they occur, why they're exploitable, and how modern systems defend against them. Understanding buffer overflows is essential for anyone working with systems software or security.

What You Will Learn

By the end of this page, you will understand: (1) The memory layout that makes buffer overflows dangerous, (2) Stack-based overflow mechanics including return address corruption, (3) Heap-based overflow mechanics and metadata corruption, (4) Integer overflows as buffer overflow enablers, (5) Modern mitigation techniques and their limitations, and (6) Secure coding practices that prevent overflows.

Memory Layout and Why Overflows Are Dangerous

To understand buffer overflows, we must first understand how programs lay out data in memory. The danger of overflows stems from the proximity of buffers to critical control data. Process Memory Layout (x86-64):

memory_layout.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
High Addresses (e.g., 0x7FFFFFFFFFFF)
┌─────────────────────────────────────┐
│           Kernel Space              │  ← Not accessible from user mode
├─────────────────────────────────────┤
│             Stack                   │  ← Grows downward (toward lower addresses)
│   ┌─────────────────────────────┐   │
│   │ Return address    [8 bytes] │   │  ← CRITICAL: If overwritten, controls execution
│   │ Saved base pointer [8 bytes]│   │
│   │ Local variables             │   │  ← Buffers typically here
│   │ (char buffer[64], int x...)│   │
│   └─────────────────────────────┘   │
│              ↓                      │
│         (grows down)                │
│                                     │
├─────────────────────────────────────┤
│                                     │
│          Memory-Mapped              │  ← Shared libraries, mmap() regions
│            Region                   │
│                                     │
├─────────────────────────────────────┤
│              ↑                      │
│         (grows up)                  │
│   ┌─────────────────────────────┐   │
│   │ Heap                        │   │
│   │ malloc'd data, metadata     │   │  ← Another overflow target
│   └─────────────────────────────┘   │
│             Heap                    │
├─────────────────────────────────────┤
│      Uninitialized Data (BSS)       │  ← Global/static uninitialized variables
├─────────────────────────────────────┤
│      Initialized Data (.data)       │  ← Global/static initialized variables
├─────────────────────────────────────┤
│         Text (Code) Segment         │  ← Executable instructions (read-only)
└─────────────────────────────────────┘
Low Addresses (e.g., 0x400000)

The Critical Insight: On the stack, local variables (including buffers) are adjacent to control information (return addresses, saved registers) . When you write past the end of a buffer, you don't hit empty space—you overwrite whatever happens to be next in memory. If a buffer is near the return address, overflowing it can redirect execution. Why Bounds Aren't Checked: C and C++ prioritize performance over safety. Array accesses compile to simple pointer arithmetic with no bounds checking:

bounds_checking.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// What you write:
char buffer[64];
buffer[70] = 'X';  // Clearly out of bounds
 
// What the compiler generates (x86-64):
// mov    BYTE PTR [rbp-64+70], 'X'
// 
// No check, no error - just writes to whatever is at that address
// In this case: 6 bytes past buffer, possibly into saved registers
 
// The philosophy:
// "Programmers are responsible for correct code"
// "Runtime checks are too expensive for systems programming"
// "Trust the programmer"
 
// The consequence:
// Millions of security vulnerabilities over 50 years

Buffer Overflow Definition: A buffer overflow occurs when a program writes data beyond the allocated bounds of a buffer, corrupting adjacent memory that may contain other data, control structures, or executable code. Buffer overflows are classified by location:

Buffer Overflow Classification
Type	Location	Typical Target	Exploitation Difficulty
Stack Buffer Overflow	Stack (local variables)	Return address, saved registers	Classic, well-understood
Heap Buffer Overflow	Heap (malloc'd memory)	Heap metadata, adjacent objects	Requires heap layout knowledge
BSS/Data Overflow	Global/static data	Function pointers, configuration data	Program-specific
Off-by-One	Any (single byte overwrite)	Terminators, pointers	Subtle but exploitable

Why This Still Matters

Despite being one of the oldest vulnerability classes, buffer overflows continue to be discovered in modern software. In 2023 alone, critical buffer overflows were found in Chrome, Firefox, Windows, Linux, iOS, and Android. Any code handling untrusted input in C/C++ is a potential overflow waiting to be discovered.

Stack-Based Buffer Overflows: The Classic Attack

Stack-based buffer overflows are the 'classic' form, extensively documented since the 1996 paper "Smashing the Stack for Fun and Profit" by Aleph One. Understanding them is foundational to understanding all overflow attacks. Stack Frame Structure: When a function is called, the processor creates a stack frame containing:

stack_frame.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Function call: caller() calls vulnerable()
 
High Addresses
┌──────────────────────────────────────┐
│   caller()'s stack frame             │
├──────────────────────────────────────┤
│   Arguments to vulnerable()          │  ← Pushed by caller
├──────────────────────────────────────┤
│   Return Address [8 bytes]           │  ← Pushed by CALL instruction
│   (where to resume after return)     │     TARGET #1 for exploitation
├──────────────────────────────────────┤
│   Saved RBP [8 bytes]                │  ← Pushed by function prologue  
│   (caller's base pointer)            │     TARGET #2 for exploitation
├──────────────────────────────────────┤  ← Current RBP points here
│   Local variable: int x              │  [rbp - 4]
├──────────────────────────────────────┤
│   Local variable: char buffer[64]    │  [rbp - 68] to [rbp - 5]
│   buffer[0] ... buffer[63]           │     OVERFLOW STARTS HERE
├──────────────────────────────────────┤  ← Current RSP (top of stack)
Low Addresses
 
Overflow Direction (if writing past buffer[63]):
→ buffer[64] corrupts 'int x'
→ buffer[68] starts corrupting saved RBP
→ buffer[76] starts corrupting return address
→ return address now points to attacker-controlled location

Vulnerable Code Example:

vulnerable_function.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
// Classic vulnerable function
#include <stdio.h>
#include <string.h>
 
void vulnerable(char* user_input) {
    char buffer[64];
    
    // DANGEROUS: strcpy doesn't check bounds
    // If user_input > 64 bytes, overflow occurs
    strcpy(buffer, user_input);
    
    printf("You entered: %s
", buffer);
}
 
int main(int argc, char* argv[]) {
    if (argc < 2) {
        printf("Usage: %s <input>
", argv[0]);
        return 1;
    }
    
    vulnerable(argv[1]);
    
    printf("Program completed normally.
");
    return 0;
}
 
/* What happens with 64 'A's:
   buffer = "AAAA...AAAA" (64 As) - fits exactly
   Output: "You entered: AAA..."
   Program completes normally
   
   What happens with 80 'A's:
   buffer = "AAAA...AAAA" (64 As)
   int x gets corrupted (if present)
   saved RBP = 0x4141414141414141 ("AAAAAAAA")
   return address = 0x4141414141414141
   When function returns, jumps to 0x4141414141414141
   SEGFAULT (or worse - exploitation if address is valid)
*/

Exploitation Goals: Overwriting the return address allows the attacker to redirect execution. Common exploitation techniques: 1. Return-to-Shellcode (Classic, now rare): Place executable code ("shellcode") in the buffer Overwrite return address to point at the buffer When function returns, shellcode executes Mitigated by: DEP/NX (non-executable stack) 2. Return-to-libc: Don't inject code; reuse existing code Overwrite return address to point at libc function (e.g., system()) Arrange stack so function receives controlled arguments Mitigated by: ASLR (randomizes libc location) 3. Return-Oriented Programming (ROP): Chain together small code snippets ("gadgets") ending in ret Each gadget performs one small operation Together, they form complete attack logic Mitigated by: CFI, Shadow Stacks (partially)

The Evolution of Exploitation

Each mitigation spawned new techniques. DEP stopped shellcode \to attackers used return-to-libc. ASLR randomized addresses \to attackers found information leaks. Stack canaries detected overwrites \to attackers found ways to leak or bypass them. This cat-and-mouse game continues today.

Heap-Based Buffer Overflows: Corrupting the Allocator

Heap overflows target dynamically allocated memory. Unlike stack overflows (which target return addresses), heap overflows typically corrupt heap metadata or adjacent heap objects. Heap Memory Layout:

heap_layout.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Heap Memory (simplified view of two adjacent allocations)
 
┌───────────────────────────────────────────────────────────┐
│   Chunk A Metadata (e.g., size, prev_size, flags)        │  ← Allocator's bookkeeping
├───────────────────────────────────────────────────────────┤
│   Chunk A User Data                                       │  ← What malloc() returns
│   buffer = malloc(64);                                    │
│   [         64 bytes of user data                    ]    │
│                                                           │  ← OVERFLOW GOES HERE
├───────────────────────────────────────────────────────────┤
│   Chunk B Metadata (e.g., size, prev_size, flags)        │  ← Corrupted by overflow!
├───────────────────────────────────────────────────────────┤
│   Chunk B User Data                                       │
│   victim = malloc(128);                                   │
│   [This might contain function pointers, objects...]     │  ← Also corruptible
└───────────────────────────────────────────────────────────┘
 
Overflow from Chunk A corrupts:
1. Chunk B's metadata → Confuses allocator → Arbitrary write primitive
2. Chunk B's user data → Corrupts application data → Depends on what's there

Heap Metadata Attacks: Heap allocators (like glibc's ptmalloc) use metadata chunks to manage free lists. Corrupting this metadata can trick the allocator into writing attacker-controlled data to attacker-controlled locations:

heap_exploitation.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
// Simplified heap overflow exploitation concepts
 
// Heap chunk structure (glibc ptmalloc2 simplified)
// When chunk is FREE:
struct malloc_chunk {
    size_t prev_size;  // Size of previous chunk (if free)  
    size_t size;       // Size of this chunk + flags in low bits
    
    // These only exist in FREE chunks:
    struct malloc_chunk* fd;  // Forward pointer in free list
    struct malloc_chunk* bk;  // Backward pointer in free list
};
 
// CLASSIC ATTACK: Unsafe unlink
// 
// When a free chunk is removed from the free list, allocator does:
//     fd->bk = bk;    // Write bk to location fd+offset
//     bk->fd = fd;    // Write fd to location bk+offset
//
// If attacker controls fd and bk (via overflow), they get:
//     Write arbitrary value (bk) to arbitrary address (fd+offset)
//     This is called "write-what-where" or "arbitrary write"
//
// Example exploitation:
//     fd = &got_entry - offset  (GOT = Global Offset Table)
//     bk = &shellcode
//     When chunk unlinked: GOT entry points to shellcode
//     Next call to that function → shellcode runs
 
// Modern allocators have many protections against this:
// - Corrupted size checks
// - Safe unlinking (verify fd->bk == p && bk->fd == p)
// - Randomized metadata locations
// - Guard pages between heap regions

Heap Spray Attacks: When precise heap layout control is difficult, attackers may use "heap spraying"—allocating many copies of the same data to increase the probability that a corrupted pointer lands in attacker-controlled memory:

heap_spray.js
JavaScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// Heap spray concept (for understanding, not actual exploitation)
 
// In browser exploits, JavaScript could spray the heap:
var spray = [];
var payload = "PAYLOAD_PLACEHOLDER";  // shellcode goes here
 
// Allocate hundreds of MB filled with our payload  
for (var i = 0; i < 200; i++) {
    // Create 1MB string filled with NOP sleds + shellcode
    var block = "";
    for (var j = 0; j < 1024 * 1024 / payload.length; j++) {
        block += payload;
    }
    spray.push(block);
}
 
// Now the heap looks like:
// [spray block][spray block][spray block]...[spray block]
// 
// If attacker can corrupt a pointer and get execution at
// ANY address in this huge sprayed region, they win.
// The "NOP sled" technique slides execution forward until
// it hits actual shellcode.
 
// Modern mitigations:
// - Heap isolation (different allocators for different object types)
// - Guard pages (unmapped pages between allocations)
// - Memory tagging (ARM MTE)
// - Reduced JavaScript precision for spray detection

Heap Exploitation is Complex

Unlike stack overflows (which have consistent structure), heap exploits require deep understanding of specific allocator implementations. Modern allocators like jemalloc, tcmalloc, and hardened glibc variants include extensive protections. However, the complexity also means bugs in allocator code itself can be catastrophic.

Integer Overflows: The Gateway to Buffer Overflows

Many buffer overflows don't stem from direct write operations exceeding bounds. Instead, they originate from integer overflows that cause incorrect buffer size calculations. Integer overflows are among the most subtle and dangerous vulnerability classes. How Integer Overflow Leads to Buffer Overflow:

integer_overflow.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
// Integer overflow leading to heap buffer overflow
 
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
 
// Vulnerable image processing function
int process_image(unsigned int width, unsigned int height, 
                  unsigned int bytes_per_pixel, 
                  unsigned char* image_data) {
    
    // Calculate buffer size for image
    // BUG: Integer overflow!
    unsigned int size = width * height * bytes_per_pixel;
    
    // Example attack values:
    // width = 0x10000 (65536)
    // height = 0x10000 (65536)  
    // bytes_per_pixel = 4
    // 
    // Expected size: 65536 * 65536 * 4 = 17,179,869,184 (17 GB)
    // Actual size after 32-bit overflow: 0 (or small value)
    
    // Allocate undersized buffer
    unsigned char* buffer = malloc(size);  // Allocates 0 or tiny amount
    if (!buffer) return -1;
    
    // Copy image data into undersized buffer
    // This writes GIGABYTES past the allocated buffer
    memcpy(buffer, image_data, width * height * bytes_per_pixel);
    
    // Massive heap overflow!
    
    free(buffer);
    return 0;
}
 
// FIXED VERSION: Check for overflow before allocation
int process_image_safe(size_t width, size_t height, 
                       size_t bytes_per_pixel,
                       unsigned char* image_data) {
    
    // Check for multiplication overflow
    size_t size;
    
    // Method 1: Manual check
    if (width > SIZE_MAX / height) {
        return -1;  // Overflow would occur
    }
    size_t area = width * height;
    
    if (area > SIZE_MAX / bytes_per_pixel) {
        return -1;  // Overflow would occur
    }
    size = area * bytes_per_pixel;
    
    // Method 2: Use compiler built-ins (GCC/Clang)
    if (__builtin_mul_overflow(width, height, &size)) {
        return -1;  // Overflow detected
    }
    if (__builtin_mul_overflow(size, bytes_per_pixel, &size)) {
        return -1;  // Overflow detected
    }
    
    // Safe to allocate
    unsigned char* buffer = malloc(size);
    // ...
}

Common Integer Overflow Patterns:

Integer Overflow Vulnerability Patterns
Pattern	Example	Result	Fix
Allocation size calculation	`malloc(n * size)`	Small allocation, large copy	Check for overflow before multiply
Length + 1 for null terminator	`malloc(strlen(s) + 1)` with len=SIZE_MAX	Wrap to 0, tiny allocation	Validate string length first
Signed/unsigned mismatch	`int len = -1; malloc(len)`	Huge allocation (len as unsigned)	Use consistent types; check values
Truncation on assignment	`uint16_t short_len = long_len`	Loses high bits, small value	Validate before narrowing cast
Subtraction underflow	`unsigned remaining = total - used`	If used > total, massive value	Check ordering before subtraction

Real-World Example: CVE-2004-0597 (libpng) The PNG image library had an integer overflow in handling image dimensions:

libpng_overflow.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
// Simplified version of CVE-2004-0597
 
// PNG file specifies dimensions
// Attacker provides: height = 0x40000000
//                    row_bytes = 3 (RGB, 1 pixel per row)
//                    (Actually width * pixel_size, calculated elsewhere)
 
size_t height = png_read_dimension();  // 0x40000000 (1 GB rows)
size_t row_bytes = calculate_row_bytes();  // Let's say 4
 
// The vulnerable calculation:
size_t buffer_size = height * row_bytes;  
 
// 0x40000000 * 4 = 0x100000000 (4 GB)
// But on 32-bit: 0x100000000 mod 2^32 = 0
// buffer_size = 0!
 
unsigned char* buffer = malloc(buffer_size);  // malloc(0) → small buffer
 
// Read entire image (billions of bytes) into zero-byte buffer...
// Catastrophic heap overflow
 
// The fix: Validate dimensions against sane limits
// and check for arithmetic overflow

Integer Safety Strategies

Use size_t for sizes/counts on 64-bit systems (larger range). 2. Check for overflow BEFORE operations using safe math functions. 3. Validate all external inputs against reasonable bounds. 4. Use compiler built-ins: __builtin_add_overflow, __builtin_mul_overflow. 5. Consider safe integer libraries for critical code.

Modern Mitigation Techniques

Modern systems employ multiple overlapping defenses against buffer overflows. Understanding these mitigations—and their limitations—is essential for both defenders and for understanding why some overflows remain exploitable.

Stack Protection Mechanisms

•Stack Canaries (Stack Protector) — A random value placed between local variables and the return address. Before returning, the function checks if the canary is intact. If corrupted, the program aborts. Bypass: information leak to learn canary value, or overwrite only what's above the canary.
•ASLR (Address Space Layout Randomization) — Randomizes the base addresses of stack, heap, libraries, and executable. Attackers can't hardcode addresses. Bypass: information leak to discover addresses, or heap spray to hit probability.
•DEP/NX (Data Execution Prevention) — Marks data regions (stack, heap) as non-executable. Injected shellcode can't run. Bypass: ROP (Return-Oriented Programming) reuses existing executable code.
•Shadow Stack — Maintains a second copy of return addresses in protected memory. On return, compares stack return address with shadow. Much harder to bypass than canaries.
•Control Flow Integrity (CFI) — Validates that indirect jumps/calls go to valid targets. Limits which code can be reached via corrupted pointers. Bypass: find allowed targets that are still useful to attacker.

Enabling Mitigations
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# Compiler flags for mitigations (GCC/Clang)
 
# Stack canaries (enabled by default in many distros)
gcc -fstack-protector-strong   # Protect functions with arrays or pointers
gcc -fstack-protector-all      # Protect all functions (more overhead)
 
# Position Independent Executable (enables ASLR for main executable)
gcc -fPIE -pie program.c -o program
 
# Full RELRO (Read-Only Relocations) - protects GOT
gcc -Wl,-z,relro,-z,now program.c
 
# FORTIFY_SOURCE - compile-time and run-time buffer overflow checks
gcc -D_FORTIFY_SOURCE=2 -O2 program.c
 
# Stack clash protection (prevents stack/heap collision)
gcc -fstack-clash-protection program.c
 
# Control-Flow Integrity (requires newer compilers)
clang -fsanitize=cfi -flto -fvisibility=hidden program.c
 
# Check binary mitigations
checksec --file=./program
# Outputs: RELRO, Stack Canary, NX, PIE, etc.
 
# Check system-wide ASLR
cat /proc/sys/kernel/randomize_va_space
# 0 = disabled, 1 = stack/mmap, 2 = full (including brk)

Hardware Protections:

Hardware-Based Memory Safety Features
Technology	Platform	Mechanism	Status
Intel CET (Shadow Stack)	Intel 11th gen+	Hardware shadow stack for return addresses	Deployed in Windows 11, Linux
ARM PAC (Pointer Auth)	ARMv8.3+	Cryptographic signatures on pointers	Deployed in iOS, macOS, some Android
ARM BTI (Branch Target ID)	ARMv8.5+	Validates indirect branch targets	Deployed in iOS, macOS
ARM MTE (Memory Tagging)	ARMv8.5+	16 color tags for memory regions	Android 12+ optional, Linux support
Intel MPX	Skylake+ (deprecated)	Hardware bounds checking	Removed from GCC, discontinued
CHERI	Research/Arm Morello	Capability-based pointers	Research/evaluation

Mitigations Are Not Perfect

Every mitigation has been bypassed in specific circumstances. ASLR entropy is sometimes brute-forceable. Canaries can be leaked. CFI implementations have coverage gaps. Defense in depth—layering multiple protections—remains essential. The goal is to make exploitation prohibitively difficult, not impossible.

Secure Coding: Preventing Buffer Overflows at the Source

While mitigations make exploitation harder, the best defense is preventing overflows from existing in the first place. Secure coding practices eliminate entire vulnerability classes. Rule 1: Never Use Unsafe Functions

Dangerous Functions and Their Safe Alternatives
Dangerous	Why	Safe Alternative	Notes
`gets()`	No length limit at all	`fgets(buf, size, stdin)`	gets() was removed from C11
`strcpy()`	No length limit	`strncpy()` or `strlcpy()`	strlcpy guarantees null termination
`strcat()`	No length limit	`strncat()` or `strlcat()`	Calculate remaining space carefully
`sprintf()`	No length limit	`snprintf()`	Always pass buffer size
`scanf("%s", buf)`	No length limit	`scanf("%63s", buf)` or fgets	Specify max width in format
`vsprintf()`	No length limit	`vsnprintf()`	For variadic formatting

safe_string_handling.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
// Safe string handling examples
 
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
 
#define BUFFER_SIZE 256
 
// WRONG: Using dangerous functions
void dangerous_copy(const char* input) {
    char buffer[BUFFER_SIZE];
    strcpy(buffer, input);  // OVERFLOW if input > 255 chars
}
 
// RIGHT: Using safe functions with explicit lengths
void safe_copy(const char* input) {
    char buffer[BUFFER_SIZE];
    
    // Option 1: strncpy (note: may not null-terminate!)
    strncpy(buffer, input, sizeof(buffer) - 1);
    buffer[sizeof(buffer) - 1] = '\0';  // Ensure termination
    
    // Option 2: snprintf (always null-terminates)
    snprintf(buffer, sizeof(buffer), "%s", input);
    
    // Option 3: Check manually and reject
    size_t input_len = strlen(input);
    if (input_len >= sizeof(buffer)) {
        fprintf(stderr, "Input too long
");
        return;
    }
    memcpy(buffer, input, input_len + 1);  // +1 for null
}
 
// PATTERN: Safe concatenation
char* safe_concat(const char* s1, const char* s2) {
    size_t len1 = strlen(s1);
    size_t len2 = strlen(s2);
    
    // Check for overflow in size calculation
    if (len1 > SIZE_MAX - len2 - 1) {
        return NULL;  // Would overflow
    }
    
    size_t total = len1 + len2 + 1;
    char* result = malloc(total);
    if (!result) return NULL;
    
    memcpy(result, s1, len1);
    memcpy(result + len1, s2, len2 + 1);  // +1 for null
    
    return result;
}
 
// PATTERN: Safe reading from file
int read_line_safely(FILE* f, char* buffer, size_t buffer_size) {
    if (fgets(buffer, buffer_size, f) == NULL) {
        return -1;  // EOF or error
    }
    
    // Remove trailing newline if present
    size_t len = strlen(buffer);
    if (len > 0 && buffer[len - 1] == '
') {
        buffer[len - 1] = '\0';
    }
    
    return 0;
}

Rule 2: Validate All External Input Treat ALL external data as potentially malicious:

input_validation.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
// Input validation patterns
 
// Validate numeric ranges
int parse_port(const char* str) {
    char* end;
    errno = 0;
    long val = strtol(str, &end, 10);
    
    // Check for conversion errors
    if (errno != 0 || end == str || *end != '\0') {
        return -1;  // Invalid
    }
    
    // Check range
    if (val < 1 || val > 65535) {
        return -1;  // Out of valid port range
    }
    
    return (int)val;
}
 
// Validate file paths (basic example)
int is_safe_path(const char* path) {
    // Reject if contains path traversal
    if (strstr(path, "..") != NULL) {
        return 0;  // Potential directory traversal
    }
    
    // Reject if starts with /
    if (path[0] == '/') {
        return 0;  // Absolute path
    }
    
    // Validate characters (example: alphanumeric + underscore only)
    for (const char* p = path; *p; p++) {
        if (!isalnum(*p) && *p != '_' && *p != '.' && *p != '/') {
            return 0;  // Invalid character
        }
    }
    
    return 1;  // Appears safe
}
 
// Validate array indices
void access_array_safely(int* array, size_t array_size, size_t index) {
    if (index >= array_size) {
        fprintf(stderr, "Index %zu out of bounds (size %zu)
", 
                index, array_size);
        abort();  // Or handle gracefully
    }
    
    // Safe to access
    int value = array[index];
}

Rule 3: Use Memory-Safe Abstractions When Possible

Memory-Safe Alternatives

•Use std::string and std::vector in C++ — Automatic bounds checking and memory management. Eliminate entire classes of overflows.
•Consider Rust for new systems code — Memory safety enforced at compile time. No buffer overflows possible in safe Rust.
•Use safe string libraries in C — Libraries like glib's GString or BSD's strlcpy/strlcat family provide safer operations.
•Wrap unsafe operations — Create safe wrapper functions that always validate bounds, then use only the wrappers.
•Enable sanitizers in testing — AddressSanitizer catches overflows at runtime during development and testing.

Defense in Depth for Code

Layer secure coding practices: Use safe functions AND validate input AND enable compiler mitigations AND run sanitizers during testing. Each layer catches what others miss. No single technique is sufficient alone.

Detecting Buffer Overflows in Practice

Buffer overflows often don't crash immediately—they may corrupt data subtly or crash much later when corrupted data is used. Detection requires specialized tools. Compile-Time Detection:

compile_time_detection.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// _FORTIFY_SOURCE catches some overflows at compile time
 
#define _FORTIFY_SOURCE 2
#include <string.h>
#include <stdio.h>
 
void fixed_size_overflow(void) {
    char buffer[10];
    
    // With _FORTIFY_SOURCE=2, compiler can detect this:
    strcpy(buffer, "This string is way too long for the buffer!");
    // WARNING: call to __builtin___strcpy_chk will always overflow
    
    // Runtime check inserted for dynamic cases:
    char* input = get_input();  // Compiler doesn't know length
    strcpy(buffer, input);      // Runtime abort if overflow detected
}
 
// Compile with:
// gcc -D_FORTIFY_SOURCE=2 -O2 -Wall program.c
 
// More static analysis options:
// gcc -fanalyzer program.c        // GCC 10+ static analyzer
// clang --analyze program.c       // Clang static analyzer
// scan-build make                 // Analyze entire build

Runtime Detection with Sanitizers: AddressSanitizer (ASan) is the gold standard for detecting buffer overflows at runtime. It instruments every memory access to check validity:

asan_example.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
// Example: Detecting heap buffer overflow with ASan
 
#include <stdlib.h>
#include <string.h>
 
int main() {
    char* buffer = malloc(10);
    
    // Write past the end of allocation
    buffer[10] = 'X';  // Off-by-one overflow!
    
    free(buffer);
    return 0;
}
 
/*
Compile with AddressSanitizer:
$ gcc -fsanitize=address -g overflow.c -o overflow
 
Run:
$ ./overflow
 
=================================================================
==12345==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000001a at pc 0x... bp 0x... sp 0x...
WRITE of size 1 at 0x60200000001a thread T0
    #0 0x... in main overflow.c:9
    #1 0x... in __libc_start_main
    #2 0x... in _start
 
0x60200000001a is located 0 bytes to the right of 10-byte region [0x602000000010,0x60200000001a)
allocated by thread T0 here:
    #0 0x... in malloc
    #1 0x... in main overflow.c:6
 
SUMMARY: AddressSanitizer: heap-buffer-overflow overflow.c:9 in main
*/

Fuzzing to Discover Overflows: Fuzz testing generates random or mutated inputs to trigger unexpected behavior. Combined with sanitizers, it effectively discovers overflows:

fuzz_testing.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# Fuzzing with AFL++ and AddressSanitizer
 
# Build with AFL++ instrumentation + ASan
AFL_USE_ASAN=1 afl-clang-fast -o program_fuzz program.c
 
# Create seed inputs
mkdir seeds
echo "normal input" > seeds/seed1
 
# Run fuzzer
afl-fuzz -i seeds -o findings ./program_fuzz
 
# AFL++ will:
# 1. Mutate inputs systematically
# 2. Track code coverage to find new paths
# 3. ASan catches any overflows triggered
# 4. Report crashes with reproducer inputs
 
# For libraries, use libFuzzer (built into LLVM):
clang -fsanitize=fuzzer,address -o fuzz_target target.c
 
# target.c contains:
# int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
#     your_function_to_test(data, size);
#     return 0;
# }
 
./fuzz_target corpus/ -max_len=1000

Continuous Fuzzing

Major projects like Chrome, Firefox, and the Linux kernel run continuous fuzzing. Google's OSS-Fuzz provides free fuzzing infrastructure for open-source projects. New overflows are regularly discovered in heavily-reviewed code—fuzzing finds what code review misses.

Buffer Overflows

When Writes Escape Their Bounds

Memory Layout and Why Overflows Are Dangerous

Stack-Based Buffer Overflows: The Classic Attack

Heap-Based Buffer Overflows: Corrupting the Allocator

Integer Overflows: The Gateway to Buffer Overflows

Modern Mitigation Techniques

Secure Coding: Preventing Buffer Overflows at the Source

Detecting Buffer Overflows in Practice

There’s more to understand.

Buffer Overflows

When Writes Escape Their Bounds

Memory Layout and Why Overflows Are Dangerous

Stack-Based Buffer Overflows: The Classic Attack

Heap-Based Buffer Overflows: Corrupting the Allocator

Integer Overflows: The Gateway to Buffer Overflows

Modern Mitigation Techniques

Secure Coding: Preventing Buffer Overflows at the Source

Detecting Buffer Overflows in Practice

There’s more to understand.