Loading content...
In the von Neumann architecture that underlies most computers, code and data share the same memory space. This elegant simplicity enabled programmable computing but created a fundamental security weakness: if an attacker can write data to memory, they might be able to execute it as code.
For decades, this was precisely how most exploitation worked. The classic buffer overflow attack wrote malicious machine instructions (shellcode) onto the stack, then redirected the return address to jump into the stack. The processor, unable to distinguish code from data, obediently executed the attacker's instructions.
Data Execution Prevention (DEP)—also known as NX (No-Execute) on Linux and XD (Execute Disable) on Intel—draws a line that should always have existed. Memory pages are explicitly marked as either executable (can run as code) or non-executable (data only). The stack, heap, and other data regions become non-executable by default. Any attempt to run instructions from these regions triggers a hardware exception, immediately terminating the attack.
First introduced by AMD as NX bit in 2003 (and Intel as XD bit in 2004), DEP quickly became a standard defense. Windows XP SP2, Linux 2.6.8, and macOS Tiger all adopted hardware DEP. Today, it's enabled by default on virtually every modern system, and shellcode injection—the dominant attack technique of the 1990s and early 2000s—has been rendered obsolete against modern targets.
Yet DEP's story doesn't end with victory. Attackers responded with Return-Oriented Programming (ROP) and other code reuse techniques that execute existing legitimate code in malicious sequences. Understanding DEP's power—and its limitations—reveals the ongoing evolution of attack and defense.
By the end of this page, you will understand: • The memory corruption problem that DEP addresses • Hardware NX/XD bit implementation at the CPU level • Software-emulated DEP and its limitations • How operating systems configure and enforce DEP • Page table entries and permission bits • The rise of ROP as a DEP bypass technique • DEP's role in the modern security stack
Before DEP, the classic exploitation technique was code injection. The attacker exploited a memory corruption vulnerability (typically a buffer overflow) to:
This was devastatingly effective because nothing in the system distinguished between legitimate code and attacker-supplied data. The CPU simply executed whatever the instruction pointer pointed to.
1234567891011121314151617181920212223242526272829303132333435363738
// Classic stack-based shellcode injection (PRE-DEP era) void vulnerable(char *input) { char buffer[256]; strcpy(buffer, input); // Buffer overflow vulnerability} // Attack payload structure:// ┌─────────────────────────────────────────────────────────────┐// │ SHELLCODE (machine instructions) │ PADDING │ RET ADDRESS │// │ execve("/bin/sh", ...) │ 'AAAA' │ → shellcode │// └─────────────────────────────────────────────────────────────┘// ↑// Points back to start of buffer // When vulnerable() returns:// 1. CPU pops RET ADDRESS into instruction pointer// 2. RET ADDRESS points to buffer[] on the stack// 3. CPU executes SHELLCODE as if it were legitimate code// 4. Shellcode spawns a shell → ATTACK SUCCESSFUL // Example x86 shellcode (Linux execve /bin/sh):char shellcode[] = "\x31\xc0" // xor eax, eax "\x50" // push eax "\x68\x2f\x2f\x73\x68" // push "//sh" "\x68\x2f\x62\x69\x6e" // push "/bin" "\x89\xe3" // mov ebx, esp "\x50" // push eax "\x53" // push ebx "\x89\xe1" // mov ecx, esp "\x99" // cdq "\xb0\x0b" // mov al, 11 (sys_execve) "\xcd\x80"; // int 0x80 // Attacker crafts input:// [shellcode | padding | address of buffer on stack]// The CPU happily executes shellcode from the stack!The fundamental issue was memory permission model inadequacy. Traditional permissions were:
There was no Execute (X) permission at the hardware level. If memory was readable, it was implicitly executable. The CPU would fetch and execute instructions from any address—stack, heap, read-only data sections, anywhere.
| Memory Region | Pre-DEP Permissions | With DEP Permissions | Attack Impact |
|---|---|---|---|
| Code (.text) | R-X (implicit) | R-X (explicit) | No change |
| Data (.data) | RW- (but implicitly executable) | RW- (truly non-executable) | No code injection here |
| Stack | RW- (but implicitly executable) | RW- (truly non-executable) | Stack shellcode blocked! |
| Heap | RW- (but implicitly executable) | RW- (truly non-executable) | Heap spraying blocked! |
| BSS | RW- (but implicitly executable) | RW- (truly non-executable) | Protected |
Before DEP, any buffer overflow that allowed writing and redirecting to writable memory was instantly exploitable. The stack was the most common target (hence "stack smashing"), but heap overflows, format string attacks, and numerous other vulnerabilities all led to code injection. DEP eliminated this entire class of attack.
Modern DEP relies on hardware support in the processor's memory management unit (MMU). Both AMD and Intel introduced dedicated bits in the page table entries to control execution permissions:
These are functionally equivalent—a single bit in each page table entry that tells the CPU: "Do not execute instructions from this page."
64-bit Page Table Entry (PTE) Structure:┌─────┬─────────────────────────────────────────────────────────────────────┐│ Bit │ Purpose │├─────┼─────────────────────────────────────────────────────────────────────┤│ 63 │ NX (No-Execute) - 1 = page cannot be executed │ ★ DEP control│62:52│ Available for software use ││51:M │ Reserved, must be 0 ││ M:12│ Physical Page Frame Number (PFN) │ Address bits│11:9 │ Available for software use ││ 8 │ Global page ││ 7 │ Page size (4KB vs 2MB/1GB) ││ 6 │ Dirty - page has been written ││ 5 │ Accessed - page has been read/written ││ 4 │ Page Cache Disable (PCD) ││ 3 │ Page Write-Through (PWT) ││ 2 │ User/Supervisor - 0=kernel only, 1=user accessible ││ 1 │ Read/Write - 0=read-only, 1=read-write ││ 0 │ Present - 1=page is in physical memory │└─────┴─────────────────────────────────────────────────────────────────────┘ Example PTE values:0x8000000012345003 = NX set, page at frame 0x12345, present, user, read-only ^ NX bit set (bit 63) → CANNOT EXECUTE from this page! 0x0000000012345003 = NX clear, page at frame 0x12345, present, user, read-only ^ NX bit clear (bit 63) → CAN execute from this pageWhen the CPU fetches an instruction, the MMU performs the following checks:
If the NX bit is set and the CPU attempts to fetch an instruction from that page, a #PF (Page Fault) exception is raised with a special error code indicating an instruction fetch violation.
12345678910111213141516171819202122232425262728293031323334353637
// What happens when code tries to execute from NX page // Scenario: Attacker redirects execution to stack (which has NX set) // 1. CPU instruction fetch:// - EIP/RIP = 0x7ffdxxxx (address on stack)// - CPU looks up page table entry for this address // 2. MMU finds PTE:// - Present = 1 (page exists)// - User = 1 (accessible)// - NX = 1 ← PROBLEM! // 3. CPU raises Page Fault (#PF):// - Error code bits:// - Bit 0 = 1 (protection violation, not missing page)// - Bit 1 = 0 (read operation — instruction fetch is a "read")// - Bit 2 = 1 (user mode access)// - Bit 4 = 1 (instruction fetch) ← KEY: This bit indicates fetch!// - Error code = 0x15 (binary: 10101) // 4. OS Page Fault Handler:void page_fault_handler(struct trap_frame *frame, uint64_t error_code) { uint64_t fault_addr; asm volatile("mov %%cr2, %0" : "=r"(fault_addr)); // Check if this was an instruction fetch to NX page if (error_code & (1 << 4)) { // Bit 4: Instruction fetch // DEP VIOLATION DETECTED! printk(KERN_ALERT "DEP violation! Attempted execution at %lx\n", fault_addr); // Terminate the offending process send_signal(current, SIGSEGV); // or on Windows: STATUS_ACCESS_VIOLATION exception }}Hardware DEP (NX/XD) is vastly superior to software alternatives. The check happens in hardware during every instruction fetch, with zero performance overhead for normal operations. Software emulation requires trapping on page faults and examining the instruction pointer—significantly slower and less reliable.
While the CPU provides the NX bit mechanism, operating systems must properly configure page tables to leverage it. Each OS has different approaches to enabling and managing DEP.
Linux enables NX by default on 64-bit systems (since kernel 2.6.8, 2004). On 32-bit systems with NX support (PAE mode), it's also enabled by default.
1234567891011121314151617181920212223242526272829
# Check if NX is enableddmesg | grep -i 'execute disable\|NX'# NX (Execute Disable) protection: active # Check CPU flaggrep -o 'nx' /proc/cpuinfo | head -1# nx # View memory map permissionscat /proc/self/maps# Address Perms ... Path# 559a12340000-... r-xp ... /bin/cat <- Code: r-x (executable)# 559a12350000-... r--p ... /bin/cat <- Read-only data# 559a12360000-... rw-p ... /bin/cat <- Writable data (NO x!)# 7ffd12340000-... rw-p ... [stack] <- Stack (NO x!)# 7f1234500000-... r-xp ... libc.so.6 <- libc code (executable) # Note: 'x' appears ONLY for code sections# Stack and heap have 'rw-' (no execute permission) # ELF marking: The executable specifies which segments need executereadelf -l /bin/cat | grep -E 'GNU_STACK|LOAD'# Type Offset VirtAddr Flags# LOAD 0x000000 0x00000000 R E <- Code, Execute# LOAD 0x001000 0x00001000 RW <- Data, No execute# GNU_STACK 0x000000 0x00000000 RW <- Stack policy: No execute # GNU_STACK with no 'E' flag = NX stack# If you see 'RWE', the binary was compiled without NX!The strongest DEP configuration enforces W^X (Write XOR Execute): a memory page can be writable OR executable, but NEVER both at the same time. This prevents attackers from writing code to executable memory.
12345678910111213141516171819202122232425262728293031323334353637
// W^X in action #include <sys/mman.h>#include <stdio.h> int main() { // Allocate page with write permission void *mem = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); // Write code to the page (like an attacker would) unsigned char shellcode[] = { 0xeb, 0xfe }; // infinite loop memcpy(mem, shellcode, sizeof(shellcode)); // Try to make it executable (attacker's next step) // On strict W^X systems, this FAILS! int result = mprotect(mem, 4096, PROT_READ | PROT_EXEC); if (result < 0) { perror("mprotect"); // "Operation not permitted" on strict W^X return 1; } // Even if mprotect succeeds, some systems track that the page // was previously writable and refuse execution // Cast to function and call void (*func)(void) = (void (*)(void))mem; func(); // On strict W^X: SIGKILL / SIGBUS / crash return 0;} // W^X enforcement levels:// 1. Basic DEP: Stack/heap non-exec, but mprotect can add X// 2. Strict W^X: Cannot change page from W to X// 3. Secure mode: No anonymous memory can ever be X (only signed code)JIT compilers (like those in web browsers, Java, and .NET) need to generate and execute code at runtime—seemingly violating W^X. Modern JITs handle this by: (1) Writing code to an RW page, (2) Calling mprotect() to change to RX before execution, (3) Never having RWX pages. Some systems require explicit entitlements for this capability.
Security professionals must verify that DEP is properly enabled and configured. Here are techniques for testing DEP on various platforms:
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253
// Test program to verify DEP is working// If DEP is enabled, this program should crash with SIGSEGV #include <stdio.h>#include <string.h>#include <signal.h>#include <setjmp.h> static sigjmp_buf jump_buffer;static volatile sig_atomic_t got_signal = 0; void signal_handler(int sig) { got_signal = 1; siglongjmp(jump_buffer, 1);} int main() { // Machine code for "return 42": mov eax, 42; ret unsigned char code[] = { 0xB8, 0x2A, 0x00, 0x00, 0x00, // mov eax, 42 0xC3 // ret }; // Place code on the stack (normally non-executable with DEP) unsigned char stack_code[16]; memcpy(stack_code, code, sizeof(code)); // Set up signal handler signal(SIGSEGV, signal_handler); signal(SIGBUS, signal_handler); if (sigsetjmp(jump_buffer, 1) == 0) { // Try to execute code from stack printf("Attempting to execute code from stack...\n"); int (*func)(void) = (int (*)(void))stack_code; int result = func(); // If we get here, DEP is NOT working! printf("WARNING: DEP FAILED! Stack was executable.\n"); printf("Returned value: %d\n", result); return 1; } else { // Signal caught - DEP is working! printf("SUCCESS: DEP blocked execution from stack.\n"); printf("System is protected against stack-based shellcode.\n"); return 0; }} // Compile and test:// gcc -o dep_test dep_test.c// ./dep_test// Expected output: "SUCCESS: DEP blocked execution from stack."The checksec tool (part of pwntools and available standalone) provides quick visibility into binary security features:
1234567891011121314151617181920212223242526272829303132333435
# Install checksecpip install pwntools # or: apt install checksec # Check a binarychecksec --file=/bin/ls # Output:# Arch: amd64-64-little# RELRO: Full RELRO# Stack: Canary found# NX: NX enabled ← DEP IS ACTIVE!# PIE: PIE enabled # Check a vulnerable binarychecksec --file=./vulnerable_app # Output (bad example):# Arch: i386-32-little# RELRO: No RELRO# Stack: No canary found# NX: NX disabled ← NO DEP! Vulnerable to shellcode!# PIE: No PIE # Force DEP during compilationgcc -z noexecstack -o secure_app app.cchecksec --file=./secure_app# NX: NX enabled # Check running processes for NX (Linux)for pid in $(pgrep -x apache2); do echo "PID $pid:" cat /proc/$pid/maps | grep -E '\[stack\]|\[heap\]' | awk '{print $2, $6}'done# rw-p [stack] ← No 'x' = DEP active# rw-p [heap] ← No 'x' = DEP activeSome older applications require executable stacks for legitimate reasons (self-modifying code, nested functions in GCC, etc.). These are marked with PT_GNU_STACK showing RWE permissions. While DEP must be disabled for these applications, they should be isolated and considered high-risk. Modern development should avoid any need for executable stacks.
DEP eliminated code injection but not exploitation. Attackers responded with Return-Oriented Programming (ROP)—a technique that chains together small pieces of existing legitimate code ("gadgets") to perform arbitrary computation without injecting any new code.
ROP works because DEP only prevents execution of new code. Existing code in libraries and the executable is marked executable. By carefully selecting short instruction sequences ending in ret, attackers can string together Turing-complete programs.
Return-Oriented Programming (ROP) Concept━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Traditional Shellcode:┌──────────────────────────────────────────┐│ Injected malicious code on stack │ ← DEP BLOCKS THIS│ mov rdi, "/bin/sh" ││ mov rax, 59 ; sys_execve ││ syscall │└──────────────────────────────────────────┘ ROP Attack:┌──────────────────────────────────────────┐│ Stack contains ONLY addresses + data │ ← DEP allows data on stack│ [addr of gadget 1] ││ [addr of gadget 2] ││ [addr of gadget 3] ││ ["/bin/sh" string or pointer to it] ││ [addr of system() in libc] │└──────────────────────────────────────────┘ Gadgets are sequences ending in RET within existing code:┌─────────────────────────────────────────────────────────┐│ Gadget 1 (found at 0x4005a3 in executable): ││ pop rdi ; load value from stack into rdi ││ ret ; return to next gadget │├─────────────────────────────────────────────────────────┤│ Gadget 2 (found at 0x400550 in executable): ││ pop rsi ; load next stack value into rsi ││ pop r15 ; consume junk ││ ret ; return to next gadget │├─────────────────────────────────────────────────────────┤│ Gadget 3 (found at 0x7f123456789a in libc): ││ xor edx, edx ; clear rdx ││ ret ; return to next gadget │└─────────────────────────────────────────────────────────┘ Execution flow:1. Overflow corrupts return address → points to Gadget 12. Gadget 1 executes: pop rdi (gets "/bin/sh" addr), ret3. ret pops next address → Gadget 24. Gadget 2 executes: pop rsi, pop r15, ret5. ret pops → system() address6. system("/bin/sh") executes! ALL executed instructions were in legitimate code sections.DEP saw nothing wrong—no execution from stack or heap!Finding useful gadgets and chaining them requires tooling:
123456789101112131415161718192021222324252627282930
# Find ROP gadgets with ROPgadgetpip install ROPgadgetROPgadget --binary /usr/lib/libc.so.6 | head -20 # Output:# 0x00000000000273e2 : pop rdi ; ret# 0x0000000000027529 : pop rsi ; pop r15 ; ret# 0x0000000000044808 : pop rdx ; pop r12 ; ret# 0x00000000001564da : syscall ; ret# ... # In pwntools (Python):from pwn import * elf = ELF('/usr/lib/libc.so.6')rop = ROP(elf) # Find specific gadgetspop_rdi = rop.find_gadget(['pop rdi', 'ret'])[0]print(f"pop rdi; ret at offset: {hex(pop_rdi)}") # Build a ROP chain to call system("/bin/sh")rop.call('system', [next(elf.search(b'/bin/sh'))])print(rop.dump())# 0x0000: 0x273e2 pop rdi; ret# 0x0008: 0x1b3d88 [arg0] rdi = 1815944 (ptr to "/bin/sh")# 0x0010: 0x50d60 system # Generate the raw payloadpayload = rop.chain()Despite ROP, DEP remains essential. ROP attacks require ASLR bypass (to find gadget addresses) and are significantly more complex than simple shellcode injection. DEP raised the bar, forcing attackers to combine multiple techniques. Defense in depth means each layer adds cost to exploitation.
DEP is most effective as part of a layered defense strategy. Each mitigation addresses different attack stages and techniques. Together, they create a formidable barrier:
| Mitigation | What It Prevents | How Attackers Bypass | Complementary Defense |
|---|---|---|---|
| DEP/NX | Executing injected code | ROP (code reuse attacks) | CFI, code signing |
| ASLR | Predicting memory addresses | Information leaks | Remove info leaks, re-randomization |
| Stack Canaries | Overwriting return address | Info leak + format string | Safe stack, shadow stack |
| RELRO | Overwriting GOT | Other writable targets | Full RELRO + CFI |
| CFI | Invalid control flow | Fine-grained CFI bypass | Hardware-enforced CFI |
| Shadow Stack | Return address corruption | Non-return control flow | Complete CFI |
Recognizing that DEP pushed attackers to ROP, modern systems have introduced additional protections:
Control Flow Integrity (CFI): Validates that indirect calls and jumps target legitimate function entries. ROP gadgets typically start in the middle of functions, so CFI can detect them.
Shadow Stack: Hardware-maintained copy of return addresses. When a function returns, the hardware compares the stack return address against the shadow stack. Any mismatch (from ROP) triggers an exception.
Intel CET (Control-flow Enforcement Technology): Hardware-based shadow stack and indirect branch tracking. Available in 11th Gen Intel Core and later. Provides performance-free protection.
1234567891011121314151617181920212223
# Checking for modern protections # Intel CET supportgrep -o 'cet\|shstk\|ibt' /proc/cpuinfo | sort -u# shstk - Shadow Stack support# ibt - Indirect Branch Tracking # Enable CET in compilation (GCC 11+)gcc -fcf-protection=full -o secure app.c # Check binary for CFIreadelf -n ./secure | grep -i cet# Properties: x86 feature: CET, SHSTK # ARM Pointer Authentication (PAC)# Automatically enabled on Apple Silicon# Signs return addresses cryptographically# Any modified return address fails signature check # Cost of CFI/CET:# - 1-5% overhead (much less than software CFI)# - Shadow stack uses ~4KB per thread# - Requires OS and linker supportDEP triggered an arms race that ultimately benefited defenders. Each attacker response (ROP, JOP, etc.) prompted new defenses (CFI, CET, PAC). The current state—hardware-enforced control flow—is far more secure than the pre-DEP era. Exploitation now requires chaining multiple advanced techniques, drastically raising the bar.
DEP represents one of the most impactful security mitigations in computing history. By enforcing the fundamental principle that data should not be executable, it eliminated entire classes of attacks.
What's Next:
DEP prevents execution of injected code. ASLR makes addresses unpredictable. Stack canaries detect control flow hijacking. But how do compilers further protect the stack and validate memory safety? The next page explores Stack Protection mechanisms beyond canaries—including safe stack, shadow stacks, and other compiler-based defenses.
You now have a comprehensive understanding of DEP—its hardware implementation, OS integration, verification methods, and its role in the modern security stack. This foundation is essential for understanding both offensive security research and defensive system design.