System Call Mechanism
Trap Instruction
Read aloud is unavailable in this browser.
The Deliberate Exception
In the previous page, we explored how the CPU enforces the user-kernel boundary and allows transitions only through controlled entry points. Now we examine the deliberate mechanism by which user code requests these transitions: the trap instruction.
Unlike exceptions caused by errors (divide by zero, page faults) or asynchronous hardware interrupts (keyboard, disk), a trap is a synchronous, intentional event. User code explicitly invokes it to request a service from the operating system. This deliberate nature distinguishes the system call trap from other privilege escalation mechanisms and has profound implications for how applications and kernels interact.
By the end of this page, you will understand the semantics of trap instructions, how they differ from interrupts and exceptions, the various trap instruction implementations across major CPU architectures, and the precise sequence of CPU state changes that occurs during trap execution.
Understanding Traps: Interrupts, Exceptions, and Traps
Before diving into trap instructions specifically, we must understand the taxonomy of events that cause privilege transitions. The terms 'interrupt,' 'exception,' and 'trap' are often used interchangeably, but they have distinct technical meanings:
Interrupts (Hardware Interrupts)
- Source: External hardware devices (keyboard, disk controller, network card, timer)
- Timing: Asynchronous—occur independently of currently executing instructions
- Resume: Execution continues at the instruction following the interrupted point
- Example: Disk controller signals data is ready; timer fires for scheduling
Exceptions (Faults, Traps, Aborts)
- Source: CPU detects an abnormal condition caused by instruction execution
- Timing: Synchronous—directly caused by executing an instruction
- Resume: Depends on exception type:
- Faults: Can be corrected; instruction re-executed (e.g., page fault)
- Traps: Intentional; execution continues at next instruction (e.g., INT 3)
- Aborts: Unrecoverable; process terminated (e.g., machine check)
Software Traps (System Calls)
- Source: Explicit trap instruction executed by user code
- Timing: Synchronous—deliberately triggered at a specific instruction
- Resume: Execution continues at the instruction following the trap
- Example: Application calls
read(), which invokesSYSCALLinstruction
| Type | Source | Timing | Cause | Resume Point |
|---|---|---|---|---|
| Hardware Interrupt | External device | Asynchronous | Device signals CPU | Next instruction |
| Fault | CPU detection | Synchronous | Error condition (recoverable) | Faulting instruction (retry) |
| Trap | Explicit instruction | Synchronous | Deliberate software request | Next instruction |
| Abort | CPU detection | Synchronous | Severe error (unrecoverable) | Process terminated |
Different CPU vendors use different terminology. Intel uses 'interrupt' for all these events collectively. ARM uses 'exception' for both synchronous and asynchronous events. The conceptual distinctions remain the same—what matters is understanding when and why the CPU transitions to a different privilege level.
The Semantics of Trap Instructions
A trap instruction is designed with specific semantics that distinguish it from other instructions:
1. Atomic Privilege Transition
The trap must atomically switch the CPU from user mode to kernel mode. There cannot be a 'half-transitioned' state where the CPU is partly in user mode and partly in kernel mode. This atomicity prevents race conditions and security vulnerabilities.
2. Controlled Destination
The trap instruction does not specify where to jump—the destination is determined by tables configured by the operating system. User code cannot choose to jump to arbitrary kernel addresses; it can only trigger a transition to predefined handler locations.
3. State Preservation
The CPU must preserve sufficient state for the kernel to:
- Identify what the user was doing when the trap occurred
- Access the user's register values (including arguments to the system call)
- Eventually return to user mode and continue where it left off
4. Interrupt Disabling (Optional)
Different trap mechanisms have different interrupt behavior. Some disable interrupts during the transition (like interrupt gates), while others leave interrupts enabled (like trap gates). The choice affects kernel complexity and latency.
5. Stack Safety
The transition mechanism must either switch to a known-safe kernel stack or provide a way for the kernel to immediately do so. Running kernel code on an untrusted user stack is a severe security vulnerability.
- •Deterministic — The same trap instruction with the same parameters always triggers the same transition (barring configuration changes).
- •Non-returnable by user — Once the trap executes, user code loses control until the kernel explicitly returns.
- •Low overhead — Modern CPUs optimize trap instructions to minimize the cycle count for privilege transitions.
- •Secure — User code cannot exploit the trap to gain unauthorized access or corrupt kernel state.
- •Bidirectional (with return instruction) — A corresponding mechanism (SYSRET, IRET) allows the kernel to return to user mode.
x86 Trap Instructions: From INT to SYSCALL
The x86 architecture has evolved multiple trap mechanisms, each addressing limitations of its predecessors:
INT n (Software Interrupt)
The original mechanism, inherited from the 8086. The INT instruction with an immediate operand triggers a software interrupt:
MOV EAX, 1 ; system call number (exit)
MOV EBX, 0 ; exit status
INT 0x80 ; trigger system call
The CPU uses the operand (0x80 in this case) as an index into the Interrupt Descriptor Table (IDT). Historically, Linux used INT 0x80 for 32-bit system calls, and Windows used INT 0x2E.
| Instruction | Year | Bits | Latency | State Saved | Used By |
|---|---|---|---|---|---|
INT 0x80 | 1979 | 16/32 | ~250-500 cycles | All to stack via IDT | Legacy Linux 32-bit |
SYSENTER | 1997 | 32 | ~100-200 cycles | Minimal (CS, EIP only) | Modern Linux 32-bit, Windows |
SYSCALL | 2003 | 64 | ~50-100 cycles | RCX=RIP, R11=RFLAGS | Modern Linux 64-bit, Windows 64-bit |
SYSENTER/SYSEXIT (Pentium II and later)
Intel introduced SYSENTER as a faster alternative to INT. Unlike INT, which uses the IDT and involves stack operations during the transition, SYSENTER uses Model-Specific Registers (MSRs) to define the entry point:
IA32_SYSENTER_CS: Kernel code segment selectorIA32_SYSENTER_EIP: Kernel entry point addressIA32_SYSENTER_ESP: Kernel stack pointer
SYSENTER does NOT save user state to the stack—it only loads the new CS, EIP, and ESP. The kernel entry code must manually save user registers.
SYSCALL/SYSRET (AMD64/x86-64)
For 64-bit mode, AMD introduced SYSCALL, which became the standard for x86-64 Linux and Windows:
IA32_STAR: CS/SS selectors for kernel and user modeIA32_LSTAR: Kernel entry point (RIP)IA32_FMASK: RFLAGS mask (bits to clear)
SYSCALL saves user RIP to RCX and user RFLAGS to R11. It does NOT switch stacks—the kernel entry code must load the kernel stack manually.
123456789101112131415161718
SYSCALL: // Save user state to registers (NOT stack) RCX ← RIP // User return address saved to RCX R11 ← RFLAGS // User flags saved to R11 // Load kernel segments from MSR CS.selector ← IA32_STAR[47:32] // Kernel CS from IA32_STAR SS.selector ← IA32_STAR[47:32] + 8 // Kernel SS // Clear flags specified by mask RFLAGS ← RFLAGS AND NOT(IA32_FMASK) // Jump to kernel entry point RIP ← IA32_LSTAR // Kernel entry address from MSR // CPL is now 0 (kernel mode) // RSP is UNCHANGED - still points to user stack! // Kernel must immediately load safe stackUnlike INT, the SYSCALL instruction does NOT automatically switch the stack pointer. RSP still points to user memory when kernel execution begins. This means the very first thing the kernel entry code must do is load a safe kernel stack pointer. Any instructions before this are extremely security-sensitive.
ARM Trap Instructions: SVC and Exception Levels
ARM processors use a different privilege model based on Exception Levels (EL):
- EL0: User applications (unprivileged)
- EL1: Operating system kernel (privileged)
- EL2: Hypervisor (for virtualization)
- EL3: Secure monitor (for TrustZone security)
SVC (Supervisor Call)
The ARM equivalent of x86's SYSCALL is the SVC instruction (formerly called SWI - Software Interrupt in ARM32):
// ARM64 (AArch64) system call example
MOV X8, #93 // System call number (exit)
MOV X0, #0 // Exit status
SVC #0 // Trap to kernel
When SVC executes:
- Exception Syndrome Register (ESR_EL1) is updated with exception cause
- Program Counter is saved to Exception Link Register (ELR_EL1)
- Processor State (PSTATE) is saved to Saved Program Status Register (SPSR_EL1)
- Exception level changes from EL0 to EL1
- Execution continues at the exception vector (defined in VBAR_EL1)
| Register | Purpose | Saved Automatically? |
|---|---|---|
| ELR_EL1 | Return address (user PC) | Yes |
| SPSR_EL1 | User processor state (PSTATE) | Yes |
| ESR_EL1 | Exception syndrome (cause + details) | Yes |
| FAR_EL1 | Faulting address (for memory exceptions) | Yes |
| SP_EL1 | Kernel stack pointer | Used automatically |
| X0-X30 | General purpose registers | No (must save manually) |
Exception Vector Table
Unlike x86's IDT which has hundreds of entries, ARM uses a compact exception vector table with a fixed layout. For AArch64, each exception level has a vector base address register (VBAR_ELx) that points to a table with 16 entries:
| Offset | Exception Type |
|---|---|
| 0x000 | Synchronous, current EL, SP0 |
| 0x080 | IRQ, current EL, SP0 |
| 0x100 | FIQ, current EL, SP0 |
| 0x180 | SError, current EL, SP0 |
| 0x200 | Synchronous, current EL, SPx |
| ... | ... |
| 0x400 | Synchronous, lower EL, AArch64 |
| 0x480 | IRQ, lower EL, AArch64 |
| ... | ... |
When EL0 code executes SVC, the CPU jumps to VBAR_EL1 + 0x400 (synchronous exception from lower EL using AArch64).
ARM64 provides banked stack pointers for each exception level. When transitioning from EL0 to EL1, the processor automatically switches from SP_EL0 to SP_EL1. This is MORE secure than x86's SYSCALL, which leaves RSP pointing to user memory.
RISC-V Trap Instructions: ECALL and Privilege Modes
RISC-V, being a modern clean-slate ISA, provides a straightforward trap mechanism designed for clarity and correctness.
Privilege Modes
- U-mode (User): Unprivileged applications
- S-mode (Supervisor): Operating system kernel
- M-mode (Machine): Firmware, bootloader, hypervisor
ECALL (Environment Call)
The RISC-V trap instruction is ECALL. It requests a service from the next higher privilege level:
# RISC-V Linux system call example
li a7, 93 # System call number (exit)
li a0, 0 # Exit status
ecall # Trap to kernel (U→S) or SBI (S→M)
ECALL behavior:
mepc(orsepcfor S-mode) is set to the ECALL instruction addressmcause(orscause) is set to identify the exception (11 for U→M, 9 for U→S)- Privilege mode changes to M-mode or S-mode
- PC is set to the value in
mtvec(orstvec) trap vector register
1234567891011121314151617181920212223242526272829303132333435363738394041
# RISC-V S-mode trap handler entry (simplified).align 4trap_entry: # At this point: # - sstatus.SPP indicates previous privilege (0=User, 1=Supervisor) # - sepc contains the address of ECALL instruction # - scause contains ECALL_FROM_U (value 8) # - sscratch contains kernel stack pointer (swapped before entry) # Save user registers to kernel stack csrrw sp, sscratch, sp # Swap user SP with kernel SP # Allocate trapframe on kernel stack addi sp, sp, -288 # Space for 36 registers (8 bytes each) # Save user registers sd ra, 0(sp) sd gp, 8(sp) sd tp, 16(sp) sd t0, 24(sp) # ... save remaining registers ... # Save sepc (user return address) csrr t0, sepc sd t0, 256(sp) # Call C handler mv a0, sp # Pass trapframe pointer call handle_exception # Restore and return ld t0, 256(sp) # Load sepc addi t0, t0, 4 # Increment past ECALL instruction csrw sepc, t0 # Restore registers... ld ra, 0(sp) # ... csrrw sp, sscratch, sp # Restore user SP sret # Return to user modeRISC-V was designed with security and simplicity in mind. Unlike x86, there's only one system call instruction (ECALL), and the trap handling mechanism is consistent across all exception types. The sscratch register provides a clean way to store the kernel stack pointer, available immediately upon trap entry.
CPU State Changes During Trap Execution
When a trap instruction executes, the CPU performs a carefully orchestrated sequence of state changes. Understanding these changes is essential for kernel developers and security researchers. Let's examine each phase:
- •The CPU's instruction decoder identifies the trap instruction (SYSCALL, SVC, ECALL, etc.).
- •The instruction is recognized as a privilege-escalating operation requiring special handling.
- •The CPU begins the trap sequence, which is atomic with respect to interrupts and other exceptions.
- •Any pending writes are completed to ensure memory consistency.
- •Program Counter (PC/RIP/EIP): The address of the next instruction is saved. On x86-64 SYSCALL, this goes to RCX; on ARM64 SVC, to ELR_EL1.
- •Processor Flags (RFLAGS/PSTATE): Condition codes and control bits are preserved. x86-64 SYSCALL saves to R11; ARM64 saves to SPSR_EL1.
- •Stack Pointer: Behavior varies by architecture. x86-64 SYSCALL leaves RSP unchanged (dangerous!). ARM64 SVC switches to SP_EL1 automatically.
- •General Purpose Registers: NOT automatically saved on most modern architectures. Kernel entry code must preserve them.
- •Privilege Level Change: CPL (x86) or Exception Level (ARM) is elevated. User mode (Ring 3/EL0) becomes kernel mode (Ring 0/EL1).
- •Segment Selector Update (x86): CS and SS are loaded with kernel segment selectors, enabling access to kernel memory.
- •Memory Protection Change: Page table interpretation may change (SMEP, SMAP effects), and kernel-only mappings become accessible.
- •PC/RIP Load: The program counter is loaded with the kernel entry point address from the appropriate source (IA32_LSTAR, VBAR_ELx, stvec).
- •Flag Modification: Certain flags are modified. SYSCALL clears bits specified by IA32_FMASK (often IF to disable interrupts temporarily).
- •Execution Begins: The CPU fetches and executes the instruction at the new PC—the first instruction of the kernel's trap handler.
The Atomicity Guarantee
This entire sequence is atomic—it cannot be interrupted halfway. If an interrupt arrives during the trap sequence, it is held pending until the trap completes. This atomicity is crucial:
- It prevents race conditions where an interrupt handler might find the CPU in an inconsistent state
- It ensures the kernel entry code always starts from a well-defined state
- It simplifies reasoning about security properties
The atomicity is implemented in microcode or hardwired logic, not in software. It's a fundamental property of the trap instruction itself.
Trap Return Instructions: SYSRET, ERET, and SRET
Every trap instruction has a corresponding return instruction that restores user mode. These return instructions are privileged—only the kernel can execute them—and they reverse the trap's state changes:
| Architecture | Instruction | PC Source | Flags Source | Stack Behavior |
|---|---|---|---|---|
| x86-64 | SYSRET | RCX | R11 | RSP must be set by kernel |
| x86 (legacy) | IRET | Stack pop | Stack pop | Full stack restore |
| ARM64 | ERET | ELR_ELx | SPSR_ELx | Automatic SP switching |
| RISC-V | SRET/MRET | sepc/mepc | sstatus/mstatus | sscratch swap pattern |
123456789101112131415161718
SYSRET: // Verify we're in Ring 0 IF CPL != 0: RAISE #GP(0) // Load user segments from MSR CS.selector ← IA32_STAR[63:48] + 16 // User CS SS.selector ← IA32_STAR[63:48] + 8 // User SS // Restore user state from registers RIP ← RCX // Return address (saved by SYSCALL) RFLAGS ← R11 // Flags (saved by SYSCALL) // Change privilege level CPL ← 3 // Back to user mode // Execution continues at user RIP // RSP must have been restored by kernel before SYSRETSYSRET on Intel CPUs has a subtle security issue: if the return RIP is non-canonical (invalid), the processor raises #GP but does so AFTER loading the user segment selectors but BEFORE actually returning to user mode. This means the #GP handler runs with user GS. Linux works around this by validating RCX and using IRET for potentially problematic returns.
Kernel Responsibilities Before Return
Before executing the return instruction, the kernel must:
-
Restore user registers: General purpose registers that were saved on entry must be restored from the kernel stack.
-
Set return values: System call results (typically in RAX on x86, X0 on ARM) must be set before return.
-
Restore user stack pointer: On x86-64, RSP must be explicitly loaded with the user's stack pointer.
-
Handle signals: If signals are pending, the kernel may need to divert return to a signal handler instead.
-
Update flags: On SYSRET, R11 is restored to RFLAGS, but the kernel may need to modify it to reflect error conditions.
-
Re-enable interrupts: If interrupts were disabled during system call handling, they must be re-enabled (or the return instruction does this automatically).
KEEP LEARNING
Keep learning for free.
Read the complete lesson, save your notes and pick up where you left off with a free account. No card needed.