Protection Domains

If protection domains were static prisons, processes trapped forever in their initial privilege set, operating systems would be far simpler—but also far less useful. The power of modern protection systems lies in controlled domain switching: the ability for a process to transition from one protection domain to another under carefully enforced rules.

Every time you execute a sudo command, make a system call, attach a debugger, or run a setuid binary, your process crosses a domain boundary. These transitions are among the most security-critical operations in computing. A bug in domain switching can grant unlimited power to unprivileged code; an overly restrictive implementation can make legitimate operations impossible.

Understanding domain switching is understanding the precise mechanisms by which privilege is gained, exercised, and relinquished.

Protection domains provide isolation, but isolation alone is insufficient for practical computing. Programs must interact with the kernel, with privileged system services, and with each other. Domain switching enables these interactions while maintaining security.

The Fundamental Tension:

• User programs need kernel services — Opening files, allocating memory, creating processes all require kernel privileges
• The kernel cannot trust user code — If user code could directly invoke kernel functions, it could corrupt system state
• Temporary privilege is often needed — Password changes require writing to /etc/shadow, but normal users shouldn't have permanent write access
• Debugging requires access to debuggee — A debugger must read/write another process's memory, violating normal isolation

Domain switching resolves these tensions by providing controlled, auditable, revocable transitions between privilege levels.

Domain switching requires coordination between hardware and software. The basic phases are:

Phase 1: Switch Request

The currently executing code requests or triggers a domain transition. This may be:

• Explicit: A system call instruction (syscall, svc, int 0x80)
• Implicit: A hardware exception (page fault, divide by zero)
• External: An interrupt from a device

Phase 2: Privilege Verification

Before the switch occurs, the system verifies the transition is permitted:

• Does the current domain have "switch" rights to the target domain?
• Is the entry point valid (not arbitrary code in the target domain)?
• Are the access conditions met (correct trap gate, valid syscall number)?

Phase 3: Context Save

The CPU and OS save the current domain's execution context:

• Program counter (return address)
• Stack pointer
• CPU flags and registers
• Memory management state (page table base)

Phase 4: Domain Transition

The actual privilege level change occurs:

• CPU privilege level changes (ring 3 → ring 0)
• Stack switches to new domain's stack
• Memory mapping may change
• New domain's code begins execution

Phase 5: Context Restore (on return)

When the target domain completes, the original context is restored and execution resumes in the original domain.

Modern CPUs provide hardware mechanisms to enforce domain boundaries and enable controlled transitions. Without hardware support, software-only protection could be bypassed by malicious code.

x86/x64 Privilege Transitions:

Entering Ring 0 (User → Kernel):

Returning to Ring 3 (Kernel → User):

The SYSCALL/SYSRET Fast Path:

Modern 64-bit systems use syscall/sysret for performance:

; User-space system call invocation
mov rax, 1          ; syscall number (write)
mov rdi, 1          ; fd = stdout
mov rsi, msg        ; buffer address
mov rdx, len        ; buffer length
syscall             ; ENTER KERNEL DOMAIN
; rax now contains return value

When syscall executes:

1. RCX ← RIP (save user instruction pointer)
2. R11 ← RFLAGS (save user flags)
3. RIP ← IA32_LSTAR MSR (jump to kernel entry point)
4. CS ← kernel code segment (privilege = 0)
5. SS ← kernel stack segment
6. RFLAGS masked (interrupts disabled per IA32_FMASK)

Note what does NOT happen automatically:

• Stack is not switched (kernel must do this in software)
• Other registers are not saved (kernel must preserve)
• Arguments are not validated (kernel must check everything)

Not all domain switches involve hardware privilege changes. Software-only domain switching occurs when the OS changes a process's effective privileges without a ring transition.

Unix setuid/setgid Mechanism:

The most common software domain switch is executing a setuid binary. When a file has the setuid bit set, executing it changes the process's effective UID to the file's owner.

$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root 68208 Jan 1 2024 /usr/bin/passwd
    ^
    setuid bit (s in owner execute position)

When a regular user executes /usr/bin/passwd:

1. Kernel performs normal execve() processing
2. Kernel notices setuid bit on the executable
3. Process's effective UID changes from user's UID to root (0)
4. Process now operates in root's domain (can write /etc/shadow)
5. passwd program drops privileges when done (changes eUID back)

This is entirely software-based—the CPU privilege level doesn't change; user code still runs in Ring 3. But the kernel's access control checks now see root privileges.

A critical security requirement for domain switching is that entry into a higher-privilege domain must occur only at controlled entry points. If an attacker could jump to arbitrary code in the kernel, protection would be meaningless.

The Gate Concept:

A gate is a controlled entry point into a protected domain. It specifies:

1. The address where execution begins (not attacker-controlled)
2. The target privilege level
3. Stack behavior (which stack to use)
4. Any additional checks (is the call allowed from this source domain?)

x86 Gate Types:

Modern Syscall Entry:

Instead of gates, modern systems use Model-Specific Registers (MSRs):

IA32_LSTAR MSR = address of syscall entry point
IA32_STAR MSR  = segment selectors for syscall transitions
IA32_FMASK MSR = flags to mask on syscall entry

Only the kernel can write these MSRs, so user code cannot redirect syscall entry.

The System Call Table:

Even with a controlled entry point, the kernel must dispatch to the correct handler. The syscall number provided by user code indexes into a table of handler functions:

// Kernel syscall table (simplified)
const syscall_fn_t sys_call_table[] = {
    [0] = sys_read,
    [1] = sys_write,
    [2] = sys_open,
    [3] = sys_close,
    // ... hundreds more
};

// Entry point code
void syscall_entry(struct pt_regs *regs) {
    long nr = regs->rax;  // syscall number from user
    
    if (nr >= NR_syscalls || nr < 0) {
        regs->rax = -ENOSYS;  // Invalid syscall number
        return;
    }
    
    regs->rax = sys_call_table[nr](regs);  // Dispatch
}

Note the bounds check—without it, a malicious syscall number could cause out-of-bounds access.

One of the most critical aspects of domain switching is stack management. The stack contains return addresses, local variables, and sensitive data. Using the wrong stack in the wrong domain is a catastrophic security vulnerability.

Why Separate Stacks Are Required:

1. User stacks are untrusted — Kernel cannot store sensitive data where user code might read it
2. User stacks may be invalid — User could have munmapped their stack or corrupted the pointer
3. Stack limits differ — Kernel needs guaranteed stack space; user stacks may be exhausted
4. Isolation requirements — Kernel data on stack must not leak to user space

The Stack Switch Process:

When transitioning from Ring 3 to Ring 0 on x86-64:

1. CPU loads new stack pointer from the Task State Segment (TSS)
2. CPU pushes user SS, RSP, RFLAGS, CS, RIP onto kernel stack
3. Optionally, CPU pushes error code (for relevant exceptions)
4. Kernel code continues on kernel stack

On return (IRET):

1. CPU pops RIP, CS, RFLAGS, RSP, SS from kernel stack
2. Execution resumes in user mode with user stack

Per-CPU and Per-Process Kernel Stacks:

Modern kernels maintain multiple stacks for different purposes:

• Per-process kernel stack: Each process has a small kernel stack (8-16KB) for syscall handling
• Per-CPU interrupt stack: Each CPU has a dedicated interrupt stack for IRQ handling
• Per-CPU IST stacks: Interrupt Stack Table entries for critical exceptions (double fault, NMI, machine check)

Why IST Stacks?

Some exceptions (like double fault) can occur even if the kernel stack is corrupted. IST entries provide known-good stacks that are always valid:

// TSS IST entries (per-CPU)
struct tss_struct {
    // ...
    u64 ist[7];  // 7 Interrupt Stack Table entries
    // IST1: Double Fault stack
    // IST2: NMI stack  
    // IST3: Debug stack
    // IST4: Machine Check stack
    // ...
};

Beyond the mechanism of domain switching, operating systems must define policies governing when and how transitions are permitted.

Policy Question 1: Who Can Enter Which Domains?

Not all domain transitions are permitted. The access matrix includes domains as objects, with a "switch" or "enter" right:

              │ Kernel Domain │ Debug Domain │ Admin Domain │
──────────────┼───────────────┼──────────────┼──────────────┤
 User Domain  │ Enter(syscall)│ -            │ -            │
──────────────┼───────────────┼──────────────┼──────────────┤
 Admin Domain │ Enter(syscall)│ Enter        │ -            │
──────────────┼───────────────┼──────────────┼──────────────┤
 Kernel Domain│ -             │ Enter        │ Enter        │
──────────────┴───────────────┴──────────────┴──────────────┘

Only admins can enter the debug domain; only the kernel can enter the admin domain; everyone can enter the kernel domain (via syscall).

Policy Question 2: What Data Crosses the Boundary?

When domains switch, what happens to register contents, memory mappings, and other state?

Conservative approaches:

• Clear all registers on domain entry (prevent information leakage)
• Switch page tables completely (prevent cross-domain access)
• Flush CPU caches (prevent side-channel attacks)

Performance-oriented approaches:

• Preserve argument registers (pass syscall arguments)
• Map kernel into all address spaces (no TLB flush on switch)
• Keep caches intact (benefit from temporal locality)

KPTI (Kernel Page Table Isolation):

Modern systems use KPTI to mitigate Meltdown-class attacks. The user-mode page tables contain minimal kernel mappings—just enough for the syscall entry point. Upon entry, the kernel switches to a different page table with full kernel mappings:

User mode page table:
├── User space: Fully mapped
└── Kernel space: Only entry trampoline mapped

Kernel mode page table:
├── User space: Fully mapped (for copying data)
└── Kernel space: Fully mapped

Domain switching is one of the most security-sensitive operations in an operating system. Historical vulnerabilities illustrate the subtleties involved:

Class 1: Improper Privilege Retention

Failing to drop privileges after temporary elevation:

// VULNERABLE: setuid program
int main(int argc, char *argv[]) {
    open_privileged_resource();  // Needs root
    // BUG: Never dropped privileges!
    execute_user_command(argv[1]);  // Runs as root!
}

Class 2: Race Conditions (TOCTOU)

Time-of-check to time-of-use races during domain switch:

// VULNERABLE: Kernel syscall handler
int sys_read(int fd, char *buf, size_t count) {
    if (!access_ok(VERIFY_WRITE, buf, count))  // CHECK
        return -EFAULT;
    // Another thread remaps 'buf' to kernel memory!
    copy_to_user(buf, kernel_data, count);     // USE
}

Class 3: Uninitialized Data Leakage

Kernel stack may contain sensitive data from previous operations:

// VULNERABLE: Stack leak
struct response {
    int status;
    char data[64];
};

int sys_getinfo(struct response *user_resp) {
    struct response resp;
    resp.status = get_status();  // data[] not initialized!
    copy_to_user(user_resp, &resp, sizeof(resp));
    // Leaks previous kernel stack contents in resp.data
}

We've explored the mechanisms, policies, and pitfalls of domain switching. Let's consolidate the key insights:

What's Next:

We've seen how domains are defined and how switching between them works. Now we'll examine protection rings—the hierarchical domain model implemented in hardware by most processors. Protection rings provide a concrete, efficient implementation of the domain concepts we've discussed.