Protection Domains

Every operating system faces a critical question: "What operations can this piece of code perform, and on which resources?" This isn't merely an academic concern—it's the foundation upon which system security, stability, and multi-tenancy rest.

Consider when you run a web browser. That browser needs to access network sockets, read configuration files, allocate memory, and render graphics. But should it be able to format your hard drive? Read your private SSH keys? Modify kernel data structures? Obviously not. Yet both your browser and the kernel are "just code" running on the same processor.

The abstraction that answers this question—that defines what code can do what to which resources—is called a protection domain. Understanding protection domains is understanding the architectural foundation of OS security itself.

A protection domain (often simply called a domain) is an abstract execution environment that specifies a set of objects and the operations permitted on those objects. Every piece of executing code operates within some protection domain, even if implicitly.

Formal Definition:

A protection domain consists of:

1. A set of objects (files, memory regions, devices, I/O ports, other processes)
2. A set of access rights for each object (read, write, execute, delete, append, own)

Mathematically, a domain D can be represented as:

D = { <O₁, rights₁>, <O₂, rights₂>, ..., <Oₙ, rightsₙ> }

Where each <Oᵢ, rightsᵢ> is a tuple denoting object Oᵢ and the set of operations permitted on it.

Protection domains exist because of a fundamental tension in computing systems:

The Capability Paradox:

• Programs need sufficient privileges to accomplish their legitimate tasks
• Programs must be prevented from performing unauthorized operations
• The same program (say, a web server) needs different privileges at different times

Without protection domains, we would face impossible choices: either grant all programs all privileges (security disaster), or grant no programs any privileges (completely useless system).

Historical Context:

Early computers (1940s-1950s) had no protection domains. A bug in any program could crash the entire system, corrupt any data, or halt the machine. The development of time-sharing systems (1960s) made this untenable—multiple users running multiple programs demanded isolation.

The Multics system (1965) was the first to formalize protection domains with rings and capabilities, directly influencing Unix and all modern operating systems.

Understanding protection domains requires distinguishing them from related concepts:

The Relationship Triangle:

         USERS
        /     \
       /       \
      ↓         ↓
  DOMAINS ←→ PROCESSES

Users are principals—entities that can own resources and have rights. Users persist across sessions.

Processes are instances of program execution. They have a lifetime (start-to-termination).

Domains define the privilege context. They specify what processes can do, not that processes exist.

Unix Example: Three UIDs

Unix processes have three user IDs that determine their domain:

1. Real UID — The user who started the process. Used for accounting.
2. Effective UID — The user whose privileges apply NOW. Determines access rights.
3. Saved UID — Stores former effective UID for later restoration.

The effective UID essentially defines the protection domain. When you run sudo, your process's effective UID becomes root (UID 0), switching you into the kernel's protection domain temporarily.

# Before sudo: Real=1000, Effective=1000 (your domain)
$ sudo whoami
# During sudo: Real=1000, Effective=0 (root domain)
root
# After sudo returns: Real=1000, Effective=1000 (back to your domain)

Protection domains are defined by their rights over objects. Understanding both concepts is essential:

What Are Objects?

In security terms, an object is any system resource that requires controlled access. Objects are passively accessed by subjects (processes executing in domains).

Object Taxonomy:

What Are Access Rights?

An access right (or permission) is a specific operation that may or may not be permitted on an object. Rights are granted to domains, not directly to processes or users.

Universal Rights (across all object types):

• Read — Extract information from the object
• Write — Modify the object's contents or state
• Execute — Use the object as executable code or invoke its functionality

Object-Specific Rights:

• Append — Add to the object without modifying existing content (log files)
• Delete — Remove the object from the system
• Owner — Full control including granting rights to others
• Copy — Duplicate the object
• Control — Perform administrative operations (device ioctls)

Meta-Rights (rights over rights):

• Grant — Give one's own rights to another domain
• Revoke — Remove rights previously granted
• Delegate — Allow recipient to further grant the right
• Audit — Monitor access attempts (successful or failed)

Protection domains can be classified by when and how their object-rights associations change:

Static Domains:

In a static domain system, domains are defined at system configuration time and remain fixed during execution. The set of objects and rights in each domain does not change.

Characteristics:

• Simple to implement and verify
• Easy to analyze for security properties
• Inflexible—cannot adapt to changing requirements
• Often over-privileged (need maximum possible rights at all times)

Example: A dedicated web server process running with a fixed user ID and fixed file permissions. The domain never changes.

Dynamic Domains:

In a dynamic domain system, the contents of a domain can change during execution. Objects can be added or removed, and rights can be acquired or released.

Characteristics:

• Enables principle of least privilege (use only rights needed NOW)
• Complex to implement correctly
• Harder to audit and verify
• Supports privilege escalation and de-escalation

Real-World Example: Android Permissions

Android uses dynamic domains through the permission system:

1. Install time: App requests permissions, domain defined
2. Runtime: App requests additional permissions, domain expands
3. User action: User revokes permission, domain contracts
4. Upgrade: New app version may request new permissions

This dynamic domain model allows Android to implement the principle of least privilege while adapting to user preferences.

How is a process associated with a domain? Different operating systems use different policies:

Policy 1: One Domain per User

Each user has exactly one domain. All processes started by the user execute in that domain.

• Advantages: Simple, intuitive, easy administration
• Disadvantages: Cannot distinguish system utilities from user programs
• Example: Basic Unix without setuid/setgid

Policy 2: One Domain per Process

Each process has its own unique domain. The domain is created when the process starts and destroyed when it terminates.

• Advantages: Maximum isolation, fine-grained control
• Disadvantages: High overhead, complex sharing
• Example: Capability-based systems, some container architectures

Policy 3: Domain per Program

Each program (executable) defines a domain. All executions of that program share the domain.

• Advantages: Program-centric security, consistent behavior
• Disadvantages: Different users running same program get same rights
• Example: macOS app sandboxing entitlements

Policy 4: Hierarchical Domains (Rings)

Domains are arranged in a hierarchy. Inner domains have more privileges than outer domains. A process in an outer ring must request service from an inner ring to perform privileged operations.

• Advantages: Hardware support available, clean privilege separation
• Disadvantages: Fixed number of levels, coarse granularity
• Example: Intel x86 protection rings (Ring 0-3)

Policy 5: Hybrid Policy (Modern Unix/Linux)

Modern Unix combines multiple policies:

• User ID establishes base domain
• setuid/setgid shifts domain temporarily
• Capabilities can be added/removed per-process
• SELinux/AppArmor add program-based policies

This hybrid approach provides the flexibility to implement security policies ranging from simple user isolation to complex mandatory access controls.

The most powerful way to visualize and reason about protection domains is the access matrix model, introduced by Butler Lampson in 1971.

The Access Matrix:

An access matrix is a two-dimensional table where:

• Rows represent protection domains (subjects)
• Columns represent objects
• Cells contain the access rights that domain has over that object

Example Access Matrix:

              │ File1  │ File2  │ Printer │ Process2 │ Domain D2 │
──────────────┼────────┼────────┼─────────┼──────────┼───────────┤
 Domain D1    │ R, W   │ R      │ Print   │ Signal   │ Switch    │
──────────────┼────────┼────────┼─────────┼──────────┼───────────┤
 Domain D2    │ R      │ R, W, X│ -       │ -        │ -         │
──────────────┼────────┼────────┼─────────┼──────────┼───────────┤
 Domain D3    │ Owner  │ -      │ Admin   │ Debug    │ Switch    │
──────────────┴────────┴────────┴─────────┴──────────┴───────────┘

Key Observations:

1. Empty cells mean no access whatsoever. Domain D2 cannot use the printer at all.

2. Domains as objects: Notice "Domain D2" is a column. This allows rights like "Switch" (ability to enter that domain) to be formally modeled.

3. Rights vary by context: Domain D1 can read both files, but only write File1.

4. Special rights: "Owner" implies all rights plus the ability to modify rights. "Admin" implies device management.

Why This Model Matters:

The access matrix provides:

• A universal language for describing any protection scheme
• A basis for proving security properties mathematically
• A framework for comparing ACLs, capabilities, and other implementations
• A mental model for reasoning about information flow

Note that the access matrix is a logical model. No real system stores the entire matrix—it would be enormous and mostly empty. Instead, systems store compressed representations (ACLs store non-empty cells by column; capabilities store non-empty cells by row).

Security researchers have identified several formal properties that protection domain systems should satisfy:

Safety Property:

A system is safe with respect to right R if, starting from an initial access matrix state, there is no sequence of legal operations that would give an unauthorized subject right R to an object.

Harrison, Ruzzo, and Ullman proved in 1976 that the general safety problem is undecidable—there is no algorithm that can determine whether an arbitrary access control system is safe.

Containment Property:

A domain D1 is contained within domain D2 if every right in D1 is also in D2. Formally: D1 ⊆ D2

Containment is useful for hierarchical domains—Ring 0 "contains" Ring 3 because Ring 0 has all Ring 3 rights plus additional privileges.

The Confinement Problem:

One of the most challenging formal problems is confinement: ensuring that a process cannot leak information outside its domain. Even if a process cannot write to external files, it might:

• Modulate CPU usage to signal through covert channels
• Use timing side-channels to transmit information
• Influence scheduling to communicate with co-conspirators

Lampson proved the confinement problem is challenging because covert channels exist in any realistic system. Modern mitigations (constant-time algorithms, resource partitioning) reduce but cannot eliminate this risk.

Information Flow Analysis:

Beyond access control, security often requires controlling information flow. The Bell-LaPadula model adds:

• No read up: A subject cannot read objects at a higher security level
• No write down: A subject cannot write to objects at a lower security level

These rules ensure information flows only in permitted directions, preventing both direct and indirect leakage.

We've established the foundational concept of protection domains. Let's consolidate the critical insights:

What's Next:

Now that we understand what protection domains are, we'll explore domain switching—how processes transition between domains. This is the mechanism that enables setuid programs, system calls, exception handlers, and controlled privilege escalation. Understanding domain switching is essential for comprehending both the security of protection systems and their vulnerabilities.