Conceptual Questions

Operating systems are the guardians of computer security. They mediate every access to hardware, manage every process's privileges, and enforce every permission check. When security fails at the OS level, nothing above it—applications, data, users—is safe.

Interviewers probe security concepts to assess whether candidates understand the fundamental mechanisms that protect systems: How does authentication work? What's the difference between discretionary and mandatory access control? How do buffer overflows lead to privilege escalation? These questions reveal depth of systems understanding.

This page consolidates security fundamentals at the level expected in operating systems interviews—from the classic protection concepts to modern defense mechanisms.

All security mechanisms serve three fundamental goals, known as the CIA triad:

Confidentiality

Definition: Ensuring that information is accessible only to those authorized to access it.

OS mechanisms:

• File permissions (read access control)
• Process isolation (one process cannot read another's memory)
• Encryption (disk encryption, encrypted swap)
• Access control lists (who can see what)

Threats:

• Unauthorized access to files
• Memory disclosure vulnerabilities (Spectre, Meltdown)
• Side-channel attacks (timing, power analysis)

Integrity

Definition: Ensuring that information and system state can only be modified by authorized parties in authorized ways.

OS mechanisms:

• Write permissions (control who can modify)
• Checksums and digital signatures (detect tampering)
• Secure boot (verify boot chain integrity)
• Mandatory access control (enforce modification policies)

Threats:

• Unauthorized modification of files or configuration
• Privilege escalation (modify what you shouldn't)
• Malware that modifies system binaries

Availability

Definition: Ensuring that authorized users have reliable access to information and resources.

OS mechanisms:

• Resource quotas (prevent one user from exhausting resources)
• Fair scheduling (no process monopolizes CPU)
• Process isolation (one crash doesn't bring down system)
• Redundancy and fault tolerance

Threats:

• Denial of service attacks (flood resources)
• Fork bombs (exhaust process table)
• Resource exhaustion (fill disk, exhaust memory)

Authentication is the process of verifying that an entity (user, process, system) is who or what it claims to be. It answers: "Are you really who you say you are?"

Authentication Factors

Something you know:

• Passwords, PINs, security questions
• Cheapest, most common, weakest (can be guessed, stolen, phished)

Something you have:

• Hardware tokens, smart cards, phones
• Harder to steal remotely; can be lost physically

Something you are:

• Biometrics: fingerprint, face, iris, voice
• Cannot be lost or forgotten; cannot be changed if compromised

Multi-factor authentication (MFA):

• Requires two or more factors from different categories
• Much stronger than single factor
• Example: Password (know) + authenticator app (have)

Password Storage: How NOT to Store Passwords

Storing passwords securely is critical. The progression of techniques:

Level 0: Plain text ❌

users table: username, password
"alice", "secret123"

Disaster: One breach exposes all passwords.

Level 1: Encrypted passwords ❌

Store encrypt(password, key)

Better, but: Key must be stored somewhere; compromise key = compromise all.

Level 2: Hashed passwords ⚠️

Store hash(password)  // e.g., SHA-256

One-way function; cannot reverse. But:

• Rainbow tables: precomputed hash → password mappings
• Dictionary attacks: hash common passwords, compare

Level 3: Salted hashes ✓

Store (salt, hash(salt + password))
Salt: random value unique per password

Rainbow tables are useless (would need table per salt). Dictionary attacks much slower (must hash each guess with each salt).

Level 4: Slow hashes (modern standard) ✓✓

Store bcrypt(password, cost_factor) or
      argon2(password, memory_cost, time_cost)

Deliberately slow to compute. GPU/ASIC attacks become impractical. Bcrypt, scrypt, Argon2 are designed for password storage.

Authorization determines what an authenticated entity is allowed to do. It answers: "Now that I know who you are, what can you access?"

The Access Control Matrix

Conceptually, access control is a matrix:

              File A    File B    Printer   Network
User alice    read      read/wrt  print     -
User bob      -         read      -         connect
Process X     read      -         -         -

Rows: subjects (users, processes) Columns: objects (files, devices, resources) Cells: permissions (read, write, execute, etc.)

This matrix is typically stored in one of two ways:

Access Control Lists (ACLs) - stored with objects:

File A: [(alice, read), (process X, read)]
File B: [(alice, read/write), (bob, read)]

Easy to see: "Who can access this file?"

Capability Lists - stored with subjects:

alice: [(File A, read), (File B, read/write), (Printer, print)]
bob: [(File B, read), (Network, connect)]

Easy to see: "What can this user access?"

Most systems use ACLs (stored with files). Unix permissions are a simplified ACL.

Discretionary Access Control (DAC)

Definition: Resource owners decide who can access their resources.

Characteristics:

• Owner of a file sets permissions
• Permissions can be delegated (owner gives others access rights)
• Flexible but vulnerable to user mistakes and trojan horses

Unix/Linux permissions:

-rwxr-x--- 1 alice developers 4096 Jan 1 file.txt
│└┬┘└┬┘└┬┘
│ │  │  └── others: no access
│ │  └───── group (developers): read, execute
│ └──────── owner (alice): read, write, execute
└────────── file type (- = regular file)

Octal: 750 (rwx=7, r-x=5, ---=0)

Limitation: If alice runs malicious code, that code inherits alice's permissions. It can read/write anything alice can. No protection against trojan horses.

Mandatory Access Control (MAC)

Definition: System-wide policy enforced by the OS, regardless of owner preferences.

Characteristics:

• Policy set by security administrator, not resource owners
• Labels/categories assigned to subjects and objects
• Access decisions based on comparing labels
• Users cannot override policy (even resource owners)

Bell-LaPadula Model (Confidentiality):

• Objects and subjects have security levels (e.g., Unclassified < Secret < Top Secret)
• "No read up": Subject cannot read objects at higher level
• "No write down": Subject cannot write to objects at lower level
• Prevents information from flowing from high to low

Biba Model (Integrity):

• Opposite of Bell-LaPadula for integrity
• "No read down": Don't read less trustworthy data
• "No write up": Don't corrupt more trustworthy data

Real implementations:

• SELinux (Security-Enhanced Linux): Labels on all resources, policy defines allowed operations
• AppArmor: Path-based MAC for Linux, simpler than SELinux
• Windows Mandatory Integrity Control: Low, Medium, High, System integrity levels

Role-Based Access Control (RBAC)

Definition: Access rights assigned to roles; users assigned to roles.

Characteristics:

• Separation between users and permissions
• Roles represent job functions (admin, developer, auditor)
• Easy to manage: change role assignment, not individual permissions
• Scalable: adding user = assigning roles

Example:

Roles:
  admin: [all_files: read/write/delete, users: manage, logs: read]
  developer: [code_repo: read/write, docs: read, logs: read]
  auditor: [all_files: read, logs: read]

Users:
  alice: [admin, developer]  → gets union of permissions
  bob: [developer]
  carol: [auditor]

Variations:

• Hierarchical RBAC: Roles inherit from parent roles
• Constrained RBAC: Mutual exclusion (can't be both auditor and admin)
• Temporal RBAC: Role assignment limited by time

Several foundational principles guide secure system design:

Principle of Least Privilege

Statement: Every process, user, and system should have only the minimum privileges necessary to perform its function.

Rationale: If an entity is compromised, damage is limited to what that entity could legitimately do.

Examples:

• Web server runs as www-data (not root)
• Database grants read-only access where write isn't needed
• Android apps request only permissions they need
• Linux capabilities: give process just CAP_NET_BIND_SERVICE, not full root

Challenge: Convenience vs. security. Users want things easy; least privilege adds friction.

Defense in Depth

Statement: Multiple layers of security, so failure of one doesn't compromise the system.

Layers:

1. Physical security (locked server room)
2. Network security (firewalls, IDS)
3. Host security (OS hardening, antivirus)
4. Application security (input validation, secure coding)
5. Data security (encryption at rest and in transit)

Rationale: Attackers must bypass multiple defenses, each reducing success probability.

Fail-Safe Defaults

Statement: Default to denying access; explicitly grant permissions.

Examples:

• Firewall default: block all, allow listed
• New user: no access until assigned
• Capability list: start empty

Opposite (fail-open) is dangerous: "Grant all, block listed" means new resources are vulnerable by default.

Understanding attacks helps understand defenses. Here are the most important OS-level attack vectors:

Buffer Overflow

The attack:

• Write more data than buffer can hold
• Overflow overwrites adjacent memory (return addresses, function pointers)
• Attacker controls program execution

void vulnerable(char *input) {
    char buffer[64];
    strcpy(buffer, input);  // No length check!
    // If input > 64 bytes, it overwrites stack
    // including return address
}

OS defenses:

• Stack canaries: Random value before return address; checked on return
• ASLR: Randomize memory layout; attacker can't predict addresses
• DEP/NX (Data Execution Prevention): Non-executable stack; can't run injected code
• Stack protection: Compiler-inserted boundary checks

Privilege Escalation

The attack:

• Start with low-privilege access
• Exploit vulnerability to gain higher privileges
• Goal: usually root/administrator access

Vectors:

• SUID binaries with bugs (exploit program running as root)
• Kernel vulnerabilities (escape user mode protections)
• Misconfigured permissions (writable system files)
• Race conditions in privilege checks (TOCTOU)

OS defenses:

• Minimize SUID binaries; audit regularly
• Kernel hardening (restrict loadable modules, lock down sysctl)
• Seccomp (limit system calls process can make)
• Namespaces (isolate what process can see)
• Capabilities (fine-grained root powers)

Side-Channel Attacks

The attack:

• Observe side effects of computation (timing, cache behavior, power consumption)
• Infer secrets from observations

Examples:

• Spectre: Exploit speculative execution to leak memory across boundaries
• Meltdown: Read kernel memory from user space via speculative execution
• Timing attacks: Measure time to check password; longer = more correct characters

OS defenses:

• KPTI (Kernel Page Table Isolation): Separate kernel and user page tables
• Retpoline: Prevent branch target injection
• Constant-time cryptography: Operations take same time regardless of values
• Cache partitioning: Prevent cross-process cache interference

Code Injection and ROP

Code injection:

• Inject executable code into process memory
• Redirect execution to injected code
• Mitigated by DEP/NX (non-executable data segments)

Return-Oriented Programming (ROP):

• Bypass DEP by chaining existing code fragments ("gadgets")
• Each gadget ends with RET instruction
• Chain gadgets on stack to perform arbitrary operations
• Much harder than injecting code; still viable attack

Defenses:

• ASLR makes gadget addresses unpredictable
• CFI (Control Flow Integrity): Validate call/return targets
• Shadow stacks: Separate protected copy of return addresses
• Intel CET: Hardware-assisted control flow enforcement

Modern security relies heavily on isolation—containing processes so that compromise of one doesn't affect others.

Process Isolation (Virtual Memory)

We covered this earlier: each process has its own address space. This is the first layer of isolation.

Guarantees:

• Process A cannot access Process B's memory
• Hardware-enforced via MMU

Limitations:

• Shared resources can leak (side channels)
• Kernel is shared across all processes
• System call interface is an attack surface

System Call Filtering (seccomp)

Concept: Limit which system calls a process can make.

Rationale: Kernel vulnerabilities often exploited via system calls. Fewer available syscalls = smaller attack surface.

// seccomp-bpf example: allow only read, write, exit
scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_KILL);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit), 0);
seccomp_load(ctx);
// Process can now only read, write, exit
// Any other syscall → killed

Usage:

• Chrome sandboxes renderers with strict seccomp policies
• Docker applies seccomp profiles to containers
• systemd services can specify allowed syscalls

Namespaces (Linux)

Concept: Partition kernel resources so processes see isolated views.

Namespace types:

• PID: Process IDs. Container sees itself as PID 1.
• Network: Separate network stack (interfaces, routing, firewall).
• Mount: Separate filesystem mount tree.
• User: Map UIDs/GIDs (container root ≠ host root).
• UTS: Separate hostname and domain name.
• IPC: Separate System V IPC and POSIX message queues.
• Cgroup: Separate cgroup root.

Effect: Process in namespace cannot see or access resources outside it.

Control Groups (cgroups)

Concept: Limit and account for resource usage.

Capabilities:

• CPU limits (shares, throttling)
• Memory limits (max usage, OOM behavior)
• I/O bandwidth limits
• PID limits (prevent fork bombs)

Usage:

• Containers get cgroup limits
• systemd manages services in cgroups
• Fair resource sharing in multi-tenant environments

Containers: Namespace + cgroups + (optionally seccomp)

Containers (Docker, Kubernetes pods) combine:

• Namespaces for isolation
• cgroups for resource limits
• seccomp for syscall filtering
• Overlay filesystems for efficient image layers

This provides strong isolation without the overhead of full virtualization.

Security must start at the lowest level. If the boot process is compromised, nothing above it can be trusted.

Secure Boot

Concept: Verify that boot code is signed and unmodified.

UEFI Secure Boot process:

1. Firmware contains trusted public keys (Platform Key, db)
2. Firmware loads bootloader only if signed by trusted key
3. Bootloader loads kernel only if signed
4. Kernel loads drivers only if signed
5. Any unverified component → boot aborted

Protection against:

• Boot sector viruses
• Rootkits that modify bootloader/kernel
• Evil maid attacks (physical access modifications)

Trusted Platform Module (TPM)

Concept: Hardware chip that provides cryptographic functions and secure storage.

Capabilities:

• Generate and store cryptographic keys (never leave TPM)
• Platform measurements (hash of boot components)
• Sealed storage (decrypt only if platform in known state)
• Random number generation

Measured Boot:

1. Each boot stage measures (hashes) the next stage
2. Measurements stored in TPM Platform Configuration Registers (PCRs)
3. Final state can be attested to remote parties
4. If any component changed, PCRs differ → detected

Use cases:

• BitLocker (Windows disk encryption) stores key in TPM
• Remote attestation: prove to server that client is unmodified
• Hardware-bound credentials

Security questions in OS interviews typically fall into a few categories. Here's how to structure answers:

"What are the security goals of an operating system?"

"Operating systems protect confidentiality, integrity, and availability—the CIA triad. Confidentiality means only authorized users can read data (enforced via permissions and isolation). Integrity means only authorized modifications occur (enforced via access control and signatures). Availability means authorized users can access resources (enforced via quotas and fair scheduling)."

"Explain the difference between authentication and authorization."

"Authentication verifies identity—it answers 'who are you?' Methods include passwords, tokens, biometrics, and MFA. Authorization determines permissions—it answers 'what can you do?' Once authenticated, the system checks ACLs, capabilities, or roles to determine access rights. Authentication precedes authorization."

"What is the principle of least privilege?"

"Every entity should have only the minimum permissions needed for its function. A web server doesn't need root—it runs as www-data. If compromised, damage is limited. Modern implementations include Linux capabilities (grant specific privileges without root), seccomp (limit available syscalls), and containerization (namespace isolation)."

Operating system security is the foundation upon which all application and data security rests. Let's consolidate the essential knowledge:

Module Complete:

This concludes Module 5: Conceptual Questions. You've now mastered the five essential conceptual topics for operating systems interviews:

1. Process vs. Thread Differences
2. Virtual Memory Benefits
3. Deadlock Conditions
4. File System Comparisons
5. Security Concepts

These topics represent the conceptual foundations that interviewers expect strong candidates to articulate clearly and deeply. Combined with numerical problem-solving skills and hands-on project experience, you are well-prepared for operating systems interviews.