File Concepts

If files are the nouns of persistent storage, then file operations are the verbs—the actions that breathe life into the static concept of a file. Every interaction between a program and a file—whether reading configuration at startup, writing logs during execution, or saving documents on user request—occurs through a well-defined set of operations.

These operations form a contract between the operating system and applications. Understand this contract, and you understand how all file I/O works, from the simplest script to the most complex database engine. Violate this contract, and you face data corruption, race conditions, and mysterious failures.

This page explores every major file operation in depth—not just what they do, but how they work, why they're designed as they are, and what pitfalls await the unwary programmer.

Every file access follows a fundamental pattern—a lifecycle that applies regardless of programming language, operating system, or file system:

The Basic Pattern:

1. OPEN    → Establish connection to file, get file descriptor
2. READ    → Transfer data from file to memory
   WRITE   → Transfer data from memory to file
   SEEK    → Change position within file
   (repeat as needed)
3. CLOSE   → Release file descriptor, flush buffers

This pattern is so fundamental that it appears in every programming interface:

Opening a file establishes a connection between a process and a file, returning a file descriptor (Unix) or handle (Windows) that represents this connection.

What Happens During Open:

1. Path Resolution: The kernel traverses the directory hierarchy to find the file
2. Permission Check: The kernel verifies the process has needed permissions
3. Inode Load: File metadata is loaded into kernel memory
4. File Table Entry: A new entry is created in the system-wide open file table
5. Descriptor Allocation: A process-local file descriptor is allocated
6. Descriptor Return: The descriptor number is returned to the application

Open Flags (Unix System Call):

int fd = open("/path/to/file", flags, mode);

Reading transfers data from a file into a process's memory buffer. The operation specifies the file descriptor, a buffer to receive data, and the maximum number of bytes to read.

Basic Read System Call (Unix):

ssize_t bytes_read = read(fd, buffer, count);

Return Values:

• Positive: Number of bytes actually read (may be less than requested)
• Zero: End of file reached
• Negative (-1): Error occurred (check errno)

Critical Insight: Short Reads

A common mistake is assuming read() returns exactly the requested number of bytes. In reality, reads can be "short" for many reasons:

• End of file: Only remaining bytes returned
• Pipe/socket: Only currently available data returned
• Interrupting signal: Read interrupted, partial data returned
• Internal buffering: Implementation-dependent batching

Correct Pattern for Reading N Bytes:

// WRONG: Assumes full read
char buffer[1000];
read(fd, buffer, 1000);  // May read less!

// CORRECT: Loop until all bytes read
char buffer[1000];
size_t total = 0;
while (total < 1000) {
    ssize_t n = read(fd, buffer + total, 1000 - total);
    if (n <= 0) break;  // EOF or error
    total += n;
}

Writing transfers data from a process's memory buffer to a file. Like read, write specifies a file descriptor, a buffer containing data, and the number of bytes to write.

Basic Write System Call (Unix):

ssize_t bytes_written = write(fd, buffer, count);

Return Values:

• Positive: Number of bytes actually written (may be less than requested)
• Zero: Possible for unusual situations (empty write, some devices)
• Negative (-1): Error occurred

Short Writes Are Real:

Like reads, writes can be short:

• Disk full: Only partial data written before ENOSPC
• Signal interruption: Write interrupted
• Pipe buffer full: Only buffer space worth of data written
• Quota exceeded: Similar to disk full

Correct Pattern for Writing N Bytes:

ssize_t write_all(int fd, const void *buf, size_t count) {
    const char *ptr = buf;
    size_t remaining = count;
    
    while (remaining > 0) {
        ssize_t n = write(fd, ptr, remaining);
        if (n < 0) {
            if (errno == EINTR) continue;  // Retry on interrupt
            return -1;  // Real error
        }
        ptr += n;
        remaining -= n;
    }
    return count;
}

Seeking changes the file's current position—the offset where the next read or write will occur. This enables random access to file contents.

The lseek System Call:

off_t new_position = lseek(fd, offset, whence);

Parameters:

• fd: File descriptor
• offset: Number of bytes to move (can be negative)
• whence: Reference point for the offset

Whence Values:

Common Seek Patterns:

// Go to beginning of file
lseek(fd, 0, SEEK_SET);

// Go to end of file
lseek(fd, 0, SEEK_END);

// Get current position (seek by 0 from current)
off_t pos = lseek(fd, 0, SEEK_CUR);

// Get file size (seek to end, note position)
off_t size = lseek(fd, 0, SEEK_END);
lseek(fd, 0, SEEK_SET);  // Return to start

// Skip 100 bytes forward
lseek(fd, 100, SEEK_CUR);

// Go back 50 bytes
lseek(fd, -50, SEEK_CUR);

// Read byte 1000 specifically
lseek(fd, 1000, SEEK_SET);
read(fd, &byte, 1);

Seeking Past End of File:

You can seek past the end of a file. If you then write, the gap is filled with zeros—creating a sparse file (or "hole"):

int fd = open("sparse.dat", O_WRONLY | O_CREAT, 0644);
lseek(fd, 1000000, SEEK_SET);  // Seek to 1 MB offset
write(fd, "X", 1);              // Write 1 byte
close(fd);
// File has logical size ~1 MB but uses minimal disk space

Closing a file releases the file descriptor and associated resources. It's the final step in the file operation lifecycle.

What Happens During Close:

1. Buffer Flush: Any buffered data is written (for buffered I/O)
2. Lock Release: Any file locks held are released
3. File Table Cleanup: The system file table entry's reference count decreases
4. Descriptor Free: The file descriptor becomes available for reuse
5. Potential File Deletion: If link count is zero and this was the last reference

The close System Call:

int result = close(fd);
// Returns 0 on success, -1 on error

Close Can Fail—But Usually Doesn't:

Most programmers assume close() always succeeds. While close failures are rare, they can indicate serious problems:

• EIO: Write error occurred (data loss on NFS)
• EINTR: Interrupted by signal (retry may be unsafe)
• ENOSPC: Delayed write failed due to full disk

Robust Close Pattern:

if (close(fd) != 0) {
    perror("close failed");
    // For critical files, this indicates data may be lost!
    // Log the error and possibly alert the user
}

Language-Level Cleanup:

# Python: with statement ensures close
with open('file.txt', 'r') as f:
    data = f.read()
# f is automatically closed here, even if exception occurs

// Java: try-with-resources
try (FileInputStream fis = new FileInputStream("file.txt")) {
    // use fis
} // fis is automatically closed

// Rust: RAII - File closes when dropped
{
    let f = File::open("file.txt")?;
    // use f
} // f is automatically closed when it goes out of scope

Truncation changes a file's size, either shrinking it (discarding data) or extending it (adding zeros).

Truncation System Calls:

// Truncate by path
int result = truncate("/path/to/file", new_size);

// Truncate by file descriptor
int result = ftruncate(fd, new_size);

Effects:

Common Truncation Patterns:

// Clear a file completely (like > file)
ftruncate(fd, 0);

// Preallocate space for a file
ftruncate(fd, 1000000);  // 1 MB

// Chop last 100 bytes from file
off_t size = lseek(fd, 0, SEEK_END);
ftruncate(fd, size - 100);

Using O_TRUNC on Open:

// Atomic open-and-truncate
int fd = open("file.txt", O_WRONLY | O_TRUNC);
// File is now empty, ready for new content

Creating Files:

Files are typically created via open() with the O_CREAT flag, or via creat() (an older interface):

// Modern creation
int fd = open("newfile.txt", O_WRONLY | O_CREAT, 0644);

// With O_EXCL for atomic creation
int fd = open("newfile.txt", O_WRONLY | O_CREAT | O_EXCL, 0644);
if (fd < 0 && errno == EEXIST) {
    // File already exists
}

// Legacy creat() - equivalent to open(path, O_CREAT|O_WRONLY|O_TRUNC, mode)
int fd = creat("newfile.txt", 0644);

The mode Argument (Permissions):

When creating files, the mode specifies initial permissions. The actual permissions are modified by the process's umask:

actual_permissions = mode & ~umask

With a common umask of 022, mode 0666 produces permissions 0644 (rw-r--r--).

Deleting Files (unlink):

int result = unlink("/path/to/file");

What unlink Actually Does:

1. Removes the directory entry (the name)
2. Decrements the file's link count
3. If link count becomes zero AND no processes have the file open:
   • File's data blocks are freed
   • Inode is deallocated

The Delayed Deletion Effect:

If a process has a file open when it's unlinked:

• The name disappears from the directory
• The file continues to exist (accessible via the open fd)
• Disk space is freed only when the last fd is closed

int fd = open("tempfile", O_RDWR | O_CREAT, 0600);
unlink("tempfile");  // Name gone, but file lives
write(fd, "secret", 6);  // Still works!
// File exists only through fd
close(fd);  // NOW the file truly disappears

Renaming changes a file's name without moving its data. Moving within the same filesystem is also just a rename. Moving across filesystems requires copying data.

The rename System Call:

int result = rename("/path/old", "/path/new");

Semantics:

• Atomic: The rename is indivisible—no moment when neither name exists
• Overwrites: If the new name exists, it's silently replaced
• Same inode: The file's inode number doesn't change
• Cross-directory: Works within the same filesystem
• Cross-filesystem: FAILS with EXDEV error

Atomic Updates with Rename:

Rename's atomicity is crucial for safe file updates:

// WRONG: Modify in place (crash leaves corrupt file)
fd = open("config.json", O_RDWR);
write_new_content(fd);
close(fd);

// CORRECT: Write-then-rename (atomic update)
fd = open("config.json.tmp", O_WRONLY | O_CREAT, 0644);
write_new_content(fd);
fsync(fd);              // Ensure data on disk
close(fd);
rename("config.json.tmp", "config.json");  // Atomic swap!
// Power failure at any point leaves valid config

When multiple processes access the same file, file locking prevents conflicts.

Types of Locks:

Locking Mechanisms:

1. flock() - Simple whole-file locks (BSD origin)
2. fcntl() - POSIX locks, supports byte-range locking
3. lockf() - Simplified interface to fcntl()
4. Open File Description Locks - Linux-specific, more predictable

flock() Example:

#include <sys/file.h>

int fd = open("shared.dat", O_RDWR);

// Acquire exclusive lock (blocks if unavailable)
if (flock(fd, LOCK_EX) == 0) {
    // Critical section - only one process here
    write(fd, data, len);
    flock(fd, LOCK_UN);  // Release lock
}

// Non-blocking attempt
if (flock(fd, LOCK_EX | LOCK_NB) == -1) {
    if (errno == EWOULDBLOCK) {
        printf("File is locked by another process\n");
    }
}

fcntl() for Byte-Range Locks:

struct flock fl = {
    .l_type = F_WRLCK,          // Exclusive lock
    .l_whence = SEEK_SET,
    .l_start = 0,               // Start at byte 0
    .l_len = 100,               // Lock 100 bytes
};

fcntl(fd, F_SETLKW, &fl);       // Wait for lock
// ... work with bytes 0-99 ...
fl.l_type = F_UNLCK;
fcntl(fd, F_SETLK, &fl);        // Release

Memory mapping maps a file directly into a process's address space, allowing file access through memory operations rather than read/write calls.

The mmap System Call:

void *addr = mmap(
    NULL,           // Let kernel choose address
    length,         // Size to map
    PROT_READ | PROT_WRITE,  // Permissions
    MAP_SHARED,     // Sharing mode
    fd,             // File descriptor
    offset          // Offset in file
);

if (addr == MAP_FAILED) {
    perror("mmap");
}

// Access file as memory
char *data = (char *)addr;
printf("First byte: %c\n", data[0]);
data[100] = 'X';  // Write to file!

// Cleanup
munmap(addr, length);

We've explored the complete landscape of file operations—from basic open/read/write/close to advanced topics like locking and memory mapping. Let's consolidate:

What's next:

Now that we understand file operations, we'll explore file types—the different kinds of files that exist beyond regular files. Directories, symbolic links, device files, pipes, and sockets all use the file abstraction but serve very different purposes.