Asynchronous Operations

Imagine a restaurant where the chef prepares each dish one at a time—taking an order, cooking it completely, plating it, and serving it—before even looking at the next order. During a busy evening, customers would wait hours as the chef blocks on each individual dish.

Now imagine a different model: the chef takes multiple orders, starts several dishes simultaneously, checks on cooking timers, delegates prep tasks to sous chefs, and orchestrates completion across many concurrent activities. The throughput increases dramatically without adding more chefs.

This is the fundamental distinction between synchronous and asynchronous operation. In computing, the synchronous model where operations block until completion creates the same bottleneck as our first chef. Asynchronous operations allow systems to initiate multiple activities, continue working on other tasks, and respond when those activities complete—dramatically improving throughput and responsiveness.

At the most fundamental level, synchronous and asynchronous describe how operations relate to the flow of program execution.

Synchronous execution means the caller initiates an operation and waits—the calling thread is suspended until the operation completes. The word 'synchronous' derives from Greek roots meaning 'same time'—the caller and the operation proceed together, in lockstep. The caller cannot proceed until the operation returns.

Asynchronous execution means the caller initiates an operation and continues immediately—the operation proceeds independently while the caller does other work. The word 'asynchronous' means 'not at the same time'—the caller and the operation proceed on separate timelines. The caller is notified of completion through some mechanism (callback, polling, or signal).

A concrete example—reading a file:

Synchronous approach:

data = read_file("/path/to/file")  // Thread blocks here
process(data)                       // Executes after read completes

Asynchronous approach:

read_file_async("/path/to/file", callback=process)  // Returns immediately
do_other_work()                                       // Executes while read proceeds
// Later: process() is called when read completes

In the synchronous model, do_other_work() cannot execute until read_file completes. In the asynchronous model, do_other_work() executes immediately, and process() is invoked later when the file read finishes.

To truly understand asynchronous programming, we must first understand what blocking means at the operating system level.

When a process or thread makes a blocking system call, the kernel moves that thread from the ready queue to a wait queue associated with the resource it's waiting for. The thread is no longer scheduled for CPU time—it's entirely suspended until the awaited event occurs.

Common blocking operations include:

• Disk I/O: Reading from or writing to files (waiting for disk head movement, sector reads)
• Network I/O: Waiting for data from a socket (packets traveling over the network)
• User Input: Waiting for keyboard, mouse, or other input devices
• Synchronization: Waiting on a lock, semaphore, or condition variable
• Sleep: Explicitly waiting for a timer to expire

The kernel wait queue mechanism:

The kernel maintains wait queues for every resource that can cause blocking. When a thread blocks:

1. State transition: Thread state changes from RUNNING to INTERRUPTIBLE (or UNINTERRUPTIBLE for non-abortable waits)
2. Queue insertion: Thread is added to the wait queue for the specific resource
3. Context switch: Scheduler immediately selects another ready thread
4. Zero CPU consumption: Blocked thread uses no CPU cycles while waiting

When the awaited event occurs (e.g., I/O completion):

1. Interrupt handler: Hardware interrupt notifies the kernel
2. Wake up: Kernel walks the wait queue and wakes all waiting threads
3. State transition: Threads move from INTERRUPTIBLE to RUNNING
4. Scheduling: Awakened threads compete for CPU time normally

Non-blocking I/O takes a different approach: instead of suspending the caller when an operation cannot complete immediately, the system call returns immediately with an indication that the operation is not yet complete.

The caller can then:

• Retry the operation later (polling)
• Proceed with other work and check periodically
• Use an event notification mechanism to learn when to retry

The key insight: Non-blocking I/O separates the initiation of an operation from the completion. The thread remains schedulable and can make progress on other tasks.

EAGAIN and EWOULDBLOCK:

These error codes are central to non-blocking I/O. They indicate that the operation cannot complete right now but is not an error—the caller should try again later when conditions change.

• EAGAIN: "Try again" - temporary resource unavailability
• EWOULDBLOCK: "This would block" - same meaning, some systems use this

On most modern systems, these are the same value. The semantics: "What you asked for isn't ready, but it's not your fault and nothing's broken."

Non-blocking connects vs reads:

Different operations have different non-blocking behaviors:

There's an important distinction between non-blocking I/O and true asynchronous I/O (AIO). They're often conflated, but the mechanisms differ significantly.

Non-blocking I/O tells you that an operation would block, returning immediately. You must still perform the actual I/O operation yourself, typically when an event notifies you that the descriptor is ready.

Asynchronous I/O initiates the entire operation in the kernel, which completes it in the background. The kernel notifies you when the data is already transferred—no additional read/write call is needed.

Think of it this way:

• Blocking: You ask someone to pour you coffee and wait for them.
• Non-blocking: You check if coffee is ready; if not, you walk away and check later.
• Asynchronous: You request coffee, go do other things, and someone brings it to you when ready.

Linux AIO implementations:

Linux has evolved through several async I/O implementations:

1. POSIX AIO (aio_read/aio_write)

• Portable but often implemented with thread pools
• Not true kernel-level async on many systems
• Signal or thread-based notification

2. Native Linux AIO (io_submit/io_getevents)

• True kernel-level async I/O
• Works well for direct I/O (O_DIRECT)
• Limited for buffered I/O

3. io_uring (Linux 5.1+)

• Modern, highly efficient async I/O
• Submission and completion queues in shared memory
• Near-zero overhead for high-frequency operations
• Supports far more operation types than previous APIs

The importance of asynchronous programming crystallized around the C10K problem—the challenge of handling 10,000 concurrent connections on a single server. This problem, articulated by Dan Kegel in 1999, exposed the fundamental limitations of thread-per-connection architectures.

The thread-per-connection model:

In traditional server designs, each client connection gets its own thread:

while (true) {
    client = accept(server_socket);  // Block waiting for connection
    spawn_thread(handle_client, client);
}

void handle_client(socket) {
    while (true) {
        request = read(socket);  // BLOCKS waiting for client
        response = process(request);
        write(socket, response); // BLOCKS until buffer available
    }
}

Why this breaks at scale:

The math of thread limits:

Even with reduced stack sizes (256KB), 10,000 threads consume 2.5 GB of stack memory. More critically, most of these threads are blocked most of the time, doing nothing but waiting for slow network I/O.

The async solution:

Asynchronous I/O inverts the model. Instead of one thread per connection, a small pool of threads (often equal to CPU cores) handles all connections:

while (true) {
    events = wait_for_events();  // Single syscall monitors ALL connections
    for event in events {
        if (event.type == READ_READY) {
            data = non_blocking_read(event.socket);
            process_data(event.socket, data);
        }
        if (event.type == WRITE_READY) {
            send_pending_data(event.socket);
        }
    }
}

This event-driven approach means:

• 4 threads can handle 100,000 connections
• Memory usage stays bounded (~300 bytes per connection state, not 8 MB)
• No context-switch overhead between connection handlers
• CPU time spent on actual work, not scheduling

Different programming languages provide varying levels of async support, from manual callback management to sophisticated runtime systems. Understanding these differences helps you choose the right tool and understand what's happening beneath abstractions.

A common source of confusion is conflating asynchronous with parallel. These are related but distinct concepts.

Asynchronous: Operations that don't block the calling thread. The operations may or may not run simultaneously—the key property is that the caller continues without waiting.

Parallel: Operations that run simultaneously on multiple CPU cores. This requires multiple threads or processes executing at the same instant.

The key insight: Async is about waiting efficiently, while parallel is about computing simultaneously.

Single-threaded async example:

Time: |--0ms--|--1ms--|--2ms--|--3ms--|--4ms--|--5ms--|--6ms--|--7ms--|

Thread 1:
  [Start A] [Start B] [Start C]   [A done]  [B done]  [C done]
       |         |         |          |          |         |
       +---------|---------|----------+          |         |
                 +---------|---------------------+         |
                           +-------------------------------+

                    ↓ All I/O operations overlap in wall-clock time
                    ↓ But only ONE thread is executing at any moment

Multi-threaded parallel example:

Time: |--0ms--|--1ms--|--2ms--|--3ms--|

Thread 1: [=====COMPUTE A=====]
Thread 2: [=====COMPUTE B=====]
Thread 3: [=====COMPUTE C=====]

            ↓ All three threads execute SIMULTANEOUSLY
            ↓ Uses 3 CPU cores at once

Combined async + parallel:

Modern systems often combine both:

• Use async for I/O operations (network, disk)
• Use parallelism for CPU-intensive work
• Example: Web server with async I/O that dispatches CPU-bound tasks to a thread pool