Transport Layer Multiplexing and Demultiplexing

We've traced the journey of network data through multiplexing at the sender, demultiplexing at the receiver, port identification, and connection identification. But there's one final step in this journey: process mapping—the mechanism by which the operating system delivers demultiplexed data to the correct application process.

When a TCP segment arrives and is matched to a socket, what happens next? The data doesn't magically appear in the application's memory. The operating system must:

1. Identify which process owns the socket
2. Place data in the appropriate process buffer
3. Notify the process that data is available
4. Manage the handoff when the process reads the data

This page explores the complete process mapping system—how sockets, file descriptors, processes, and the I/O subsystem work together to complete data delivery.

In Unix-like operating systems (Linux, macOS, BSD), sockets are represented as file descriptors. This elegant design means that network I/O uses the same interface as file I/O—read(), write(), close()—providing a unified programming model.

What is a File Descriptor?

A file descriptor (fd) is a small non-negative integer that serves as a handle to an open I/O resource. When a process opens a file, creates a socket, or opens a pipe, the kernel assigns a file descriptor and returns it to the process.

Process File Descriptor Table:
┌────┬──────────────────────────────┐
│ FD │ Resource                     │
├────┼──────────────────────────────┤
│  0 │ stdin (standard input)       │
│  1 │ stdout (standard output)     │
│  2 │ stderr (standard error)      │
│  3 │ /var/log/app.log (file)      │
│  4 │ TCP socket (listen :8080)    │
│  5 │ TCP socket (connected)       │
│  6 │ UDP socket                   │
└────┴──────────────────────────────┘

The first three descriptors (0, 1, 2) are reserved by convention. New resources get the lowest available number.

Socket Creation and File Descriptor Assignment:

// Create a TCP socket
int sockfd = socket(AF_INET, SOCK_STREAM, 0);
// sockfd now holds a file descriptor, e.g., 3

// Bind to local address
bind(sockfd, (struct sockaddr*)&addr, sizeof(addr));

// Listen for connections
listen(sockfd, backlog);

// Accept a connection - creates NEW file descriptor
int clientfd = accept(sockfd, NULL, NULL);
// clientfd is a different fd, e.g., 4

// Now we have:
// sockfd (3) - listening socket
// clientfd (4) - connected socket for specific client

Windows Socket Handles:

Windows uses a different model—sockets are handles (SOCKET type), not file descriptors. They use different API calls (recv/send vs read/write) but the conceptual mapping is similar.

// Windows
SOCKET sock = socket(AF_INET, SOCK_STREAM, 0);
recv(sock, buffer, length, 0);  // Not read()
send(sock, buffer, length, 0);  // Not write()
closesocket(sock);               // Not close()

When demultiplexing delivers data to a socket, it actually delivers to a kernel data structure. Understanding this structure reveals how process mapping works.

The Socket Kernel Object:

Each socket file descriptor references a kernel socket structure that contains:

1. Protocol information: TCP, UDP, etc.
2. Local and remote addresses: The identification tuple
3. State: Connection state (for TCP)
4. Buffers: Send and receive queues
5. Wait queues: Processes waiting for events
6. Options: Socket options (SO_REUSEADDR, etc.)
7. Statistics: Bytes sent, received, errors

The Path from File Descriptor to Socket:

Process calls recv(fd=5, buffer, len)
    │
    ▼
Kernel looks up fd 5 in process's file descriptor table
    │
    ▼
File descriptor table entry points to struct file
    │
    ▼
struct file contains pointer to struct socket
    │
    ▼
struct socket contains receive_queue with buffered data
    │
    ▼
Data is copied from receive_queue to user buffer

Key Insight:

The file descriptor is just an index. The actual socket state (buffers, addresses, etc.) lives in kernel memory, protected from direct user access. System calls like recv() provide the controlled interface to access this kernel state.

Sockets are owned by processes. This ownership determines which process receives data, which process can send, and what happens when the process terminates.

Ownership Establishment:

A process owns a socket if:

1. The process created the socket (socket() call)
2. The process inherited the socket from a parent (fork())
3. The socket was passed via Unix domain socket (fd passing)

Ownership and Fork:

int sockfd = socket(AF_INET, SOCK_STREAM, 0);
bind(sockfd, ...);
listen(sockfd, ...);

pid_t child = fork();

if (child == 0) {
    // Child process
    // Inherits sockfd - can accept() connections
    int clientfd = accept(sockfd, NULL, NULL);
} else {
    // Parent process  
    // Also has sockfd - both can accept()
}

After fork(), both parent and child share the same socket. This is how multi-process servers (like Apache's prefork MPM) work.

Process Termination and Sockets:

When a process terminates, all its open file descriptors are closed:

Process exits with open TCP connections:
    │
    ▼
Kernel closes all file descriptors
    │
    ▼
For each TCP socket:
    └── If empty send buffer: Send FIN, enter FIN_WAIT_1
    └── If data in send buffer: Send data, then FIN
    └── Socket enters TIME_WAIT after full close
    │
    ▼
Eventually, socket resources are freed

Orphaned Connections:

If a process crashes abruptly (kill -9), the kernel still handles graceful close. However, the application may not have flushed all data. This is why applications should handle shutdown gracefully when possible.

Viewing Process-Socket Relationships:

# Linux: Show sockets with process info
ss -tnp
# State  Recv-Q  Send-Q  Local     Foreign     Process
# ESTAB  0       0       *:443     10.0.0.1    nginx,pid=1234,fd=10

# lsof: List open files (including sockets)
lsof -i :443
# COMMAND   PID   USER   FD   TYPE    SIZE/OFF NODE NAME
# nginx     1234  www    10u  IPv4    12345    TCP  *:443 (LISTEN)

When demultiplexing routes a segment to a socket, how does the data actually reach the application process? This involves several steps within the kernel.

Step 1: Kernel Receives Segment

Network card receives frame
    │
    ▼
Interrupt handler copies frame to kernel memory
    │
    ▼
IP layer processes, validates, passes to transport
    │
    ▼
Transport layer demultiplexes to specific socket

Step 2: Data Enqueued on Socket

TCP receives validated segment
    │
    ▼
Sequence number checking, reassembly if needed
    │
    ▼
Data added to socket's receive_queue (sk_buff chain)
    │
    ▼
Socket buffer counter updated

Step 3: Process Notification

Socket checks if process is waiting for data
    │
    ├── If waiting: Wake process from sleep
    │
    ├── If using epoll/select: Mark fd as readable
    │
    └── If using signals (SIGIO): Send signal to process

Step 4: Process Reads Data

Process calls recv(sockfd, buffer, len, flags)
    │
    ▼
Kernel copies data from socket receive_queue to user buffer
    │
    ▼
receive_queue freed, socket buffer counters updated
    │
    ▼
Kernel sends window update to peer (more space in buffer)

How a process waits for data significantly affects process mapping behavior. There are two fundamental modes: blocking and non-blocking.

Blocking I/O (Default):

When a process calls recv() on a blocking socket:

// Blocking mode (default)
int bytes = recv(sockfd, buffer, 1024, 0);
// Process sleeps here until:
// - Data arrives (bytes > 0)
// - Connection closed (bytes = 0)
// - Error occurs (bytes = -1)

Internally:

1. Process is placed on socket's wait queue
2. Process state set to SLEEPING
3. Scheduler runs other processes
4. When data arrives, process is woken
5. recv() returns with data

Non-Blocking I/O:

When a socket is set to non-blocking:

// Set non-blocking mode
int flags = fcntl(sockfd, F_GETFL, 0);
fcntl(sockfd, F_SETFL, flags | O_NONBLOCK);

int bytes = recv(sockfd, buffer, 1024, 0);
// Returns immediately!
// If data available: bytes > 0
// If no data: bytes = -1, errno = EWOULDBLOCK/EAGAIN
// If closed: bytes = 0

The process is never put to sleep; it must poll or use I/O multiplexing.

Wait Queue Mechanics:

Blocking I/O uses kernel wait queues:

Process A calls recv() on empty socket:
    │
    ▼
Process A added to socket->wait_queue
    │
    ▼
Process A state = TASK_INTERRUPTIBLE (sleeping)
    │
    ▼
Scheduler switches to another process

... later, data arrives ...

Network stack adds data to socket receive_queue
    │
    ▼
Kernel walks socket->wait_queue, wakes all waiters
    │
    ▼
Process A state = TASK_RUNNING (runnable)
    │
    ▼
Scheduler eventually runs Process A
    │
    ▼
recv() copies data and returns

A server handling thousands of connections needs an efficient way to know which sockets have data ready. I/O multiplexing APIs solve this problem.

The Problem:

Server has 10,000 connected sockets
How to efficiently wait for data on any of them?

Option A: Thread-per-socket (10,000 threads - expensive!)
Option B: Busy-poll all sockets (100% CPU wasteful)
Option C: I/O multiplexing (efficient solution ✓)

I/O Multiplexing APIs:

1. select() - The Classic:

fd_set readfds;
FD_ZERO(&readfds);
FD_SET(sock1, &readfds);
FD_SET(sock2, &readfds);

int ready = select(maxfd + 1, &readfds, NULL, NULL, &timeout);

if (FD_ISSET(sock1, &readfds)) {
    // sock1 has data
}

Limitations: O(n) per call, limited to 1024 fds typically.

2. poll() - Slightly Better:

struct pollfd fds[2];
fds[0] = {.fd = sock1, .events = POLLIN};
fds[1] = {.fd = sock2, .events = POLLIN};

int ready = poll(fds, 2, timeout_ms);

if (fds[0].revents & POLLIN) {
    // sock1 has data
}

No fd limit, still O(n) per call.

3. epoll() - Linux High Performance:

int epfd = epoll_create1(0);

struct epoll_event ev = {.events = EPOLLIN, .data.fd = sock1};
epoll_ctl(epfd, EPOLL_CTL_ADD, sock1, &ev);

struct epoll_event events[100];
int n = epoll_wait(epfd, events, 100, timeout_ms);

for (int i = 0; i < n; i++) {
    // events[i].data.fd has data
}

O(1) for adding/removing, O(ready) for wait - scales to millions of fds.

How epoll Works Internally:

epoll_ctl(ADD) registers socket:
    │
    ▼
Socket's wait_queue gets special epoll callback
    │
    ▼
When data arrives on socket:
    └── Callback adds socket to epoll's ready list
    │
    ▼
epoll_wait() returns only ready sockets
    └── No need to scan all registered sockets

This is why epoll scales: instead of asking "is socket ready?" for each socket, it gets told when any socket becomes ready.

Event-Driven Architecture:

Modern servers combine non-blocking I/O with epoll:

while (running) {
    int n = epoll_wait(epfd, events, MAX_EVENTS, -1);
    
    for (int i = 0; i < n; i++) {
        if (events[i].data.fd == listen_sock) {
            accept_new_connection();
        } else {
            handle_client_data(events[i].data.fd);
        }
    }
}

A single thread can handle tens of thousands of connections efficiently.

How sockets are distributed across threads significantly impacts process mapping and performance. Several models exist.

Model 1: Thread-Per-Connection

Main Thread:
    while (true):
        clientfd = accept(listenfd)
        spawn_thread(handle_client, clientfd)

Worker Thread:
    while (connection_open):
        data = recv(clientfd)  # Blocking OK
        response = process(data)
        send(clientfd, response)
    close(clientfd)

• Simple programming model
• Context switching overhead with many connections
• Memory overhead (stack per thread)
• Works well for <100-1000 connections

Model 2: Event Loop (Single-Threaded)

Single Thread:
    while (true):
        ready_fds = epoll_wait(epfd)
        for fd in ready_fds:
            if fd is listen_socket:
                new_client = accept(fd)
                epoll_add(new_client)
            else:
                data = recv(fd)  # Non-blocking
                response = process(data)
                send(fd, response)

• No thread synchronization needed
• Can't utilize multiple CPU cores
• Must avoid blocking operations
• Good for I/O-bound workloads

Model 3: Multi-Threaded with I/O Multiplexing

Acceptor Thread:
    while (true):
        clientfd = accept(listenfd)
        worker = select_worker()  # Round-robin or least-loaded
        assign_to_worker(worker, clientfd)

Worker Thread (one per CPU core):
    while (true):
        ready_fds = epoll_wait(my_epfd)
        for fd in ready_fds:
            handle_event(fd)

This hybrid model (used by nginx, Node.js cluster):

• Utilizes all CPU cores
• Each worker handles many connections with epoll
• Low context switching
• Scales to millions of connections

SO_REUSEPORT: Kernel-Level Load Balancing

// Multiple processes can bind to same port
setsockopt(sockfd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof(one));
bind(sockfd, ...);
listen(sockfd, ...);

// Kernel distributes incoming connections across all listeners

With SO_REUSEPORT, the kernel handles distribution—each worker has its own listening socket, and the kernel load-balances incoming connections.

Let's trace a complete example from packet arrival to application processing, showing every step of process mapping.

Scenario:

• Server: nginx listening on port 443
• Client: Browser requesting a webpage
• Data: HTTP request "GET / HTTP/1.1\r\n..."

Timeline:

Key Observations:

1. Multiple transitions: Interrupt → Softirq → Kernel thread → User process
2. Kernel as intermediary: All data passes through kernel buffers
3. Scheduler involvement: Process must be scheduled to run recv()
4. Copy operations: Data copied from NIC → kernel → user space
5. Sub-millisecond: Modern systems process packets very quickly

What Affects Latency:

Ultra-Low-Latency Approaches:

• Kernel bypass (DPDK): Application directly accesses NIC, no kernel
• Busy polling: Process spins waiting for data instead of sleeping
• CPU pinning: Dedicate CPU cores to network processing

We've completed our exploration of multiplexing and demultiplexing with process mapping—the final link connecting network segments to application processes. Let's consolidate the key concepts:

Module Complete:

You've now mastered the complete multiplexing and demultiplexing system:

1. Multiplexing at sender: Aggregating data from multiple sockets into a single outbound stream
2. Demultiplexing at receiver: Routing incoming segments to correct sockets based on ports and connections
3. Port identification: The 16-bit identifiers enabling process-level addressing
4. Connection identification: The 4-tuple enabling TCP's massive concurrency
5. Process mapping: The final delivery from socket to application process

This knowledge forms the foundation for understanding how all networked applications function—from simple clients to high-performance servers handling millions of connections.