Blocking Non Blocking Io - Learning Module

Loading content...

0/240

select/poll/epoll

The Three Generations of Multi-plexing

Unix's journey to scalable I/O spans four decades and three major APIs: select() (1983), poll() (1986), and epoll (2002). Each generation addressed limitations of its predecessor, culminating in epoll's ability to handle millions of connections efficiently.

Understanding all three matters. select() is universal but limited; poll() is portable but slow at scale; epoll is performant but Linux-specific. Real-world systems use all three depending on requirements. This page provides deep, practical coverage of each mechanism.

What You Will Learn

By the end of this page, you will master select(), poll(), and epoll APIs. You'll understand their performance characteristics through first principles, recognize when to use each, and be able to write robust multiplexed I/O code. We'll cover APIs, edge cases, performance traps, and production patterns.

select() — The Original

select() is the oldest and most portable I/O multiplexing mechanism. Originating in 4.2BSD (1983), it's available on virtually every Unix-like system and even Windows (with slight differences).

Function signature:

select_signature.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#include <sys/select.h>
 
int select(
    int nfds,              // Highest fd + 1
    fd_set *readfds,       // Watch for readability
    fd_set *writefds,      // Watch for writability  
    fd_set *exceptfds,     // Watch for exceptions
    struct timeval *timeout // NULL = block forever, {0,0} = poll
);
 
// Returns:
// > 0: Number of ready descriptors
// = 0: Timeout expired, none ready
// < 0: Error (check errno)
 
// fd_set manipulation macros
void FD_ZERO(fd_set *set);           // Clear all bits
void FD_SET(int fd, fd_set *set);    // Set bit for fd
void FD_CLR(int fd, fd_set *set);    // Clear bit for fd  
int  FD_ISSET(int fd, fd_set *set);  // Test if fd's bit is set

How select() works:

Create three fd_set bitmasks (readfds, writefds, exceptfds)
Set bits for fds you're interested in
Call select() — it blocks until at least one fd is ready
Check which fds are ready using FD_ISSET
Important: select() modifies the fd_sets. You must rebuild them before each call.

Complete example:

select_example.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
#include <sys/select.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
 
#define MAX_CLIENTS 100
 
/**
 * Simple echo server using select()
 */
int main() {
    int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
    
    // Bind and listen setup...
    struct sockaddr_in addr = {
        .sin_family = AF_INET,
        .sin_addr.s_addr = INADDR_ANY,
        .sin_port = htons(8080)
    };
    
    int opt = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
    bind(listen_fd, (struct sockaddr *)&addr, sizeof(addr));
    listen(listen_fd, 128);
    
    int clients[MAX_CLIENTS];
    int num_clients = 0;
    
    printf("Server listening on port 8080
");
    
    while (1) {
        // MUST rebuild fd_sets every iteration - select modifies them!
        fd_set readfds;
        FD_ZERO(&readfds);
        
        // Always watch the listening socket for new connections
        FD_SET(listen_fd, &readfds);
        int max_fd = listen_fd;
        
        // Add all connected clients
        for (int i = 0; i < num_clients; i++) {
            FD_SET(clients[i], &readfds);
            if (clients[i] > max_fd) max_fd = clients[i];
        }
        
        // Wait for activity (block indefinitely)
        int ready = select(max_fd + 1, &readfds, NULL, NULL, NULL);
        
        if (ready < 0) {
            if (errno == EINTR) continue;  // Interrupted by signal
            perror("select");
            break;
        }
        
        // Check for new connections
        if (FD_ISSET(listen_fd, &readfds)) {
            int new_fd = accept(listen_fd, NULL, NULL);
            if (new_fd >= 0 && num_clients < MAX_CLIENTS) {
                clients[num_clients++] = new_fd;
                printf("New client: fd=%d (total: %d)
", new_fd, num_clients);
            }
        }
        
        // Check clients for data
        for (int i = 0; i < num_clients; i++) {
            if (FD_ISSET(clients[i], &readfds)) {
                char buf[1024];
                ssize_t n = read(clients[i], buf, sizeof(buf));
                
                if (n <= 0) {
                    // Client disconnected or error
                    printf("Client fd=%d disconnected
", clients[i]);
                    close(clients[i]);
                    // Remove from array (swap with last)
                    clients[i] = clients[--num_clients];
                    i--;  // Recheck this index
                } else {
                    // Echo back
                    write(clients[i], buf, n);
                }
            }
        }
    }
    
    close(listen_fd);
    return 0;
}

select() Limitations

FD_SETSIZE limit: fd_set is a fixed-size bitmask (typically 1024 bits). File descriptors >= FD_SETSIZE cause undefined behavior.
O(n) scanning: Both kernel and user code must scan all fds.
fd_sets modified: Must be rebuilt before every call.
nfds overhead: Must track the maximum fd.

For these reasons, select() is unsuitable for high-scale servers.

poll() — Removing the Limits

poll() was introduced in SVR4 to address select()'s fd limit. Instead of fixed-size bitmasks, poll() uses an array of pollfd structures.

Function signature:

poll_signature.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#include <poll.h>
 
int poll(
    struct pollfd *fds,    // Array of descriptors to watch
    nfds_t nfds,           // Number of entries in array
    int timeout            // Milliseconds, -1 = forever, 0 = poll
);
 
struct pollfd {
    int   fd;        // File descriptor
    short events;    // Events we're interested in (input)
    short revents;   // Events that occurred (output)
};
 
// Common event flags:
// POLLIN   - Data available to read (or connection request)
// POLLOUT  - Write won't block
// POLLHUP  - Hang up (peer closed connection)
// POLLERR  - Error condition
// POLLNVAL - Invalid fd
 
// Returns:
// > 0: Number of pollfd with non-zero revents
// = 0: Timeout expired
// < 0: Error

Advantages over select():

No artificial fd limit (limited only by memory)
Cleaner separation of input (events) and output (revents)
revents cleared by kernel, so partial array rebuilding is possible
More event types available (POLLHUP, POLLNVAL)

Complete example:

poll_example.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
#include <poll.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
 
#define INITIAL_CAPACITY 64
 
/**
 * Echo server using poll() with dynamic array
 */
int main() {
    int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
    
    // Bind and listen...
    struct sockaddr_in addr = {
        .sin_family = AF_INET,
        .sin_addr.s_addr = INADDR_ANY,
        .sin_port = htons(8080)
    };
    int opt = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
    bind(listen_fd, (struct sockaddr *)&addr, sizeof(addr));
    listen(listen_fd, 128);
    
    // Dynamic array of pollfds
    struct pollfd *pollfds = malloc(INITIAL_CAPACITY * sizeof(struct pollfd));
    int capacity = INITIAL_CAPACITY;
    int nfds = 1;  // Start with just listen socket
    
    // First entry is always the listening socket
    pollfds[0].fd = listen_fd;
    pollfds[0].events = POLLIN;
    pollfds[0].revents = 0;
    
    printf("Server listening on port 8080
");
    
    while (1) {
        // Unlike select, we don't need to rebuild the entire array
        // Just ensure revents are checked correctly
        
        int ready = poll(pollfds, nfds, -1);  // Block forever
        
        if (ready < 0) {
            if (errno == EINTR) continue;
            perror("poll");
            break;
        }
        
        // Check listening socket first
        if (pollfds[0].revents & POLLIN) {
            int new_fd = accept(listen_fd, NULL, NULL);
            if (new_fd >= 0) {
                // Grow array if needed
                if (nfds >= capacity) {
                    capacity *= 2;
                    pollfds = realloc(pollfds, capacity * sizeof(struct pollfd));
                }
                
                pollfds[nfds].fd = new_fd;
                pollfds[nfds].events = POLLIN;
                pollfds[nfds].revents = 0;
                nfds++;
                
                printf("New client: fd=%d (total: %d)
", new_fd, nfds - 1);
            }
        }
        
        // Check client sockets
        for (int i = 1; i < nfds; i++) {
            if (pollfds[i].revents & (POLLIN | POLLHUP | POLLERR)) {
                char buf[1024];
                ssize_t n = read(pollfds[i].fd, buf, sizeof(buf));
                
                if (n <= 0) {
                    printf("Client fd=%d disconnected
", pollfds[i].fd);
                    close(pollfds[i].fd);
                    
                    // Swap with last entry
                    pollfds[i] = pollfds[--nfds];
                    i--;  // Recheck this index
                } else {
                    // Echo back
                    write(pollfds[i].fd, buf, n);
                }
            }
        }
    }
    
    free(pollfds);
    close(listen_fd);
    return 0;
}

poll() still has O(n) performance:

Despite removing the fd limit, poll() still requires:

Copying the entire pollfd array to kernel space on every call
The kernel scanning all entries to find ready fds
User code scanning all entries to check revents

With 10,000 connections but only 10 active at any moment, poll() still does O(10,000) work. This becomes problematic at scale.

ppoll() — poll() with Signals

ppoll() is like poll() but allows atomically setting a signal mask during the wait. This helps avoid race conditions between signal checks and blocking. It's the POSIX-preferred interface for signal-safe polling:

int ppoll(struct pollfd *fds, nfds_t nfds, const struct timespec *tmo_p, const sigset_t *sigmask);

epoll — Scalable Multiplexing

epoll is Linux's answer to the C10K problem. Introduced in Linux 2.6 (2002), it provides O(1) event notification regardless of the number of monitored file descriptors.

The key insight:

Instead of passing the entire interest set on every call, epoll maintains a persistent kernel-side data structure. You register interest once; the kernel tracks it. When you wait, only ready events are returned—no scanning.

Three-step API:

epoll_api.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#include <sys/epoll.h>
 
// Step 1: Create an epoll instance
int epoll_create1(int flags);
// Returns: epoll file descriptor, -1 on error
// flags: 0 or EPOLL_CLOEXEC
 
// Step 2: Control monitored fds (add/modify/delete)
int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event);
// op: EPOLL_CTL_ADD, EPOLL_CTL_MOD, EPOLL_CTL_DEL
 
struct epoll_event {
    uint32_t events;    // Bitmask of events (EPOLLIN, EPOLLOUT, etc.)
    epoll_data_t data;  // User data (union: ptr, fd, u32, u64)
};
 
typedef union epoll_data {
    void    *ptr;
    int      fd;
    uint32_t u32;
    uint64_t u64;
} epoll_data_t;
 
// Step 3: Wait for events
int epoll_wait(int epfd, struct epoll_event *events, 
               int maxevents, int timeout);
// Returns: number of ready fds, 0 on timeout, -1 on error
// events: output array of ready events
// timeout: milliseconds, -1 = forever, 0 = poll
 
// Common event flags:
// EPOLLIN      - Read ready
// EPOLLOUT     - Write ready
// EPOLLERR     - Error (always monitored)
// EPOLLHUP     - Hang up (always monitored)
// EPOLLET      - Edge-triggered mode
// EPOLLONESHOT - Disable after one event (must re-arm)
// EPOLLRDHUP   - Peer closed connection (Linux 2.6.17+)

Complete example:

epoll_example.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
#include <sys/epoll.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
 
#define MAX_EVENTS 1024
 
// Make fd non-blocking (essential for edge-triggered)
int set_nonblocking(int fd) {
    int flags = fcntl(fd, F_GETFL, 0);
    return fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}
 
/**
 * Scalable echo server using epoll
 */
int main() {
    int listen_fd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
    
    struct sockaddr_in addr = {
        .sin_family = AF_INET,
        .sin_addr.s_addr = INADDR_ANY,
        .sin_port = htons(8080)
    };
    int opt = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
    bind(listen_fd, (struct sockaddr *)&addr, sizeof(addr));
    listen(listen_fd, 1024);
    
    // Create epoll instance
    int epoll_fd = epoll_create1(0);
    if (epoll_fd < 0) {
        perror("epoll_create1");
        return 1;
    }
    
    // Register listening socket
    struct epoll_event ev;
    ev.events = EPOLLIN;
    ev.data.fd = listen_fd;
    epoll_ctl(epoll_fd, EPOLL_CTL_ADD, listen_fd, &ev);
    
    struct epoll_event events[MAX_EVENTS];
    
    printf("Server listening on port 8080
");
    
    while (1) {
        // Wait for events - this is O(1) regardless of fd count!
        int nready = epoll_wait(epoll_fd, events, MAX_EVENTS, -1);
        
        if (nready < 0) {
            if (errno == EINTR) continue;
            perror("epoll_wait");
            break;
        }
        
        // Only iterate over READY fds (not all monitored fds)
        for (int i = 0; i < nready; i++) {
            int fd = events[i].data.fd;
            uint32_t revents = events[i].events;
            
            if (fd == listen_fd) {
                // Accept all pending connections
                while (1) {
                    int client = accept4(listen_fd, NULL, NULL, SOCK_NONBLOCK);
                    if (client < 0) {
                        if (errno == EAGAIN || errno == EWOULDBLOCK) {
                            break;  // No more pending
                        }
                        perror("accept");
                        break;
                    }
                    
                    // Register new client
                    ev.events = EPOLLIN | EPOLLET;  // Edge-triggered
                    ev.data.fd = client;
                    epoll_ctl(epoll_fd, EPOLL_CTL_ADD, client, &ev);
                    
                    printf("New client: fd=%d
", client);
                }
            } else {
                // Client socket
                if (revents & (EPOLLERR | EPOLLHUP)) {
                    printf("Client fd=%d error/hangup
", fd);
                    epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, NULL);
                    close(fd);
                    continue;
                }
                
                if (revents & EPOLLIN) {
                    // Edge-triggered: must read all available data
                    char buf[4096];
                    while (1) {
                        ssize_t n = read(fd, buf, sizeof(buf));
                        
                        if (n > 0) {
                            // Echo back (simplified - should handle partial writes)
                            write(fd, buf, n);
                        } else if (n == 0) {
                            // Client closed
                            printf("Client fd=%d closed
", fd);
                            epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, NULL);
                            close(fd);
                            break;
                        } else {
                            if (errno == EAGAIN || errno == EWOULDBLOCK) {
                                break;  // No more data for now
                            }
                            perror("read");
                            epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, NULL);
                            close(fd);
                            break;
                        }
                    }
                }
            }
        }
    }
    
    close(epoll_fd);
    close(listen_fd);
    return 0;
}

Why epoll is O(1)

The kernel maintains a red-black tree of monitored fds and a linked list of ready fds. When data arrives, the socket's callback adds it to the ready list (O(1)). epoll_wait() just returns the ready list—no scanning needed. With 100,000 monitored fds but only 10 ready, epoll does O(10) work, not O(100,000).

Performance Deep Dive

Let's analyze the performance characteristics of each mechanism in detail.

Performance Characteristics
Operation	select()	poll()	epoll
Add fd to watch set	O(1) - set bit	O(1) - set array element	O(log N) - tree insert
Remove fd from watch set	O(1) - clear bit	O(N) - find in array	O(log N) - tree delete
Per-call kernel overhead	O(N) - copy & scan fd_set	O(N) - copy & scan array	O(1) - check ready list
Return to user space	O(N) - copy modified fd_set	O(N) - copy modified array	O(ready) - copy ready events
User-side checking	O(N) - scan all bits	O(N) - scan all entries	O(ready) - only ready events
Memory per wait call	O(N) - 3 fd_sets on stack	O(N) - pollfd array	O(maxevents) - event buffer

Benchmark scenario:

10,000 connected clients, 100 active at any moment:

select(): 10,000 bits copied to/from kernel, scanned in kernel, scanned in user code. Same work whether 1 or 100 are active.
poll(): 10,000 * sizeof(pollfd) ≈ 80KB copied to/from kernel. Same scanning overhead.
epoll: ~100 events returned. Only 100 * sizeof(epoll_event) copied. No scanning of idle connections.

The difference grows with scale. At 100,000 connections, select/poll become CPU-bound; epoll remains efficient.

benchmark_differences.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
/*
 * Simplified performance model
 * 
 * N = total monitored fds
 * A = active (ready) fds per call
 * 
 * select/poll per call:
 * - Kernel: O(N) to copy and scan
 * - User: O(N) to check results
 * - Total: O(N) regardless of A
 * 
 * epoll per call:
 * - Kernel: O(A) to return ready events
 * - User: O(A) to process ready events
 * - Total: O(A)
 * 
 * At high N with low A (typical server pattern):
 * - Server with 10,000 clients, ~100 active per call
 * - select/poll: 10,000 operations per call
 * - epoll: 100 operations per call
 * 
 * Result: epoll is 100x more efficient in this scenario
 * 
 * At low N, the overhead matters less:
 * - Application with 10 fds
 * - All mechanisms perform similarly
 * - select() is actually fastest for very small sets
 */
 
/*
 * Real-world benchmark example
 * (Representative numbers from production systems)
 * 
 * HTTP server, 1000 requests/second, 10000 connections:
 * 
 * select():
 * - ~50% CPU in kernel scanning fd_sets
 * - ~500 syscalls/second wasted on idle connections
 * 
 * poll():
 * - ~40% CPU in kernel scanning pollfds
 * - Better than select (no FD_SETSIZE limit)
 * 
 * epoll:
 * - ~5% CPU in I/O handling
 * - Scales to 100K+ connections
 */

epoll Isn't Always Faster

For very small fd sets (< ~100), select() or poll() can be faster due to lower overhead. epoll's tree operations and kernel data structures have fixed costs. Use epoll when scaling to hundreds or thousands of connections; for small counts, simpler mechanisms suffice.

epoll Edge Cases and Gotchas

epoll has several subtleties that can trip up developers. Understanding these is crucial for production use.

Edge-Triggered Mode Pitfalls

With EPOLLET, you get one notification per state change. If you don't handle it completely, data can be left unprocessed forever.

edge_triggered_correct.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// CORRECT: Edge-triggered read loop
void handle_read_et(int fd) {
    char buf[4096];
    
    // MUST loop until EAGAIN
    while (1) {
        ssize_t n = read(fd, buf, sizeof(buf));
        
        if (n > 0) {
            process_data(buf, n);
            continue;  // There may be more data!
        }
        
        if (n == 0) {
            // EOF - peer closed
            close_connection(fd);
            return;
        }
        
        // n < 0
        if (errno == EAGAIN || errno == EWOULDBLOCK) {
            // Done for now - will be notified when more arrives
            return;
        }
        
        // Actual error
        handle_error(fd);
        return;
    }
}
 
// WRONG: Will lose data!
void handle_read_et_BROKEN(int fd) {
    char buf[4096];
    ssize_t n = read(fd, buf, sizeof(buf));
    if (n > 0) process_data(buf, n);
    // BUG: If more than 4096 bytes arrived, rest is lost!
    // Edge-triggered won't notify again for existing data
}

EPOLLONESHOT for Thread Safety

In multithreaded scenarios, multiple threads might handle the same fd simultaneously. EPOLLONESHOT disables an fd after one event, requiring explicit re-arming.

epoll_oneshot.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// Thread-safe epoll usage with EPOLLONESHOT
 
// Registration with EPOLLONESHOT
struct epoll_event ev;
ev.events = EPOLLIN | EPOLLET | EPOLLONESHOT;
ev.data.fd = client_fd;
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, client_fd, &ev);
 
// After handling an event, MUST re-arm to receive more
void handle_client(int epoll_fd, int client_fd) {
    // ... process data ...
    
    // Re-arm the fd for more events
    struct epoll_event ev;
    ev.events = EPOLLIN | EPOLLET | EPOLLONESHOT;
    ev.data.fd = client_fd;
    epoll_ctl(epoll_fd, EPOLL_CTL_MOD, client_fd, &ev);
}
 
// Without EPOLLONESHOT, if thread A is handling fd X
// and more data arrives, thread B might also try to handle fd X
// = race condition and data corruption

File Descriptor Reuse

When a fd is closed and its number reused, epoll automatically removes it. But if you close a fd that's still registered, and then the same number gets assigned to a new socket, the old registration is gone—no need to explicitly delete.

fd_reuse.c
C
1
2
3
4
5
6
7
8
9
10
11
12
// fd reuse behavior
int fd = accept(listen_fd, NULL, NULL);  // fd = 5
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev);
 
// Later...
close(fd);  // Automatically unregisters fd 5 from epoll
 
// Even later...
int new_fd = accept(listen_fd, NULL, NULL);  // new_fd = 5 again!
// The old registration for '5' is gone
// Must register the new socket explicitly
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, new_fd, &ev);

Use EPOLLRDHUP

EPOLLRDHUP (Linux 2.6.17+) notifies you when the peer closes their write side. This lets you detect half-closes without attempting a read. Always include EPOLLRDHUP in your event mask for sockets.

Practical Patterns

Let's examine common patterns used in production multiplexed I/O systems.

Pattern 1: Connection State Machine

For protocols with multiple stages (connect, handshake, request, response), use explicit state tracking:

connection_state_machine.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
typedef enum {
    CONN_STATE_CONNECTING,
    CONN_STATE_HANDSHAKING,
    CONN_STATE_READING_REQUEST,
    CONN_STATE_PROCESSING,
    CONN_STATE_WRITING_RESPONSE,
    CONN_STATE_CLOSING
} ConnState;
 
typedef struct {
    int fd;
    ConnState state;
    char read_buf[8192];
    size_t read_len;
    char write_buf[8192];
    size_t write_len;
    size_t write_pos;
} Connection;
 
void handle_connection(Connection *conn, uint32_t events) {
    switch (conn->state) {
        case CONN_STATE_CONNECTING:
            if (events & EPOLLOUT) {
                // Check connection result
                int err;
                socklen_t len = sizeof(err);
                getsockopt(conn->fd, SOL_SOCKET, SO_ERROR, &err, &len);
                if (err == 0) {
                    conn->state = CONN_STATE_HANDSHAKING;
                    // Modify epoll for read
                    modify_epoll(conn->fd, EPOLLIN);
                }
            }
            break;
            
        case CONN_STATE_READING_REQUEST:
            if (events & EPOLLIN) {
                // Read available data
                ssize_t n = read_available(conn);
                if (request_complete(conn)) {
                    conn->state = CONN_STATE_PROCESSING;
                    process_request(conn);
                    conn->state = CONN_STATE_WRITING_RESPONSE;
                    // Switch to write mode
                    modify_epoll(conn->fd, EPOLLOUT);
                }
            }
            break;
            
        case CONN_STATE_WRITING_RESPONSE:
            if (events & EPOLLOUT) {
                // Write pending data
                write_pending(conn);
                if (conn->write_pos >= conn->write_len) {
                    // Response complete
                    conn->state = CONN_STATE_READING_REQUEST;
                    modify_epoll(conn->fd, EPOLLIN);
                }
            }
            break;
            
        // ... other states
    }
}

Pattern 2: Accept Loop

When using edge-triggered mode on the listening socket, accept all pending connections:

accept_loop.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
void handle_accept(int listen_fd, int epoll_fd) {
    // In edge-triggered mode, MUST accept ALL pending connections
    while (1) {
        struct sockaddr_in client_addr;
        socklen_t len = sizeof(client_addr);
        
        int client = accept4(listen_fd, 
                            (struct sockaddr *)&client_addr, 
                            &len,
                            SOCK_NONBLOCK | SOCK_CLOEXEC);
        
        if (client < 0) {
            if (errno == EAGAIN || errno == EWOULDBLOCK) {
                // No more pending connections
                break;
            }
            if (errno == EMFILE || errno == ENFILE) {
                // Too many open files - log and continue
                // Consider closing idle connections
                log_error("File descriptor limit reached");
                break;
            }
            perror("accept4");
            break;
        }
        
        // Set up new connection
        Connection *conn = connection_create(client);
        
        struct epoll_event ev;
        ev.events = EPOLLIN | EPOLLET | EPOLLRDHUP;
        ev.data.ptr = conn;  // Store connection pointer
        
        if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, client, &ev) < 0) {
            perror("epoll_ctl add");
            connection_destroy(conn);
            close(client);
            continue;
        }
    }
}

Pattern 3: Write Buffer Management

Handle partial writes by buffering and monitoring EPOLLOUT:

write_buffer.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
// When write would block, enable EPOLLOUT and buffer data
 
int send_data(Connection *conn, const void *data, size_t len) {
    // If buffer has pending data, append and wait
    if (conn->write_len > conn->write_pos) {
        return buffer_append(conn, data, len);
    }
    
    // Try to write directly
    ssize_t written = write(conn->fd, data, len);
    
    if (written == len) {
        return 0;  // All sent
    }
    
    if (written < 0) {
        if (errno != EAGAIN && errno != EWOULDBLOCK) {
            return -1;  // Error
        }
        written = 0;
    }
    
    // Partial write - buffer remainder and watch for writability
    buffer_append(conn, (char *)data + written, len - written);
    
    // Add EPOLLOUT to watch for write readiness
    struct epoll_event ev;
    ev.events = EPOLLIN | EPOLLOUT | EPOLLET | EPOLLRDHUP;
    ev.data.ptr = conn;
    epoll_ctl(epoll_fd, EPOLL_CTL_MOD, conn->fd, &ev);
    
    return 0;
}
 
void handle_write_ready(Connection *conn) {
    while (conn->write_pos < conn->write_len) {
        ssize_t n = write(conn->fd, 
                          conn->write_buf + conn->write_pos,
                          conn->write_len - conn->write_pos);
        
        if (n < 0) {
            if (errno == EAGAIN || errno == EWOULDBLOCK) {
                return;  // Wait for next EPOLLOUT
            }
            // Error
            close_connection(conn);
            return;
        }
        
        conn->write_pos += n;
    }
    
    // All data sent, disable EPOLLOUT
    conn->write_len = 0;
    conn->write_pos = 0;
    
    struct epoll_event ev;
    ev.events = EPOLLIN | EPOLLET | EPOLLRDHUP;  // No EPOLLOUT
    ev.data.ptr = conn;
    epoll_ctl(epoll_fd, EPOLL_CTL_MOD, conn->fd, &ev);
}

Choosing the Right Mechanism

Each multiplexing mechanism has its place. Here's a decision guide:

Decision Guide

•select() — When: Extreme portability needed (even ancient systems), very small fd sets (< 64), simple one-off scripts. Why not: FD_SETSIZE limit, poor scaling.
•poll() — When: POSIX portability without fd limit, moderate scale (< 1000 connections), simple code preferred. Why not: Still O(n), copying overhead.
•epoll (level-triggered) — When: Linux-only is acceptable, moderate to high scale, simpler mental model. Why not: Slightly less efficient than edge-triggered.
•epoll (edge-triggered) — When: Highest performance needed, team understands the model, already using non-blocking I/O correctly. Why not: Easy to introduce bugs.
•Abstraction library (libevent, libuv) — When: Cross-platform needed, rapid development, don't want to manage multiplexing details. Why not: Overhead, dependency.

Quick Selection Guide
Scenario	Recommendation
Cross-platform CLI tool	select() or poll()
Cross-platform library/framework	libevent or libuv
Linux server, < 1000 connections	poll() or epoll LT
Linux server, thousands of connections	epoll LT or ET
Linux server, extreme scale (10K+)	epoll ET with careful coding
macOS/BSD server	kqueue (analogous to epoll)
Windows server	IOCP
Mobile app	Platform async framework

Start Simple, Optimize Later

Unless you're building for extreme scale from day one, start with the simplest approach that meets your needs. poll() is portable and works well for most applications. Optimize to epoll when profiling shows multiplexing overhead is significant.

Summary: select/poll/epoll

We've explored Unix's three generations of I/O multiplexing in depth. Let's consolidate the key insights:

Key Takeaways

•select() is universal but limited — 1024 fd limit, O(n) scanning, must rebuild fd_sets. Use for portability with small fd counts.
•poll() removes limits but scales poorly — No fd_set limit, cleaner API, but still O(n) per call. Use for moderate scale or portability.
•epoll delivers O(1) event notification — Persistent kernel state, returns only ready events. Use for Linux high-scale servers.
•Edge-triggered requires careful coding — Must drain fds completely or events are lost. Use for maximum performance when you understand the model.
•Non-blocking I/O is essential — Always use non-blocking mode with multiplexing for robust code.
•Write buffering is necessary — Handle partial writes by buffering and monitoring EPOLLOUT.
•Choose based on requirements — Portability, scale, complexity all factor into the decision.

Module Complete:

You've now mastered the fundamental I/O models: blocking, non-blocking, asynchronous, and multiplexed I/O. You understand the theoretical foundations and practical APIs. These concepts underpin virtually all high-performance servers, from web servers to databases to message queues.

The next modules in I/O Software cover buffering, caching, and spooling—techniques that optimize I/O performance at the application level.

Module Complete

Congratulations! You've completed the Blocking and Non-Blocking I/O module. You now understand blocking, non-blocking, and asynchronous I/O models; the concept and importance of I/O multiplexing; and the practical APIs of select, poll, and epoll. These skills are fundamental to building scalable, responsive systems.

select/poll/epoll

The Three Generations of Multi-plexing

What You Will Learn

select() — The Original

select() is the oldest and most portable I/O multiplexing mechanism. Originating in 4.2BSD (1983), it's available on virtually every Unix-like system and even Windows (with slight differences).

Function signature:

select_signature.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#include <sys/select.h>
 
int select(
    int nfds,              // Highest fd + 1
    fd_set *readfds,       // Watch for readability
    fd_set *writefds,      // Watch for writability  
    fd_set *exceptfds,     // Watch for exceptions
    struct timeval *timeout // NULL = block forever, {0,0} = poll
);
 
// Returns:
// > 0: Number of ready descriptors
// = 0: Timeout expired, none ready
// < 0: Error (check errno)
 
// fd_set manipulation macros
void FD_ZERO(fd_set *set);           // Clear all bits
void FD_SET(int fd, fd_set *set);    // Set bit for fd
void FD_CLR(int fd, fd_set *set);    // Clear bit for fd  
int  FD_ISSET(int fd, fd_set *set);  // Test if fd's bit is set

How select() works:

Create three fd_set bitmasks (readfds, writefds, exceptfds)
Set bits for fds you're interested in
Call select() — it blocks until at least one fd is ready
Check which fds are ready using FD_ISSET
Important: select() modifies the fd_sets. You must rebuild them before each call.

Complete example:

select_example.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
#include <sys/select.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
 
#define MAX_CLIENTS 100
 
/**
 * Simple echo server using select()
 */
int main() {
    int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
    
    // Bind and listen setup...
    struct sockaddr_in addr = {
        .sin_family = AF_INET,
        .sin_addr.s_addr = INADDR_ANY,
        .sin_port = htons(8080)
    };
    
    int opt = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
    bind(listen_fd, (struct sockaddr *)&addr, sizeof(addr));
    listen(listen_fd, 128);
    
    int clients[MAX_CLIENTS];
    int num_clients = 0;
    
    printf("Server listening on port 8080
");
    
    while (1) {
        // MUST rebuild fd_sets every iteration - select modifies them!
        fd_set readfds;
        FD_ZERO(&readfds);
        
        // Always watch the listening socket for new connections
        FD_SET(listen_fd, &readfds);
        int max_fd = listen_fd;
        
        // Add all connected clients
        for (int i = 0; i < num_clients; i++) {
            FD_SET(clients[i], &readfds);
            if (clients[i] > max_fd) max_fd = clients[i];
        }
        
        // Wait for activity (block indefinitely)
        int ready = select(max_fd + 1, &readfds, NULL, NULL, NULL);
        
        if (ready < 0) {
            if (errno == EINTR) continue;  // Interrupted by signal
            perror("select");
            break;
        }
        
        // Check for new connections
        if (FD_ISSET(listen_fd, &readfds)) {
            int new_fd = accept(listen_fd, NULL, NULL);
            if (new_fd >= 0 && num_clients < MAX_CLIENTS) {
                clients[num_clients++] = new_fd;
                printf("New client: fd=%d (total: %d)
", new_fd, num_clients);
            }
        }
        
        // Check clients for data
        for (int i = 0; i < num_clients; i++) {
            if (FD_ISSET(clients[i], &readfds)) {
                char buf[1024];
                ssize_t n = read(clients[i], buf, sizeof(buf));
                
                if (n <= 0) {
                    // Client disconnected or error
                    printf("Client fd=%d disconnected
", clients[i]);
                    close(clients[i]);
                    // Remove from array (swap with last)
                    clients[i] = clients[--num_clients];
                    i--;  // Recheck this index
                } else {
                    // Echo back
                    write(clients[i], buf, n);
                }
            }
        }
    }
    
    close(listen_fd);
    return 0;
}

select() Limitations

FD_SETSIZE limit: fd_set is a fixed-size bitmask (typically 1024 bits). File descriptors >= FD_SETSIZE cause undefined behavior.
O(n) scanning: Both kernel and user code must scan all fds.
fd_sets modified: Must be rebuilt before every call.
nfds overhead: Must track the maximum fd.

For these reasons, select() is unsuitable for high-scale servers.

poll() — Removing the Limits

poll() was introduced in SVR4 to address select()'s fd limit. Instead of fixed-size bitmasks, poll() uses an array of pollfd structures.

Function signature:

poll_signature.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#include <poll.h>
 
int poll(
    struct pollfd *fds,    // Array of descriptors to watch
    nfds_t nfds,           // Number of entries in array
    int timeout            // Milliseconds, -1 = forever, 0 = poll
);
 
struct pollfd {
    int   fd;        // File descriptor
    short events;    // Events we're interested in (input)
    short revents;   // Events that occurred (output)
};
 
// Common event flags:
// POLLIN   - Data available to read (or connection request)
// POLLOUT  - Write won't block
// POLLHUP  - Hang up (peer closed connection)
// POLLERR  - Error condition
// POLLNVAL - Invalid fd
 
// Returns:
// > 0: Number of pollfd with non-zero revents
// = 0: Timeout expired
// < 0: Error

Advantages over select():

No artificial fd limit (limited only by memory)
Cleaner separation of input (events) and output (revents)
revents cleared by kernel, so partial array rebuilding is possible
More event types available (POLLHUP, POLLNVAL)

Complete example:

poll_example.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
#include <poll.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
 
#define INITIAL_CAPACITY 64
 
/**
 * Echo server using poll() with dynamic array
 */
int main() {
    int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
    
    // Bind and listen...
    struct sockaddr_in addr = {
        .sin_family = AF_INET,
        .sin_addr.s_addr = INADDR_ANY,
        .sin_port = htons(8080)
    };
    int opt = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
    bind(listen_fd, (struct sockaddr *)&addr, sizeof(addr));
    listen(listen_fd, 128);
    
    // Dynamic array of pollfds
    struct pollfd *pollfds = malloc(INITIAL_CAPACITY * sizeof(struct pollfd));
    int capacity = INITIAL_CAPACITY;
    int nfds = 1;  // Start with just listen socket
    
    // First entry is always the listening socket
    pollfds[0].fd = listen_fd;
    pollfds[0].events = POLLIN;
    pollfds[0].revents = 0;
    
    printf("Server listening on port 8080
");
    
    while (1) {
        // Unlike select, we don't need to rebuild the entire array
        // Just ensure revents are checked correctly
        
        int ready = poll(pollfds, nfds, -1);  // Block forever
        
        if (ready < 0) {
            if (errno == EINTR) continue;
            perror("poll");
            break;
        }
        
        // Check listening socket first
        if (pollfds[0].revents & POLLIN) {
            int new_fd = accept(listen_fd, NULL, NULL);
            if (new_fd >= 0) {
                // Grow array if needed
                if (nfds >= capacity) {
                    capacity *= 2;
                    pollfds = realloc(pollfds, capacity * sizeof(struct pollfd));
                }
                
                pollfds[nfds].fd = new_fd;
                pollfds[nfds].events = POLLIN;
                pollfds[nfds].revents = 0;
                nfds++;
                
                printf("New client: fd=%d (total: %d)
", new_fd, nfds - 1);
            }
        }
        
        // Check client sockets
        for (int i = 1; i < nfds; i++) {
            if (pollfds[i].revents & (POLLIN | POLLHUP | POLLERR)) {
                char buf[1024];
                ssize_t n = read(pollfds[i].fd, buf, sizeof(buf));
                
                if (n <= 0) {
                    printf("Client fd=%d disconnected
", pollfds[i].fd);
                    close(pollfds[i].fd);
                    
                    // Swap with last entry
                    pollfds[i] = pollfds[--nfds];
                    i--;  // Recheck this index
                } else {
                    // Echo back
                    write(pollfds[i].fd, buf, n);
                }
            }
        }
    }
    
    free(pollfds);
    close(listen_fd);
    return 0;
}

poll() still has O(n) performance:

Despite removing the fd limit, poll() still requires:

Copying the entire pollfd array to kernel space on every call
The kernel scanning all entries to find ready fds
User code scanning all entries to check revents

With 10,000 connections but only 10 active at any moment, poll() still does O(10,000) work. This becomes problematic at scale.

ppoll() — poll() with Signals

int ppoll(struct pollfd *fds, nfds_t nfds, const struct timespec *tmo_p, const sigset_t *sigmask);

epoll — Scalable Multiplexing

epoll is Linux's answer to the C10K problem. Introduced in Linux 2.6 (2002), it provides O(1) event notification regardless of the number of monitored file descriptors.

The key insight:

Three-step API:

epoll_api.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#include <sys/epoll.h>
 
// Step 1: Create an epoll instance
int epoll_create1(int flags);
// Returns: epoll file descriptor, -1 on error
// flags: 0 or EPOLL_CLOEXEC
 
// Step 2: Control monitored fds (add/modify/delete)
int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event);
// op: EPOLL_CTL_ADD, EPOLL_CTL_MOD, EPOLL_CTL_DEL
 
struct epoll_event {
    uint32_t events;    // Bitmask of events (EPOLLIN, EPOLLOUT, etc.)
    epoll_data_t data;  // User data (union: ptr, fd, u32, u64)
};
 
typedef union epoll_data {
    void    *ptr;
    int      fd;
    uint32_t u32;
    uint64_t u64;
} epoll_data_t;
 
// Step 3: Wait for events
int epoll_wait(int epfd, struct epoll_event *events, 
               int maxevents, int timeout);
// Returns: number of ready fds, 0 on timeout, -1 on error
// events: output array of ready events
// timeout: milliseconds, -1 = forever, 0 = poll
 
// Common event flags:
// EPOLLIN      - Read ready
// EPOLLOUT     - Write ready
// EPOLLERR     - Error (always monitored)
// EPOLLHUP     - Hang up (always monitored)
// EPOLLET      - Edge-triggered mode
// EPOLLONESHOT - Disable after one event (must re-arm)
// EPOLLRDHUP   - Peer closed connection (Linux 2.6.17+)

Complete example:

epoll_example.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
#include <sys/epoll.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
 
#define MAX_EVENTS 1024
 
// Make fd non-blocking (essential for edge-triggered)
int set_nonblocking(int fd) {
    int flags = fcntl(fd, F_GETFL, 0);
    return fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}
 
/**
 * Scalable echo server using epoll
 */
int main() {
    int listen_fd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
    
    struct sockaddr_in addr = {
        .sin_family = AF_INET,
        .sin_addr.s_addr = INADDR_ANY,
        .sin_port = htons(8080)
    };
    int opt = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
    bind(listen_fd, (struct sockaddr *)&addr, sizeof(addr));
    listen(listen_fd, 1024);
    
    // Create epoll instance
    int epoll_fd = epoll_create1(0);
    if (epoll_fd < 0) {
        perror("epoll_create1");
        return 1;
    }
    
    // Register listening socket
    struct epoll_event ev;
    ev.events = EPOLLIN;
    ev.data.fd = listen_fd;
    epoll_ctl(epoll_fd, EPOLL_CTL_ADD, listen_fd, &ev);
    
    struct epoll_event events[MAX_EVENTS];
    
    printf("Server listening on port 8080
");
    
    while (1) {
        // Wait for events - this is O(1) regardless of fd count!
        int nready = epoll_wait(epoll_fd, events, MAX_EVENTS, -1);
        
        if (nready < 0) {
            if (errno == EINTR) continue;
            perror("epoll_wait");
            break;
        }
        
        // Only iterate over READY fds (not all monitored fds)
        for (int i = 0; i < nready; i++) {
            int fd = events[i].data.fd;
            uint32_t revents = events[i].events;
            
            if (fd == listen_fd) {
                // Accept all pending connections
                while (1) {
                    int client = accept4(listen_fd, NULL, NULL, SOCK_NONBLOCK);
                    if (client < 0) {
                        if (errno == EAGAIN || errno == EWOULDBLOCK) {
                            break;  // No more pending
                        }
                        perror("accept");
                        break;
                    }
                    
                    // Register new client
                    ev.events = EPOLLIN | EPOLLET;  // Edge-triggered
                    ev.data.fd = client;
                    epoll_ctl(epoll_fd, EPOLL_CTL_ADD, client, &ev);
                    
                    printf("New client: fd=%d
", client);
                }
            } else {
                // Client socket
                if (revents & (EPOLLERR | EPOLLHUP)) {
                    printf("Client fd=%d error/hangup
", fd);
                    epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, NULL);
                    close(fd);
                    continue;
                }
                
                if (revents & EPOLLIN) {
                    // Edge-triggered: must read all available data
                    char buf[4096];
                    while (1) {
                        ssize_t n = read(fd, buf, sizeof(buf));
                        
                        if (n > 0) {
                            // Echo back (simplified - should handle partial writes)
                            write(fd, buf, n);
                        } else if (n == 0) {
                            // Client closed
                            printf("Client fd=%d closed
", fd);
                            epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, NULL);
                            close(fd);
                            break;
                        } else {
                            if (errno == EAGAIN || errno == EWOULDBLOCK) {
                                break;  // No more data for now
                            }
                            perror("read");
                            epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, NULL);
                            close(fd);
                            break;
                        }
                    }
                }
            }
        }
    }
    
    close(epoll_fd);
    close(listen_fd);
    return 0;
}

Why epoll is O(1)

Performance Deep Dive

Let's analyze the performance characteristics of each mechanism in detail.

Performance Characteristics
Operation	select()	poll()	epoll
Add fd to watch set	O(1) - set bit	O(1) - set array element	O(log N) - tree insert
Remove fd from watch set	O(1) - clear bit	O(N) - find in array	O(log N) - tree delete
Per-call kernel overhead	O(N) - copy & scan fd_set	O(N) - copy & scan array	O(1) - check ready list
Return to user space	O(N) - copy modified fd_set	O(N) - copy modified array	O(ready) - copy ready events
User-side checking	O(N) - scan all bits	O(N) - scan all entries	O(ready) - only ready events
Memory per wait call	O(N) - 3 fd_sets on stack	O(N) - pollfd array	O(maxevents) - event buffer

Benchmark scenario:

10,000 connected clients, 100 active at any moment:

select(): 10,000 bits copied to/from kernel, scanned in kernel, scanned in user code. Same work whether 1 or 100 are active.
poll(): 10,000 * sizeof(pollfd) ≈ 80KB copied to/from kernel. Same scanning overhead.
epoll: ~100 events returned. Only 100 * sizeof(epoll_event) copied. No scanning of idle connections.

The difference grows with scale. At 100,000 connections, select/poll become CPU-bound; epoll remains efficient.

benchmark_differences.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
/*
 * Simplified performance model
 * 
 * N = total monitored fds
 * A = active (ready) fds per call
 * 
 * select/poll per call:
 * - Kernel: O(N) to copy and scan
 * - User: O(N) to check results
 * - Total: O(N) regardless of A
 * 
 * epoll per call:
 * - Kernel: O(A) to return ready events
 * - User: O(A) to process ready events
 * - Total: O(A)
 * 
 * At high N with low A (typical server pattern):
 * - Server with 10,000 clients, ~100 active per call
 * - select/poll: 10,000 operations per call
 * - epoll: 100 operations per call
 * 
 * Result: epoll is 100x more efficient in this scenario
 * 
 * At low N, the overhead matters less:
 * - Application with 10 fds
 * - All mechanisms perform similarly
 * - select() is actually fastest for very small sets
 */
 
/*
 * Real-world benchmark example
 * (Representative numbers from production systems)
 * 
 * HTTP server, 1000 requests/second, 10000 connections:
 * 
 * select():
 * - ~50% CPU in kernel scanning fd_sets
 * - ~500 syscalls/second wasted on idle connections
 * 
 * poll():
 * - ~40% CPU in kernel scanning pollfds
 * - Better than select (no FD_SETSIZE limit)
 * 
 * epoll:
 * - ~5% CPU in I/O handling
 * - Scales to 100K+ connections
 */

epoll Isn't Always Faster

epoll Edge Cases and Gotchas

epoll has several subtleties that can trip up developers. Understanding these is crucial for production use.

Edge-Triggered Mode Pitfalls

With EPOLLET, you get one notification per state change. If you don't handle it completely, data can be left unprocessed forever.

edge_triggered_correct.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// CORRECT: Edge-triggered read loop
void handle_read_et(int fd) {
    char buf[4096];
    
    // MUST loop until EAGAIN
    while (1) {
        ssize_t n = read(fd, buf, sizeof(buf));
        
        if (n > 0) {
            process_data(buf, n);
            continue;  // There may be more data!
        }
        
        if (n == 0) {
            // EOF - peer closed
            close_connection(fd);
            return;
        }
        
        // n < 0
        if (errno == EAGAIN || errno == EWOULDBLOCK) {
            // Done for now - will be notified when more arrives
            return;
        }
        
        // Actual error
        handle_error(fd);
        return;
    }
}
 
// WRONG: Will lose data!
void handle_read_et_BROKEN(int fd) {
    char buf[4096];
    ssize_t n = read(fd, buf, sizeof(buf));
    if (n > 0) process_data(buf, n);
    // BUG: If more than 4096 bytes arrived, rest is lost!
    // Edge-triggered won't notify again for existing data
}

EPOLLONESHOT for Thread Safety

In multithreaded scenarios, multiple threads might handle the same fd simultaneously. EPOLLONESHOT disables an fd after one event, requiring explicit re-arming.

epoll_oneshot.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// Thread-safe epoll usage with EPOLLONESHOT
 
// Registration with EPOLLONESHOT
struct epoll_event ev;
ev.events = EPOLLIN | EPOLLET | EPOLLONESHOT;
ev.data.fd = client_fd;
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, client_fd, &ev);
 
// After handling an event, MUST re-arm to receive more
void handle_client(int epoll_fd, int client_fd) {
    // ... process data ...
    
    // Re-arm the fd for more events
    struct epoll_event ev;
    ev.events = EPOLLIN | EPOLLET | EPOLLONESHOT;
    ev.data.fd = client_fd;
    epoll_ctl(epoll_fd, EPOLL_CTL_MOD, client_fd, &ev);
}
 
// Without EPOLLONESHOT, if thread A is handling fd X
// and more data arrives, thread B might also try to handle fd X
// = race condition and data corruption

File Descriptor Reuse

fd_reuse.c
C
1
2
3
4
5
6
7
8
9
10
11
12
// fd reuse behavior
int fd = accept(listen_fd, NULL, NULL);  // fd = 5
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev);
 
// Later...
close(fd);  // Automatically unregisters fd 5 from epoll
 
// Even later...
int new_fd = accept(listen_fd, NULL, NULL);  // new_fd = 5 again!
// The old registration for '5' is gone
// Must register the new socket explicitly
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, new_fd, &ev);

Use EPOLLRDHUP

EPOLLRDHUP (Linux 2.6.17+) notifies you when the peer closes their write side. This lets you detect half-closes without attempting a read. Always include EPOLLRDHUP in your event mask for sockets.

Practical Patterns

Let's examine common patterns used in production multiplexed I/O systems.

Pattern 1: Connection State Machine

For protocols with multiple stages (connect, handshake, request, response), use explicit state tracking:

connection_state_machine.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
typedef enum {
    CONN_STATE_CONNECTING,
    CONN_STATE_HANDSHAKING,
    CONN_STATE_READING_REQUEST,
    CONN_STATE_PROCESSING,
    CONN_STATE_WRITING_RESPONSE,
    CONN_STATE_CLOSING
} ConnState;
 
typedef struct {
    int fd;
    ConnState state;
    char read_buf[8192];
    size_t read_len;
    char write_buf[8192];
    size_t write_len;
    size_t write_pos;
} Connection;
 
void handle_connection(Connection *conn, uint32_t events) {
    switch (conn->state) {
        case CONN_STATE_CONNECTING:
            if (events & EPOLLOUT) {
                // Check connection result
                int err;
                socklen_t len = sizeof(err);
                getsockopt(conn->fd, SOL_SOCKET, SO_ERROR, &err, &len);
                if (err == 0) {
                    conn->state = CONN_STATE_HANDSHAKING;
                    // Modify epoll for read
                    modify_epoll(conn->fd, EPOLLIN);
                }
            }
            break;
            
        case CONN_STATE_READING_REQUEST:
            if (events & EPOLLIN) {
                // Read available data
                ssize_t n = read_available(conn);
                if (request_complete(conn)) {
                    conn->state = CONN_STATE_PROCESSING;
                    process_request(conn);
                    conn->state = CONN_STATE_WRITING_RESPONSE;
                    // Switch to write mode
                    modify_epoll(conn->fd, EPOLLOUT);
                }
            }
            break;
            
        case CONN_STATE_WRITING_RESPONSE:
            if (events & EPOLLOUT) {
                // Write pending data
                write_pending(conn);
                if (conn->write_pos >= conn->write_len) {
                    // Response complete
                    conn->state = CONN_STATE_READING_REQUEST;
                    modify_epoll(conn->fd, EPOLLIN);
                }
            }
            break;
            
        // ... other states
    }
}

Pattern 2: Accept Loop

When using edge-triggered mode on the listening socket, accept all pending connections:

accept_loop.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
void handle_accept(int listen_fd, int epoll_fd) {
    // In edge-triggered mode, MUST accept ALL pending connections
    while (1) {
        struct sockaddr_in client_addr;
        socklen_t len = sizeof(client_addr);
        
        int client = accept4(listen_fd, 
                            (struct sockaddr *)&client_addr, 
                            &len,
                            SOCK_NONBLOCK | SOCK_CLOEXEC);
        
        if (client < 0) {
            if (errno == EAGAIN || errno == EWOULDBLOCK) {
                // No more pending connections
                break;
            }
            if (errno == EMFILE || errno == ENFILE) {
                // Too many open files - log and continue
                // Consider closing idle connections
                log_error("File descriptor limit reached");
                break;
            }
            perror("accept4");
            break;
        }
        
        // Set up new connection
        Connection *conn = connection_create(client);
        
        struct epoll_event ev;
        ev.events = EPOLLIN | EPOLLET | EPOLLRDHUP;
        ev.data.ptr = conn;  // Store connection pointer
        
        if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, client, &ev) < 0) {
            perror("epoll_ctl add");
            connection_destroy(conn);
            close(client);
            continue;
        }
    }
}

Pattern 3: Write Buffer Management

Handle partial writes by buffering and monitoring EPOLLOUT:

write_buffer.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
// When write would block, enable EPOLLOUT and buffer data
 
int send_data(Connection *conn, const void *data, size_t len) {
    // If buffer has pending data, append and wait
    if (conn->write_len > conn->write_pos) {
        return buffer_append(conn, data, len);
    }
    
    // Try to write directly
    ssize_t written = write(conn->fd, data, len);
    
    if (written == len) {
        return 0;  // All sent
    }
    
    if (written < 0) {
        if (errno != EAGAIN && errno != EWOULDBLOCK) {
            return -1;  // Error
        }
        written = 0;
    }
    
    // Partial write - buffer remainder and watch for writability
    buffer_append(conn, (char *)data + written, len - written);
    
    // Add EPOLLOUT to watch for write readiness
    struct epoll_event ev;
    ev.events = EPOLLIN | EPOLLOUT | EPOLLET | EPOLLRDHUP;
    ev.data.ptr = conn;
    epoll_ctl(epoll_fd, EPOLL_CTL_MOD, conn->fd, &ev);
    
    return 0;
}
 
void handle_write_ready(Connection *conn) {
    while (conn->write_pos < conn->write_len) {
        ssize_t n = write(conn->fd, 
                          conn->write_buf + conn->write_pos,
                          conn->write_len - conn->write_pos);
        
        if (n < 0) {
            if (errno == EAGAIN || errno == EWOULDBLOCK) {
                return;  // Wait for next EPOLLOUT
            }
            // Error
            close_connection(conn);
            return;
        }
        
        conn->write_pos += n;
    }
    
    // All data sent, disable EPOLLOUT
    conn->write_len = 0;
    conn->write_pos = 0;
    
    struct epoll_event ev;
    ev.events = EPOLLIN | EPOLLET | EPOLLRDHUP;  // No EPOLLOUT
    ev.data.ptr = conn;
    epoll_ctl(epoll_fd, EPOLL_CTL_MOD, conn->fd, &ev);
}

Choosing the Right Mechanism

Each multiplexing mechanism has its place. Here's a decision guide:

Decision Guide

•select() — When: Extreme portability needed (even ancient systems), very small fd sets (< 64), simple one-off scripts. Why not: FD_SETSIZE limit, poor scaling.
•poll() — When: POSIX portability without fd limit, moderate scale (< 1000 connections), simple code preferred. Why not: Still O(n), copying overhead.
•epoll (level-triggered) — When: Linux-only is acceptable, moderate to high scale, simpler mental model. Why not: Slightly less efficient than edge-triggered.
•epoll (edge-triggered) — When: Highest performance needed, team understands the model, already using non-blocking I/O correctly. Why not: Easy to introduce bugs.
•Abstraction library (libevent, libuv) — When: Cross-platform needed, rapid development, don't want to manage multiplexing details. Why not: Overhead, dependency.

Quick Selection Guide
Scenario	Recommendation
Cross-platform CLI tool	select() or poll()
Cross-platform library/framework	libevent or libuv
Linux server, < 1000 connections	poll() or epoll LT
Linux server, thousands of connections	epoll LT or ET
Linux server, extreme scale (10K+)	epoll ET with careful coding
macOS/BSD server	kqueue (analogous to epoll)
Windows server	IOCP
Mobile app	Platform async framework

Start Simple, Optimize Later

Summary: select/poll/epoll

We've explored Unix's three generations of I/O multiplexing in depth. Let's consolidate the key insights:

Key Takeaways

•select() is universal but limited — 1024 fd limit, O(n) scanning, must rebuild fd_sets. Use for portability with small fd counts.
•poll() removes limits but scales poorly — No fd_set limit, cleaner API, but still O(n) per call. Use for moderate scale or portability.
•epoll delivers O(1) event notification — Persistent kernel state, returns only ready events. Use for Linux high-scale servers.
•Edge-triggered requires careful coding — Must drain fds completely or events are lost. Use for maximum performance when you understand the model.
•Non-blocking I/O is essential — Always use non-blocking mode with multiplexing for robust code.
•Write buffering is necessary — Handle partial writes by buffering and monitoring EPOLLOUT.
•Choose based on requirements — Portability, scale, complexity all factor into the decision.

Module Complete:

The next modules in I/O Software cover buffering, caching, and spooling—techniques that optimize I/O performance at the application level.

Module Complete