A socket server that handles only one client at a time is nearly useless. Real-world servers must handle hundreds, thousands, or even millions of concurrent connections—all while maintaining responsiveness and efficiency.

The challenge of connection handling is fundamental to network programming. How do you accept new connections while serving existing ones? How do you prevent one slow client from blocking all others? How do you scale to handle more connections than you have CPU cores?

Over decades of network programming evolution, several approaches have emerged: multi-process servers, multi-threaded servers, event-driven I/O, and hybrid models. Each has tradeoffs between simplicity, scalability, and resource usage. Understanding these approaches is essential for building production-quality networked applications.

The C10K problem, articulated by Dan Kegel in 1999, asked: "How do you handle 10,000 simultaneous connections on a single server?" At the time, this was considered challenging. Today, we face the C10M problem—ten million connections.

The difficulty isn't CPU-bound work—a modern server has plenty of CPU cycles. The challenge is managing the I/O overhead of many connections:

Why is this hard?

Early servers used one process or thread per connection. At 10,000 connections, that meant 10,000 threads—10GB of stack memory alone. Modern solutions use event-driven I/O with much lower overhead.

Modern Scalability Hierarchy:

┌──────────────────────────────────────────────────────────────────┐
│  Connections   │   Model              │  Typical Tech          │
├────────────────┼──────────────────────┼────────────────────────│
│  < 100         │  Thread per client   │  Classic Apache        │
│  100 - 1,000   │  Thread pool         │  Tomcat, thread pools  │
│  1,000 - 10K   │  Event-driven        │  nginx, Node.js        │
│  10K - 100K    │  epoll + thread pool │  nginx workers, Go     │
│  100K - 1M     │  io_uring / DPDK     │  Custom high-perf      │
│  1M+           │  Kernel bypass       │  Specialized systems   │
└──────────────────────────────────────────────────────────────────┘

The simplest server handles one client at a time, completing that client's request before accepting the next. This is called an iterative server.

int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
bind(listen_fd, (struct sockaddr *)&addr, sizeof(addr));
listen(listen_fd, SOMAXCONN);

while (1) {
    // Accept next connection (blocks until client connects)
    int client_fd = accept(listen_fd, NULL, NULL);
    
    // Handle this client completely
    handle_client(client_fd);
    
    // Close and move to next client
    close(client_fd);
}

When Iterative Servers Work:

• Request/response is fast (< 1ms)
• No concurrent clients expected
• Simplicity is more important than performance
• Educational/prototype purposes

The traditional Unix approach: fork a child process for each connection. The parent accepts connections; each child handles one client.

int listen_fd = create_listening_socket();

while (1) {
    int client_fd = accept(listen_fd, NULL, NULL);
    
    pid_t pid = fork();
    
    if (pid == 0) {
        // Child process
        close(listen_fd);  // Child doesn't need listening socket
        handle_client(client_fd);
        close(client_fd);
        exit(0);  // Child exits when done
    } else if (pid > 0) {
        // Parent process
        close(client_fd);  // Parent doesn't need client socket
        // Continue accepting
    } else {
        // Fork failed
        perror("fork");
        close(client_fd);
    }
}

Handling Zombie Processes:

When a child exits, it becomes a "zombie" until the parent calls wait(). Without proper handling, zombies accumulate:

// Solution 1: SIGCHLD handler
void sigchld_handler(int sig) {
    while (waitpid(-1, NULL, WNOHANG) > 0);
}

signal(SIGCHLD, sigchld_handler);

// Solution 2: Double fork (child forks and exits, grandchild is adopted by init)
pid_t pid = fork();
if (pid == 0) {
    if (fork() > 0) exit(0);  // Original child exits
    // Grandchild continues, orphaned to init
    handle_client(client_fd);
    exit(0);
}
waitpid(pid, NULL, 0);  // Reap original child immediately

Threads are lighter weight than processes, sharing address space while having separate execution contexts. Many patterns exist for multi-threaded servers.

Pattern 1: Thread per Connection

void *client_handler(void *arg) {
    int client_fd = *(int *)arg;
    free(arg);
    
    handle_client(client_fd);
    close(client_fd);
    return NULL;
}

int main() {
    int listen_fd = create_listening_socket();
    
    while (1) {
        int *client_fd = malloc(sizeof(int));
        *client_fd = accept(listen_fd, NULL, NULL);
        
        pthread_t thread;
        pthread_create(&thread, NULL, client_handler, client_fd);
        pthread_detach(thread);  // Don't need to join
    }
}

Pattern 2: Thread Pool

Creating threads is expensive. A thread pool reuses threads:

#define POOL_SIZE 16
#define QUEUE_SIZE 1024

int connection_queue[QUEUE_SIZE];
int queue_head = 0, queue_tail = 0;
pthread_mutex_t queue_lock = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t queue_cond = PTHREAD_COND_INITIALIZER;

void *worker_thread(void *arg) {
    while (1) {
        pthread_mutex_lock(&queue_lock);
        while (queue_head == queue_tail) {  // Queue empty
            pthread_cond_wait(&queue_cond, &queue_lock);
        }
        int client_fd = connection_queue[queue_head];
        queue_head = (queue_head + 1) % QUEUE_SIZE;
        pthread_mutex_unlock(&queue_lock);
        
        handle_client(client_fd);
        close(client_fd);
    }
}

void enqueue_connection(int client_fd) {
    pthread_mutex_lock(&queue_lock);
    connection_queue[queue_tail] = client_fd;
    queue_tail = (queue_tail + 1) % QUEUE_SIZE;
    pthread_cond_signal(&queue_cond);
    pthread_mutex_unlock(&queue_lock);
}

I/O multiplexing allows a single thread to monitor multiple file descriptors, waiting for any of them to become ready for I/O. This is the foundation of event-driven programming. Three mechanisms are common: select(), poll(), and epoll (Linux).

select() — The Original

fd_set read_fds;
FD_ZERO(&read_fds);
FD_SET(listen_fd, &read_fds);
for (int i = 0; i < num_clients; i++) {
    FD_SET(client_fds[i], &read_fds);
}

struct timeval timeout = {5, 0};  // 5 seconds
int max_fd = find_max_fd(listen_fd, client_fds, num_clients);

int ready = select(max_fd + 1, &read_fds, NULL, NULL, &timeout);

if (FD_ISSET(listen_fd, &read_fds)) {
    // New connection ready
    int client = accept(listen_fd, NULL, NULL);
}

for (int i = 0; i < num_clients; i++) {
    if (FD_ISSET(client_fds[i], &read_fds)) {
        // Client has data to read
        handle_client_data(client_fds[i]);
    }
}

select() limitations:

• FD_SETSIZE limit (typically 1024)
• O(n) scan of all FDs on each call
• fd_set is modified, must rebuild each time

poll() — Improved Interface

struct pollfd fds[MAX_CLIENTS + 1];
int nfds = 0;

fds[nfds].fd = listen_fd;
fds[nfds].events = POLLIN;
nfds++;

for (int i = 0; i < num_clients; i++) {
    fds[nfds].fd = client_fds[i];
    fds[nfds].events = POLLIN;
    nfds++;
}

int ready = poll(fds, nfds, 5000);  // 5 second timeout

for (int i = 0; i < nfds; i++) {
    if (fds[i].revents & POLLIN) {
        if (fds[i].fd == listen_fd) {
            accept_new_client();
        } else {
            handle_client_data(fds[i].fd);
        }
    }
}

poll() improvements over select():

• No FD_SETSIZE limit
• Separate input/output events (events/revents)
• Doesn't require max_fd calculation

Still has O(n) scanning problem for both kernel and userspace.

epoll (Linux) — Scalable I/O

// Create epoll instance
int epfd = epoll_create1(0);

// Add listening socket
struct epoll_event ev;
ev.events = EPOLLIN;
ev.data.fd = listen_fd;
epoll_ctl(epfd, EPOLL_CTL_ADD, listen_fd, &ev);

// Event loop
struct epoll_event events[MAX_EVENTS];

while (1) {
    int ready = epoll_wait(epfd, events, MAX_EVENTS, -1);
    
    for (int i = 0; i < ready; i++) {
        if (events[i].data.fd == listen_fd) {
            int client = accept(listen_fd, NULL, NULL);
            set_nonblocking(client);
            
            ev.events = EPOLLIN | EPOLLET;  // Edge-triggered
            ev.data.fd = client;
            epoll_ctl(epfd, EPOLL_CTL_ADD, client, &ev);
        } else {
            handle_client_data(events[i].data.fd);
        }
    }
}

epoll advantages:

• O(1) for getting ready events
• Kernel maintains persistent interest set
• Edge-triggered mode for maximum efficiency
• Scales to hundreds of thousands of connections

The Reactor pattern is a design pattern for handling concurrent I/O events. It's the foundation of event-driven frameworks like libuv (Node.js), Twisted (Python), and Boost.Asio (C++).

Core Components:

1. Event Demultiplexer: Waits for events on multiple handles (epoll/kqueue/IOCP)
2. Event Handlers: Callbacks that process specific event types
3. Reactor: Dispatches events to appropriate handlers

// Simplified Reactor Implementation
typedef void (*event_handler)(int fd, void *data);

struct reactor {
    int epfd;
    event_handler handlers[MAX_FDS];
    void *handler_data[MAX_FDS];
};

void reactor_register(struct reactor *r, int fd, event_handler h, void *data) {
    struct epoll_event ev = {EPOLLIN, {.fd = fd}};
    epoll_ctl(r->epfd, EPOLL_CTL_ADD, fd, &ev);
    r->handlers[fd] = h;
    r->handler_data[fd] = data;
}

void reactor_run(struct reactor *r) {
    struct epoll_event events[64];
    
    while (1) {
        int n = epoll_wait(r->epfd, events, 64, -1);
        for (int i = 0; i < n; i++) {
            int fd = events[i].data.fd;
            r->handlers[fd](fd, r->handler_data[fd]);
        }
    }
}

The Proactor Pattern (Alternative):

While the Reactor waits for I/O readiness and then performs the operation, the Proactor initiates asynchronous operations and handles completion notifications:

Reactor:                          Proactor:
1. Wait for FD to be readable     1. Initiate async read
2. Call handler                   2. Kernel performs read
3. Handler does read()            3. Completion notification
4. Handler processes data         4. Handler processes data

Windows IOCP (I/O Completion Ports) is a Proactor. Linux io_uring provides Proactor-style semantics as well.

Multi-threaded Reactor:

For multi-core utilization, combine the reactor with threads:

• Single reactor + thread pool: Reactor dispatches to worker threads
• Reactor per thread: Each thread has its own epoll (with SO_REUSEPORT)
• Main-sub reactor: Main reactor accepts, sub-reactors handle clients

This is how nginx achieves its famous performance: multiple worker processes, each running an event loop with epoll.

In non-blocking I/O, operations may complete partially. A send() might accept only some of your data; a recv() might return only part of a message. Proper handling of partial operations is essential for correct non-blocking code.

Partial Sends:

struct send_buffer {
    char *data;
    size_t len;
    size_t sent;
};

int handle_writable(int fd, struct send_buffer *buf) {
    while (buf->sent < buf->len) {
        ssize_t n = send(fd, buf->data + buf->sent, 
                         buf->len - buf->sent, MSG_NOSIGNAL);
        if (n < 0) {
            if (errno == EAGAIN || errno == EWOULDBLOCK) {
                // Socket buffer full, try again when writable
                return WANT_WRITE;
            }
            return ERROR;
        }
        buf->sent += n;
    }
    return COMPLETE;
}

Partial Receives with Message Framing:

struct recv_buffer {
    char *data;
    size_t capacity;
    size_t received;
    size_t expected;  // 0 = reading header, >0 = reading body
};

int handle_readable(int fd, struct recv_buffer *buf) {
    while (1) {
        ssize_t space = buf->capacity - buf->received;
        ssize_t n = recv(fd, buf->data + buf->received, space, 0);
        
        if (n < 0) {
            if (errno == EAGAIN || errno == EWOULDBLOCK) {
                return WANT_READ;  // No more data available
            }
            return ERROR;
        }
        if (n == 0) {
            return CLOSED;  // Peer closed connection
        }
        
        buf->received += n;
        
        // Length-prefix framing: first 4 bytes are message length
        if (buf->expected == 0 && buf->received >= 4) {
            buf->expected = ntohl(*(uint32_t *)buf->data);
        }
        
        if (buf->expected > 0 && buf->received >= 4 + buf->expected) {
            return MESSAGE_COMPLETE;
        }
    }
}

Connection pooling reuses existing connections rather than creating new ones for each request. This amortizes the cost of TCP connection establishment and is essential for high-performance clients.

Why Pool Connections?

For a connection to a server 50ms away, establishing a new TLS connection takes ~200ms. Reusing a pooled connection: essentially free.

Basic Connection Pool:

struct connection_pool {
    pthread_mutex_t lock;
    int connections[POOL_SIZE];
    int available[POOL_SIZE];  // 1 = available, 0 = in use
    int size;
};

int pool_acquire(struct connection_pool *pool) {
    pthread_mutex_lock(&pool->lock);
    
    for (int i = 0; i < pool->size; i++) {
        if (pool->available[i]) {
            pool->available[i] = 0;
            pthread_mutex_unlock(&pool->lock);
            
            // Verify connection is still good
            if (is_connection_alive(pool->connections[i])) {
                return pool->connections[i];
            }
            // Connection dead, create new one
            pool->connections[i] = create_connection();
            return pool->connections[i];
        }
    }
    
    pthread_mutex_unlock(&pool->lock);
    
    // Pool exhausted—either wait or create temporary connection
    return -1;
}

void pool_release(struct connection_pool *pool, int conn) {
    pthread_mutex_lock(&pool->lock);
    for (int i = 0; i < pool->size; i++) {
        if (pool->connections[i] == conn) {
            pool->available[i] = 1;
            break;
        }
    }
    pthread_mutex_unlock(&pool->lock);
}

Effective connection handling is the difference between a toy server and a production system. Let's consolidate the key strategies and their applicability:

What's Next:

With connection handling mastered, we'll explore application development—putting everything together to build complete networked applications with proper structure, error handling, logging, and production considerations.