System Design (HLD)Latency Optimization

Latency Optimization: Eliminating Performance Bottlenecks

LevelAdvanced

Duration90 mins

TopicLatency Optimization

5 / 5

Connection Pooling

The Cost of 'Hello'

Every time your application connects to a database, cache, or external service, it pays a connection establishment tax. This tax—TCP handshake, TLS negotiation, authentication, protocol initialization—can add 10-500ms to what should be a 1ms operation.

Consider the anatomy of a new database connection:

Step	Time	Cumulative
DNS resolution	1-50ms	50ms
TCP handshake (1 RTT)	0.5-50ms	100ms
TLS handshake (1-2 RTT)	1-100ms	200ms
Database authentication	5-50ms	250ms
Connection initialization	1-10ms	260ms
Your actual query	1-10ms	270ms

On a cold connection, a 5ms query takes 270ms. On a pooled connection, it takes 5ms. Connection pooling is a 50x latency improvement for short operations.

What You Will Master

By completing this page, you will understand connection pool mechanics, learn to size pools correctly, implement pooling for databases and HTTP clients, and troubleshoot common pooling issues that cause latency spikes.

How Connection Pooling Works

A connection pool maintains a set of pre-established connections that application threads can borrow and return. Instead of creating and destroying connections per request, connections are reused across many requests.

Pool Lifecycle:

┌─────────────────────────────────────────────────────────────┐
│                    CONNECTION POOL                          │
├─────────────────────────────────────────────────────────────┤
│  ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐                   │
│  │Conn1│ │Conn2│ │Conn3│ │Conn4│ │Conn5│  ← Idle            │
│  └─────┘ └─────┘ └─────┘ └─────┘ └─────┘                   │
│  ┌─────┐ ┌─────┐ ┌─────┐                                    │
│  │Conn6│ │Conn7│ │Conn8│  ← In Use (borrowed)               │
│  └─────┘ └─────┘ └─────┘                                    │
└─────────────────────────────────────────────────────────────┘

Request flow:
1. Thread requests connection from pool
2. Pool returns an idle connection (or creates new if under max)
3. Thread executes operations on connection
4. Thread returns connection to pool
5. Connection becomes available for next request

Key Pool Parameters:

Connection Pool Configuration Parameters
Parameter	Description	Typical Value
Minimum Pool Size	Connections kept even when idle	5-10
Maximum Pool Size	Hard limit on total connections	10-100
Connection Timeout	Max wait for available connection	10-30 seconds
Idle Timeout	Close idle connections after this time	5-30 minutes
Max Lifetime	Force close after this time (even if active)	30-60 minutes
Validation Query	Query to verify connection is alive	SELECT 1
Validation Interval	How often to check idle connections	30-60 seconds

Connection States:

Idle: Connection is established but not in use. Available for borrowing.
In Use: Connection is borrowed by a thread, executing operations.
Validating: Pool is verifying connection is still valid before lending.
Creating: Pool is establishing a new connection to grow the pool.
Evicting: Connection exceeded max lifetime or failed validation, being closed.

Pool Behavior Under Load:

Low load: Most requests get connections immediately from idle pool
Medium load: Pool may need to create new connections up to max size
High load: Requests queue waiting for connections to return
Overload: Requests timeout waiting for connections (pool exhaustion)

Connection Leaks

The most common pool problem is leaked connections—code that borrows but never returns. Over time, all connections are 'in use' but actually abandoned. Diagnosis: pool shows max connections in use, but database shows them idle. Prevention: always use try-finally or RAII patterns to ensure return. Most pools support leak detection with warnings after a connection is held too long.

Pool Sizing: The Goldilocks Problem

Pool sizing is critical—too small causes request queuing and latency spikes; too large overwhelms the backend and wastes resources.

Too Small:

Requests wait for available connections
Latency spikes under load
Throughput limited by connection availability

Too Large:

Database overwhelmed by concurrent queries
Context switching overhead increases
Memory wasted on idle connections
May hit database connection limits

The Sizing Formula:

For databases, a good starting point:

Optimal Pool Size = (Number of CPU cores × 2) + Spindle count

For SSDs, spindle count is effectively 0-1. For a 4-core server:

HDD: (4 × 2) + 1 = 9 connections
SSD: (4 × 2) + 0 = 8 connections

This formula accounts for the fact that while one query is waiting for disk I/O, another can use the CPU.

Sizing for Application Pools:

The database pool size formula applies to a single application instance. When you have multiple application servers:

Total DB connections = App instances × Pool size per instance

20 app servers × 10 connections = 200 total database connections

This often exceeds what the database can handle efficiently. Solutions:

Connection proxy (PgBouncer, ProxySQL): Multiplexes many app connections over fewer database connections
Smaller per-instance pools: Reduce per-app pool size as you scale horizontally
Read replicas: Spread read load across replicas, reducing connections to primary

Signs Pool Is Too Small

•Connection wait time > 0ms regularly
•Connection timeout errors
•Latency spikes during peak hours
•Pool utilization consistently at 100%
•Request queue growing

Signs Pool Is Too Large

•Database CPU/connections maxed out
•Many idle connections (wasted resources)
•Database connection limit errors
•Query latency increases under load
•Connection creation failures

pool-monitoring.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
// Monitor pool metrics for optimal sizing
import { Pool } from 'pg';
 
const pool = new Pool({
    host: 'localhost',
    database: 'app',
    max: 20,                // Maximum connections
    min: 5,                 // Minimum connections
    idleTimeoutMillis: 30000,
    connectionTimeoutMillis: 10000,
});
 
// Track pool metrics
setInterval(() => {
    const metrics = {
        total: pool.totalCount,      // Total connections (active + idle)
        idle: pool.idleCount,        // Available connections
        waiting: pool.waitingCount,  // Requests waiting for connection
    };
    
    console.log(`Pool: total=${metrics.total}, idle=${metrics.idle}, waiting=${metrics.waiting}`);
    
    // Alert on pool exhaustion
    if (metrics.waiting > 0) {
        console.warn(`⚠️ ${metrics.waiting} requests waiting for connections!`);
    }
    
    // Alert on low utilization (pool too large)
    const utilization = (metrics.total - metrics.idle) / pool.options.max!;
    if (utilization < 0.1 && metrics.total > pool.options.min!) {
        console.info(`Pool underutilized: ${(utilization * 100).toFixed(1)}%`);
    }
    
    // Export to monitoring system
    gauge.set({ name: 'db_pool_total' }, metrics.total);
    gauge.set({ name: 'db_pool_idle' }, metrics.idle);
    gauge.set({ name: 'db_pool_waiting' }, metrics.waiting);
    
}, 10000); // Check every 10 seconds
 
// Measure connection acquisition time
async function queryWithMetrics<T>(
    queryFn: () => Promise<T>
): Promise<T> {
    const acquireStart = Date.now();
    
    const client = await pool.connect();
    
    const acquireTime = Date.now() - acquireStart;
    histogram.observe({ type: 'connection_acquire_ms' }, acquireTime);
    
    if (acquireTime > 10) {
        console.warn(`Slow connection acquire: ${acquireTime}ms`);
    }
    
    try {
        const queryStart = Date.now();
        const result = await queryFn();
        const queryTime = Date.now() - queryStart;
        histogram.observe({ type: 'query_ms' }, queryTime);
        return result;
    } finally {
        client.release();
    }
}

Start Small, Scale Up

Begin with a small pool (5-10 connections) and increase based on monitoring. It's easier to diagnose 'too few' problems than 'too many' problems. Monitor connection wait time—if it's consistently > 0, consider increasing pool size. If database is struggling, decrease pool size or add replicas.

Database Connection Pooling

Database connections are the most common pooling target. Each database platform has specific considerations.

PostgreSQL Pooling:

PostgreSQL spawns a new process for each connection (not a thread), making connection overhead particularly high. External poolers like PgBouncer are essential for high-scale applications.

postgres-pooling.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
// Node.js pg pool configuration
import { Pool, PoolConfig } from 'pg';
 
const poolConfig: PoolConfig = {
    host: process.env.DB_HOST,
    port: parseInt(process.env.DB_PORT || '5432'),
    database: process.env.DB_NAME,
    user: process.env.DB_USER,
    password: process.env.DB_PASSWORD,
    
    // Pool sizing
    min: 5,                     // Minimum idle connections
    max: 20,                    // Maximum connections
    
    // Timeouts
    idleTimeoutMillis: 30000,   // Close idle connections after 30s
    connectionTimeoutMillis: 10000, // Fail if can't get connection in 10s
    
    // Connection health
    allowExitOnIdle: false,     // Don't exit if pool is idle
    
    // SSL for production
    ssl: process.env.NODE_ENV === 'production' 
        ? { rejectUnauthorized: true }
        : false,
    
    // Application name for debugging
    application_name: 'my-app',
};
 
const pool = new Pool(poolConfig);
 
// Handle pool errors
pool.on('error', (err, client) => {
    console.error('Unexpected pool error:', err);
});
 
// Connection release handler
pool.on('release', (err, client) => {
    if (err) {
        console.error('Connection release error:', err);
    }
});
 
// Graceful shutdown
async function shutdown() {
    console.log('Closing database pool...');
    await pool.end();
    console.log('Pool closed');
}
 
process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);
 
// Usage patterns
 
// Single query (automatic acquire/release)
async function getUser(id: string) {
    const result = await pool.query('SELECT * FROM users WHERE id = $1', [id]);
    return result.rows[0];
}
 
// Transaction (manual acquire/release)
async function transferFunds(fromId: string, toId: string, amount: number) {
    const client = await pool.connect();
    
    try {
        await client.query('BEGIN');
        
        await client.query(
            'UPDATE accounts SET balance = balance - $1 WHERE id = $2',
            [amount, fromId]
        );
        
        await client.query(
            'UPDATE accounts SET balance = balance + $1 WHERE id = $2',
            [amount, toId]
        );
        
        await client.query('COMMIT');
    } catch (err) {
        await client.query('ROLLBACK');
        throw err;
    } finally {
        client.release(); // ALWAYS release in finally
    }
}

PgBouncer for High-Scale PostgreSQL:

PgBouncer is a connection pooler that sits between applications and PostgreSQL. It multiplexes thousands of application connections over a smaller number of database connections.

Pooling Modes:

Mode	Description	Use Case
Session	Client keeps one connection for entire session	Long-lived connections, transactions
Transaction	Connection returned after each transaction	Most web applications
Statement	Connection returned after each statement	High-volume, simple queries

Transaction mode is the default and works for most applications. Statement mode offers highest multiplexing but breaks transactions.

pgbouncer.ini
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[databases]
mydb = host=localhost dbname=mydb
 
[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
 
; Pool configuration
pool_mode = transaction         ; Use transaction pooling
default_pool_size = 20          ; Connections per user/db pair
max_client_conn = 1000          ; Max client connections
max_db_connections = 100        ; Max connections to database
 
; Timeouts
server_idle_timeout = 600       ; Close idle server connections after 10min
client_idle_timeout = 0         ; Don't close idle clients
server_connect_timeout = 15     ; Timeout connecting to server
server_login_retry = 15         ; Retry failed connections
 
; Performance
tcp_keepalive = 1
tcp_keepidle = 30
tcp_keepintvl = 10
tcp_keepcnt = 6
 
; Query timeouts
query_timeout = 0               ; No query timeout (app should handle)
query_wait_timeout = 120        ; Max time waiting for connection
 
; Logging
log_connections = 1
log_disconnections = 1
log_pooler_errors = 1

Prepared Statements with PgBouncer

In transaction mode, PgBouncer routes each transaction to potentially different database connections. This breaks named prepared statements, which are connection-specific. Solutions: use unnamed statements (protocol-level), enable 'server_reset_query' to deallocate, or use session mode for apps with heavy prepared statement use.

HTTP Client Connection Pooling

HTTP connections have the same establishment overhead as database connections—TCP + TLS handshake. HTTP/1.1 keep-alive and HTTP/2 multiplexing help, but require connection pooling to be effective.

HTTP/1.1 Keep-Alive:

Reuses connections for sequential requests to the same host. But limited to one request at a time per connection—need multiple connections for parallelism.

HTTP/2 Multiplexing:

Single connection handles multiple concurrent requests. Reduces need for multiple connections. Most APIs and services now support HTTP/2.

http-pooling.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
// Node.js HTTP agent pooling
import http from 'http';
import https from 'https';
 
// HTTP/1.1 agent with connection pooling
const httpsAgent = new https.Agent({
    keepAlive: true,           // Enable connection reuse
    keepAliveMsecs: 1000,      // TCP keepalive probe interval
    maxSockets: 100,           // Max sockets per host
    maxFreeSockets: 50,        // Max idle sockets to keep
    timeout: 60000,            // Socket timeout
    scheduling: 'fifo',        // First-in-first-out scheduling
});
 
// Use agent with fetch (Node.js 18+)
const response = await fetch('https://api.example.com/data', {
    agent: httpsAgent,
});
 
// Or with axios
import axios from 'axios';
 
const apiClient = axios.create({
    httpsAgent,
    timeout: 30000,
    baseURL: 'https://api.example.com',
});
 
// HTTP/2 client pooling
import http2 from 'http2';
 
class Http2Pool {
    private sessions: Map<string, ReturnType<typeof http2.connect>> = new Map();
    
    private getSession(origin: string) {
        let session = this.sessions.get(origin);
        
        if (!session || session.destroyed || session.closed) {
            session = http2.connect(origin, {
                // Connection settings
                settings: {
                    maxConcurrentStreams: 100,  // Max parallel requests
                },
                // Automatically ping to keep alive
                peerMaxConcurrentStreams: 100,
            });
            
            // Handle session errors
            session.on('error', (err) => {
                console.error(`HTTP/2 session error for ${origin}:`, err);
                this.sessions.delete(origin);
            });
            
            session.on('close', () => {
                this.sessions.delete(origin);
            });
            
            // Periodic ping to detect dead connections
            session.on('connect', () => {
                const pingInterval = setInterval(() => {
                    if (session.destroyed || session.closed) {
                        clearInterval(pingInterval);
                        return;
                    }
                    session.ping((err, duration) => {
                        if (err) {
                            console.warn(`HTTP/2 ping failed for ${origin}`);
                            session.destroy();
                        }
                    });
                }, 30000);
            });
            
            this.sessions.set(origin, session);
        }
        
        return session;
    }
    
    async request(url: string, options: {
        method?: string;
        headers?: Record<string, string>;
        body?: string | Buffer;
    } = {}): Promise<{ status: number; data: Buffer }> {
        const urlObj = new URL(url);
        const origin = urlObj.origin;
        const session = this.getSession(origin);
        
        return new Promise((resolve, reject) => {
            const req = session.request({
                ':path': urlObj.pathname + urlObj.search,
                ':method': options.method || 'GET',
                ...options.headers,
            });
            
            const chunks: Buffer[] = [];
            let status = 0;
            
            req.on('response', (headers) => {
                status = headers[':status'] as number;
            });
            
            req.on('data', (chunk) => chunks.push(chunk));
            
            req.on('end', () => {
                resolve({ status, data: Buffer.concat(chunks) });
            });
            
            req.on('error', reject);
            
            if (options.body) {
                req.write(options.body);
            }
            req.end();
        });
    }
    
    async close() {
        for (const [origin, session] of this.sessions) {
            session.close();
        }
        this.sessions.clear();
    }
}
 
// Usage
const http2Pool = new Http2Pool();
 
async function fetchData() {
    const { status, data } = await http2Pool.request(
        'https://api.example.com/data',
        { method: 'GET' }
    );
    console.log(`Status: ${status}, Body: ${data.toString()}`);
}

DNS Caching for HTTP Pools

HTTP connection pools cache connections by hostname. If DNS changes, cached connections may point to old IPs. For high-availability scenarios, implement DNS TTL-aware connection recycling, or use IP-based pools with separate DNS resolution. Most cloud load balancers handle this transparently.

Redis Connection Pooling

Redis is single-threaded, so connection pooling works differently than databases. The benefits are:

Avoid connection overhead — Same TCP/TLS benefits
Handle high concurrency — One connection can block; pool provides alternatives
Pipelining efficiency — Pool connections for parallel pipelines

However, a single Redis connection can handle thousands of operations per second through pipelining. You often need fewer connections than you'd think.

redis-pooling.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
// ioredis built-in connection management
import Redis from 'ioredis';
 
// Single connection (often sufficient)
const redis = new Redis({
    host: 'localhost',
    port: 6379,
    // Automatic reconnection
    retryStrategy: (times) => {
        const delay = Math.min(times * 50, 2000);
        return delay;
    },
    // Connection timeout
    connectTimeout: 10000,
    // Enable keepalive
    keepAlive: 30000,
    // Enable offline queue (buffer commands during reconnect)
    enableOfflineQueue: true,
    // Max retry delay
    maxRetriesPerRequest: 3,
});
 
// For high-concurrency, use Redis cluster or pool
import { Cluster } from 'ioredis';
 
const cluster = new Cluster([
    { host: 'redis-1', port: 6379 },
    { host: 'redis-2', port: 6379 },
    { host: 'redis-3', port: 6379 },
], {
    redisOptions: {
        password: process.env.REDIS_PASSWORD,
    },
    // Scale reads across replicas
    scaleReads: 'slave',
});
 
// Generic pool using generic-pool
import { createPool, Pool } from 'generic-pool';
 
const redisPool = createPool({
    create: async () => {
        const client = new Redis({
            host: 'localhost',
            port: 6379,
            enableOfflineQueue: false, // Fail fast for pool
        });
        
        // Wait for connection
        await new Promise<void>((resolve, reject) => {
            client.on('ready', resolve);
            client.on('error', reject);
        });
        
        return client;
    },
    destroy: async (client) => {
        await client.quit();
    },
    validate: async (client) => {
        try {
            await client.ping();
            return true;
        } catch {
            return false;
        }
    },
}, {
    min: 2,
    max: 10,
    acquireTimeoutMillis: 5000,
    idleTimeoutMillis: 30000,
    testOnBorrow: true,
});
 
// Usage
async function getCached(key: string): Promise<string | null> {
    const client = await redisPool.acquire();
    try {
        return await client.get(key);
    } finally {
        await redisPool.release(client);
    }
}
 
// Pipelining for batch operations (single connection, multiple commands)
async function batchGet(keys: string[]): Promise<(string | null)[]> {
    const pipeline = redis.pipeline();
    
    for (const key of keys) {
        pipeline.get(key);
    }
    
    const results = await pipeline.exec();
    return results?.map(([err, value]) => value as string | null) || [];
}
 
// Pipelining is often better than pooling for batch operations
// 100 GETs pipelined = 1 round trip
// 100 GETs on pool = up to 10 round trips (depending on pool size)

Pipelining vs Pooling

For Redis, pipelining is often more effective than pooling. Pipelining batches many commands into a single network round-trip. A single pipelined connection can outperform 10 unpipelined pooled connections. Use pools when: you need blocking commands (BLPOP), have truly parallel workloads, or use transactions that can't be pipelined.

Connection Pool Troubleshooting

Connection pools can be sources of latency issues when misconfigured or under stress. Here are common problems and solutions.

Common Pool Problems and Solutions
Problem	Symptoms	Solution
Pool Exhaustion	Connection timeout errors, high wait time	Increase pool size, fix connection leaks, add connection proxy
Connection Leaks	Pool at max, database shows idle connections	Always release in finally, enable leak detection logging
Stale Connections	Random errors after idle periods	Reduce idle timeout, enable validation on borrow
Connection Storm	Spike of connections on startup/restart	Use min size, warm pool on startup, gradual startup
Network Issues	Intermittent connection failures	Add retries, connection validation, health checks
Max Lifetime Churn	Periodic latency spikes	Stagger max lifetime with jitter, overlap replacement

pool-diagnostics.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
// Connection leak detection
class LeakDetectingPool {
    private borrowedConnections: Map<string, {
        borrowedAt: number;
        stack: string;
    }> = new Map();
    
    private leakThresholdMs = 30000; // Warn after 30s
    
    async acquire(): Promise<Connection> {
        const conn = await this.pool.acquire();
        const id = conn.id;
        
        this.borrowedConnections.set(id, {
            borrowedAt: Date.now(),
            stack: new Error().stack || '',
        });
        
        return conn;
    }
    
    release(conn: Connection): void {
        this.borrowedConnections.delete(conn.id);
        this.pool.release(conn);
    }
    
    // Run periodically
    checkForLeaks(): void {
        const now = Date.now();
        
        for (const [id, info] of this.borrowedConnections) {
            const heldFor = now - info.borrowedAt;
            
            if (heldFor > this.leakThresholdMs) {
                console.warn(
                    `Potential connection leak: ${id} held for ${heldFor}ms\n` +
                    `Borrowed at: ${info.stack}`
                );
            }
        }
    }
}
 
// Stale connection handling
const pool = new Pool({
    // Validate connections before use
    testOnBorrow: true,
    
    // Custom validation query
    validationQuery: 'SELECT 1',
    
    // Timeout for validation
    validationTimeout: 5000,
    
    // Evict idle connections periodically
    evictionRunIntervalMillis: 30000,
    
    // Close connections after 30 minutes
    softIdleTimeoutMillis: 1800000,
    
    // Close connections after 1 hour regardless
    maxLifetimeMillis: 3600000,
});
 
// Graceful connection recycling with jitter
function getMaxLifetimeWithJitter(baseLifetimeMs: number): number {
    // Add ±10% jitter to avoid thundering herd
    const jitter = baseLifetimeMs * 0.1;
    return baseLifetimeMs + (Math.random() - 0.5) * 2 * jitter;
}
 
// Pool warm-up on startup
async function warmPool(pool: Pool, targetSize: number): Promise<void> {
    console.log(`Warming pool to ${targetSize} connections...`);
    
    const connections: Connection[] = [];
    
    for (let i = 0; i < targetSize; i++) {
        try {
            const conn = await pool.acquire();
            connections.push(conn);
            
            // Small delay to avoid connection storm
            await new Promise(r => setTimeout(r, 100));
        } catch (err) {
            console.error('Failed to warm connection:', err);
        }
    }
    
    // Release all warmed connections
    for (const conn of connections) {
        pool.release(conn);
    }
    
    console.log(`Pool warmed with ${connections.length} connections`);
}
 
// Call on application startup
warmPool(pool, pool.options.min || 5);

The Thundering Herd on Restart

When an application restarts, all requests hit with empty pool, causing a spike of connection creation. This can overwhelm the database. Mitigate with: pool warm-up before accepting traffic, min pool size, gradual traffic ramp-up (load balancer draining), and connection proxies that absorb spikes.

Advanced Pooling Patterns

Beyond basic pooling, advanced patterns address specific challenges in production systems.

Separate Pools for Different Workloads:

Mix of quick queries and long-running analytics can cause head-of-line blocking. Solution: separate pools.

advanced-pooling.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
// Separate pools for different workloads
const transactionalPool = new Pool({
    ...baseConfig,
    max: 20,
    connectionTimeoutMillis: 5000,  // Fast timeout for OLTP
    idleTimeoutMillis: 60000,
});
 
const analyticsPool = new Pool({
    ...baseConfig,
    max: 5,                          // Fewer connections for heavy queries
    connectionTimeoutMillis: 30000,  // Longer timeout acceptable
    statement_timeout: '300000',     // 5 minute query timeout
});
 
// Read replica routing
interface PoolSet {
    primary: Pool;
    replicas: Pool[];
    currentReplica: number;
}
 
function createPoolSet(config: PoolConfig, replicaConfigs: PoolConfig[]): PoolSet {
    return {
        primary: new Pool(config),
        replicas: replicaConfigs.map(c => new Pool(c)),
        currentReplica: 0,
    };
}
 
function getReadPool(poolSet: PoolSet): Pool {
    // Round-robin across replicas
    if (poolSet.replicas.length === 0) {
        return poolSet.primary;
    }
    
    const pool = poolSet.replicas[poolSet.currentReplica];
    poolSet.currentReplica = (poolSet.currentReplica + 1) % poolSet.replicas.length;
    return pool;
}
 
function getWritePool(poolSet: PoolSet): Pool {
    return poolSet.primary;
}
 
const dbPools = createPoolSet(
    { host: 'primary.db.internal', ...baseConfig },
    [
        { host: 'replica-1.db.internal', ...baseConfig },
        { host: 'replica-2.db.internal', ...baseConfig },
    ]
);
 
// Usage
async function getUser(id: string) {
    const pool = getReadPool(dbPools);
    return pool.query('SELECT * FROM users WHERE id = $1', [id]);
}
 
async function updateUser(id: string, data: Partial<User>) {
    const pool = getWritePool(dbPools);
    return pool.query('UPDATE users SET name = $1 WHERE id = $2', [data.name, id]);
}
 
// Priority-based connection acquisition
class PriorityPool {
    private high: Pool;
    private low: Pool;
    
    constructor(config: PoolConfig) {
        // 70% for high priority, 30% for low
        const highMax = Math.ceil(config.max! * 0.7);
        const lowMax = Math.floor(config.max! * 0.3);
        
        this.high = new Pool({ ...config, max: highMax });
        this.low = new Pool({ ...config, max: lowMax });
    }
    
    acquire(priority: 'high' | 'low' = 'low'): Promise<PoolClient> {
        return priority === 'high' 
            ? this.high.connect()
            : this.low.connect();
    }
}
 
// Usage: critical path gets high priority
async function processPayment(data: PaymentData) {
    const client = await priorityPool.acquire('high');
    // ...
}
 
async function generateReport(params: ReportParams) {
    const client = await priorityPool.acquire('low');
    // ...
}

Monitor All Pools

When using multiple pools, monitor each separately. One exhausted pool (e.g., analytics) shouldn't alert as a production issue, but primary pool exhaustion is critical. Label pool metrics with pool name for clear observability.

Summary: Connection Pooling

Connection pooling eliminates one of the most common hidden latency costs—connection establishment. By reusing pre-established connections, operations that would take 200-500ms can complete in 5-10ms.

Key Takeaways

•Understand connection cost — TCP handshake + TLS + auth + init can add 200ms+ to each new connection. Pooling amortizes this across many operations.
•Size pools appropriately — Start with (cores × 2) + spindles. Too small = request queuing; too large = backend overload.
•Use connection proxies at scale — PgBouncer, ProxySQL multiplex many app connections over fewer database connections.
•Pool HTTP clients too — Same TCP/TLS overhead applies. Use keep-alive agents, HTTP/2 for multiplexing.
•Redis needs less pooling — Single-threaded Redis benefits more from pipelining. Pool when blocking commands or true parallelism needed.
•Monitor pool health — Track wait time, active/idle counts, acquisition time. Alert on exhaustion before users feel it.
•Prevent common issues — Leak detection, connection validation, warm-up on startup, jittered max lifetime.
•Use separate pools for different workloads — OLTP vs analytics, reads vs writes, critical vs background.

Module Complete:

You've now completed the Latency Optimization module. You've learned to reduce latency through:

Network optimization — Protocol tuning, geographic distribution, payload compression
Database query optimization — Indexing, query rewriting, execution plan analysis
Caching — Multi-layer caching, access patterns, invalidation strategies
Async processing — Queues, event-driven architecture, saga patterns
Connection pooling — Pool mechanics, sizing, advanced patterns

These techniques, applied systematically, can reduce latency by orders of magnitude—transforming slow, frustrating systems into responsive, delightful user experiences.

Module Complete

Congratulations! You've mastered the comprehensive toolkit for latency optimization. From network protocol tuning to database optimization to caching strategies to async processing to connection pooling—you now have the knowledge to identify and eliminate latency bottlenecks throughout the stack.

5 / 5

Loading learning content...

System Design (HLD)Latency Optimization

Latency Optimization: Eliminating Performance Bottlenecks

LevelAdvanced

Duration90 mins

TopicLatency Optimization

5 / 5

Connection Pooling

The Cost of 'Hello'

Consider the anatomy of a new database connection:

Step	Time	Cumulative
DNS resolution	1-50ms	50ms
TCP handshake (1 RTT)	0.5-50ms	100ms
TLS handshake (1-2 RTT)	1-100ms	200ms
Database authentication	5-50ms	250ms
Connection initialization	1-10ms	260ms
Your actual query	1-10ms	270ms

On a cold connection, a 5ms query takes 270ms. On a pooled connection, it takes 5ms. Connection pooling is a 50x latency improvement for short operations.

What You Will Master

How Connection Pooling Works

Pool Lifecycle:

┌─────────────────────────────────────────────────────────────┐
│                    CONNECTION POOL                          │
├─────────────────────────────────────────────────────────────┤
│  ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐                   │
│  │Conn1│ │Conn2│ │Conn3│ │Conn4│ │Conn5│  ← Idle            │
│  └─────┘ └─────┘ └─────┘ └─────┘ └─────┘                   │
│  ┌─────┐ ┌─────┐ ┌─────┐                                    │
│  │Conn6│ │Conn7│ │Conn8│  ← In Use (borrowed)               │
│  └─────┘ └─────┘ └─────┘                                    │
└─────────────────────────────────────────────────────────────┘

Request flow:
1. Thread requests connection from pool
2. Pool returns an idle connection (or creates new if under max)
3. Thread executes operations on connection
4. Thread returns connection to pool
5. Connection becomes available for next request

Key Pool Parameters:

Connection Pool Configuration Parameters
Parameter	Description	Typical Value
Minimum Pool Size	Connections kept even when idle	5-10
Maximum Pool Size	Hard limit on total connections	10-100
Connection Timeout	Max wait for available connection	10-30 seconds
Idle Timeout	Close idle connections after this time	5-30 minutes
Max Lifetime	Force close after this time (even if active)	30-60 minutes
Validation Query	Query to verify connection is alive	SELECT 1
Validation Interval	How often to check idle connections	30-60 seconds

Connection States:

Idle: Connection is established but not in use. Available for borrowing.
In Use: Connection is borrowed by a thread, executing operations.
Validating: Pool is verifying connection is still valid before lending.
Creating: Pool is establishing a new connection to grow the pool.
Evicting: Connection exceeded max lifetime or failed validation, being closed.

Pool Behavior Under Load:

Low load: Most requests get connections immediately from idle pool
Medium load: Pool may need to create new connections up to max size
High load: Requests queue waiting for connections to return
Overload: Requests timeout waiting for connections (pool exhaustion)

Connection Leaks

Pool Sizing: The Goldilocks Problem

Pool sizing is critical—too small causes request queuing and latency spikes; too large overwhelms the backend and wastes resources.

Too Small:

Requests wait for available connections
Latency spikes under load
Throughput limited by connection availability

Too Large:

Database overwhelmed by concurrent queries
Context switching overhead increases
Memory wasted on idle connections
May hit database connection limits

The Sizing Formula:

For databases, a good starting point:

Optimal Pool Size = (Number of CPU cores × 2) + Spindle count

For SSDs, spindle count is effectively 0-1. For a 4-core server:

HDD: (4 × 2) + 1 = 9 connections
SSD: (4 × 2) + 0 = 8 connections

This formula accounts for the fact that while one query is waiting for disk I/O, another can use the CPU.

Sizing for Application Pools:

The database pool size formula applies to a single application instance. When you have multiple application servers:

Total DB connections = App instances × Pool size per instance

20 app servers × 10 connections = 200 total database connections

This often exceeds what the database can handle efficiently. Solutions:

Connection proxy (PgBouncer, ProxySQL): Multiplexes many app connections over fewer database connections
Smaller per-instance pools: Reduce per-app pool size as you scale horizontally
Read replicas: Spread read load across replicas, reducing connections to primary

Signs Pool Is Too Small

•Connection wait time > 0ms regularly
•Connection timeout errors
•Latency spikes during peak hours
•Pool utilization consistently at 100%
•Request queue growing

Signs Pool Is Too Large

•Database CPU/connections maxed out
•Many idle connections (wasted resources)
•Database connection limit errors
•Query latency increases under load
•Connection creation failures

pool-monitoring.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
// Monitor pool metrics for optimal sizing
import { Pool } from 'pg';
 
const pool = new Pool({
    host: 'localhost',
    database: 'app',
    max: 20,                // Maximum connections
    min: 5,                 // Minimum connections
    idleTimeoutMillis: 30000,
    connectionTimeoutMillis: 10000,
});
 
// Track pool metrics
setInterval(() => {
    const metrics = {
        total: pool.totalCount,      // Total connections (active + idle)
        idle: pool.idleCount,        // Available connections
        waiting: pool.waitingCount,  // Requests waiting for connection
    };
    
    console.log(`Pool: total=${metrics.total}, idle=${metrics.idle}, waiting=${metrics.waiting}`);
    
    // Alert on pool exhaustion
    if (metrics.waiting > 0) {
        console.warn(`⚠️ ${metrics.waiting} requests waiting for connections!`);
    }
    
    // Alert on low utilization (pool too large)
    const utilization = (metrics.total - metrics.idle) / pool.options.max!;
    if (utilization < 0.1 && metrics.total > pool.options.min!) {
        console.info(`Pool underutilized: ${(utilization * 100).toFixed(1)}%`);
    }
    
    // Export to monitoring system
    gauge.set({ name: 'db_pool_total' }, metrics.total);
    gauge.set({ name: 'db_pool_idle' }, metrics.idle);
    gauge.set({ name: 'db_pool_waiting' }, metrics.waiting);
    
}, 10000); // Check every 10 seconds
 
// Measure connection acquisition time
async function queryWithMetrics<T>(
    queryFn: () => Promise<T>
): Promise<T> {
    const acquireStart = Date.now();
    
    const client = await pool.connect();
    
    const acquireTime = Date.now() - acquireStart;
    histogram.observe({ type: 'connection_acquire_ms' }, acquireTime);
    
    if (acquireTime > 10) {
        console.warn(`Slow connection acquire: ${acquireTime}ms`);
    }
    
    try {
        const queryStart = Date.now();
        const result = await queryFn();
        const queryTime = Date.now() - queryStart;
        histogram.observe({ type: 'query_ms' }, queryTime);
        return result;
    } finally {
        client.release();
    }
}

Start Small, Scale Up

Database Connection Pooling

Database connections are the most common pooling target. Each database platform has specific considerations.

PostgreSQL Pooling:

PostgreSQL spawns a new process for each connection (not a thread), making connection overhead particularly high. External poolers like PgBouncer are essential for high-scale applications.

postgres-pooling.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
// Node.js pg pool configuration
import { Pool, PoolConfig } from 'pg';
 
const poolConfig: PoolConfig = {
    host: process.env.DB_HOST,
    port: parseInt(process.env.DB_PORT || '5432'),
    database: process.env.DB_NAME,
    user: process.env.DB_USER,
    password: process.env.DB_PASSWORD,
    
    // Pool sizing
    min: 5,                     // Minimum idle connections
    max: 20,                    // Maximum connections
    
    // Timeouts
    idleTimeoutMillis: 30000,   // Close idle connections after 30s
    connectionTimeoutMillis: 10000, // Fail if can't get connection in 10s
    
    // Connection health
    allowExitOnIdle: false,     // Don't exit if pool is idle
    
    // SSL for production
    ssl: process.env.NODE_ENV === 'production' 
        ? { rejectUnauthorized: true }
        : false,
    
    // Application name for debugging
    application_name: 'my-app',
};
 
const pool = new Pool(poolConfig);
 
// Handle pool errors
pool.on('error', (err, client) => {
    console.error('Unexpected pool error:', err);
});
 
// Connection release handler
pool.on('release', (err, client) => {
    if (err) {
        console.error('Connection release error:', err);
    }
});
 
// Graceful shutdown
async function shutdown() {
    console.log('Closing database pool...');
    await pool.end();
    console.log('Pool closed');
}
 
process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);
 
// Usage patterns
 
// Single query (automatic acquire/release)
async function getUser(id: string) {
    const result = await pool.query('SELECT * FROM users WHERE id = $1', [id]);
    return result.rows[0];
}
 
// Transaction (manual acquire/release)
async function transferFunds(fromId: string, toId: string, amount: number) {
    const client = await pool.connect();
    
    try {
        await client.query('BEGIN');
        
        await client.query(
            'UPDATE accounts SET balance = balance - $1 WHERE id = $2',
            [amount, fromId]
        );
        
        await client.query(
            'UPDATE accounts SET balance = balance + $1 WHERE id = $2',
            [amount, toId]
        );
        
        await client.query('COMMIT');
    } catch (err) {
        await client.query('ROLLBACK');
        throw err;
    } finally {
        client.release(); // ALWAYS release in finally
    }
}

PgBouncer for High-Scale PostgreSQL:

PgBouncer is a connection pooler that sits between applications and PostgreSQL. It multiplexes thousands of application connections over a smaller number of database connections.

Pooling Modes:

Mode	Description	Use Case
Session	Client keeps one connection for entire session	Long-lived connections, transactions
Transaction	Connection returned after each transaction	Most web applications
Statement	Connection returned after each statement	High-volume, simple queries

Transaction mode is the default and works for most applications. Statement mode offers highest multiplexing but breaks transactions.

pgbouncer.ini
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[databases]
mydb = host=localhost dbname=mydb
 
[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
 
; Pool configuration
pool_mode = transaction         ; Use transaction pooling
default_pool_size = 20          ; Connections per user/db pair
max_client_conn = 1000          ; Max client connections
max_db_connections = 100        ; Max connections to database
 
; Timeouts
server_idle_timeout = 600       ; Close idle server connections after 10min
client_idle_timeout = 0         ; Don't close idle clients
server_connect_timeout = 15     ; Timeout connecting to server
server_login_retry = 15         ; Retry failed connections
 
; Performance
tcp_keepalive = 1
tcp_keepidle = 30
tcp_keepintvl = 10
tcp_keepcnt = 6
 
; Query timeouts
query_timeout = 0               ; No query timeout (app should handle)
query_wait_timeout = 120        ; Max time waiting for connection
 
; Logging
log_connections = 1
log_disconnections = 1
log_pooler_errors = 1

Prepared Statements with PgBouncer

HTTP Client Connection Pooling

HTTP connections have the same establishment overhead as database connections—TCP + TLS handshake. HTTP/1.1 keep-alive and HTTP/2 multiplexing help, but require connection pooling to be effective.

HTTP/1.1 Keep-Alive:

Reuses connections for sequential requests to the same host. But limited to one request at a time per connection—need multiple connections for parallelism.

HTTP/2 Multiplexing:

Single connection handles multiple concurrent requests. Reduces need for multiple connections. Most APIs and services now support HTTP/2.

http-pooling.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
// Node.js HTTP agent pooling
import http from 'http';
import https from 'https';
 
// HTTP/1.1 agent with connection pooling
const httpsAgent = new https.Agent({
    keepAlive: true,           // Enable connection reuse
    keepAliveMsecs: 1000,      // TCP keepalive probe interval
    maxSockets: 100,           // Max sockets per host
    maxFreeSockets: 50,        // Max idle sockets to keep
    timeout: 60000,            // Socket timeout
    scheduling: 'fifo',        // First-in-first-out scheduling
});
 
// Use agent with fetch (Node.js 18+)
const response = await fetch('https://api.example.com/data', {
    agent: httpsAgent,
});
 
// Or with axios
import axios from 'axios';
 
const apiClient = axios.create({
    httpsAgent,
    timeout: 30000,
    baseURL: 'https://api.example.com',
});
 
// HTTP/2 client pooling
import http2 from 'http2';
 
class Http2Pool {
    private sessions: Map<string, ReturnType<typeof http2.connect>> = new Map();
    
    private getSession(origin: string) {
        let session = this.sessions.get(origin);
        
        if (!session || session.destroyed || session.closed) {
            session = http2.connect(origin, {
                // Connection settings
                settings: {
                    maxConcurrentStreams: 100,  // Max parallel requests
                },
                // Automatically ping to keep alive
                peerMaxConcurrentStreams: 100,
            });
            
            // Handle session errors
            session.on('error', (err) => {
                console.error(`HTTP/2 session error for ${origin}:`, err);
                this.sessions.delete(origin);
            });
            
            session.on('close', () => {
                this.sessions.delete(origin);
            });
            
            // Periodic ping to detect dead connections
            session.on('connect', () => {
                const pingInterval = setInterval(() => {
                    if (session.destroyed || session.closed) {
                        clearInterval(pingInterval);
                        return;
                    }
                    session.ping((err, duration) => {
                        if (err) {
                            console.warn(`HTTP/2 ping failed for ${origin}`);
                            session.destroy();
                        }
                    });
                }, 30000);
            });
            
            this.sessions.set(origin, session);
        }
        
        return session;
    }
    
    async request(url: string, options: {
        method?: string;
        headers?: Record<string, string>;
        body?: string | Buffer;
    } = {}): Promise<{ status: number; data: Buffer }> {
        const urlObj = new URL(url);
        const origin = urlObj.origin;
        const session = this.getSession(origin);
        
        return new Promise((resolve, reject) => {
            const req = session.request({
                ':path': urlObj.pathname + urlObj.search,
                ':method': options.method || 'GET',
                ...options.headers,
            });
            
            const chunks: Buffer[] = [];
            let status = 0;
            
            req.on('response', (headers) => {
                status = headers[':status'] as number;
            });
            
            req.on('data', (chunk) => chunks.push(chunk));
            
            req.on('end', () => {
                resolve({ status, data: Buffer.concat(chunks) });
            });
            
            req.on('error', reject);
            
            if (options.body) {
                req.write(options.body);
            }
            req.end();
        });
    }
    
    async close() {
        for (const [origin, session] of this.sessions) {
            session.close();
        }
        this.sessions.clear();
    }
}
 
// Usage
const http2Pool = new Http2Pool();
 
async function fetchData() {
    const { status, data } = await http2Pool.request(
        'https://api.example.com/data',
        { method: 'GET' }
    );
    console.log(`Status: ${status}, Body: ${data.toString()}`);
}

DNS Caching for HTTP Pools

Redis Connection Pooling

Redis is single-threaded, so connection pooling works differently than databases. The benefits are:

Avoid connection overhead — Same TCP/TLS benefits
Handle high concurrency — One connection can block; pool provides alternatives
Pipelining efficiency — Pool connections for parallel pipelines

However, a single Redis connection can handle thousands of operations per second through pipelining. You often need fewer connections than you'd think.

redis-pooling.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
// ioredis built-in connection management
import Redis from 'ioredis';
 
// Single connection (often sufficient)
const redis = new Redis({
    host: 'localhost',
    port: 6379,
    // Automatic reconnection
    retryStrategy: (times) => {
        const delay = Math.min(times * 50, 2000);
        return delay;
    },
    // Connection timeout
    connectTimeout: 10000,
    // Enable keepalive
    keepAlive: 30000,
    // Enable offline queue (buffer commands during reconnect)
    enableOfflineQueue: true,
    // Max retry delay
    maxRetriesPerRequest: 3,
});
 
// For high-concurrency, use Redis cluster or pool
import { Cluster } from 'ioredis';
 
const cluster = new Cluster([
    { host: 'redis-1', port: 6379 },
    { host: 'redis-2', port: 6379 },
    { host: 'redis-3', port: 6379 },
], {
    redisOptions: {
        password: process.env.REDIS_PASSWORD,
    },
    // Scale reads across replicas
    scaleReads: 'slave',
});
 
// Generic pool using generic-pool
import { createPool, Pool } from 'generic-pool';
 
const redisPool = createPool({
    create: async () => {
        const client = new Redis({
            host: 'localhost',
            port: 6379,
            enableOfflineQueue: false, // Fail fast for pool
        });
        
        // Wait for connection
        await new Promise<void>((resolve, reject) => {
            client.on('ready', resolve);
            client.on('error', reject);
        });
        
        return client;
    },
    destroy: async (client) => {
        await client.quit();
    },
    validate: async (client) => {
        try {
            await client.ping();
            return true;
        } catch {
            return false;
        }
    },
}, {
    min: 2,
    max: 10,
    acquireTimeoutMillis: 5000,
    idleTimeoutMillis: 30000,
    testOnBorrow: true,
});
 
// Usage
async function getCached(key: string): Promise<string | null> {
    const client = await redisPool.acquire();
    try {
        return await client.get(key);
    } finally {
        await redisPool.release(client);
    }
}
 
// Pipelining for batch operations (single connection, multiple commands)
async function batchGet(keys: string[]): Promise<(string | null)[]> {
    const pipeline = redis.pipeline();
    
    for (const key of keys) {
        pipeline.get(key);
    }
    
    const results = await pipeline.exec();
    return results?.map(([err, value]) => value as string | null) || [];
}
 
// Pipelining is often better than pooling for batch operations
// 100 GETs pipelined = 1 round trip
// 100 GETs on pool = up to 10 round trips (depending on pool size)

Pipelining vs Pooling

Connection Pool Troubleshooting

Connection pools can be sources of latency issues when misconfigured or under stress. Here are common problems and solutions.

Common Pool Problems and Solutions
Problem	Symptoms	Solution
Pool Exhaustion	Connection timeout errors, high wait time	Increase pool size, fix connection leaks, add connection proxy
Connection Leaks	Pool at max, database shows idle connections	Always release in finally, enable leak detection logging
Stale Connections	Random errors after idle periods	Reduce idle timeout, enable validation on borrow
Connection Storm	Spike of connections on startup/restart	Use min size, warm pool on startup, gradual startup
Network Issues	Intermittent connection failures	Add retries, connection validation, health checks
Max Lifetime Churn	Periodic latency spikes	Stagger max lifetime with jitter, overlap replacement

pool-diagnostics.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
// Connection leak detection
class LeakDetectingPool {
    private borrowedConnections: Map<string, {
        borrowedAt: number;
        stack: string;
    }> = new Map();
    
    private leakThresholdMs = 30000; // Warn after 30s
    
    async acquire(): Promise<Connection> {
        const conn = await this.pool.acquire();
        const id = conn.id;
        
        this.borrowedConnections.set(id, {
            borrowedAt: Date.now(),
            stack: new Error().stack || '',
        });
        
        return conn;
    }
    
    release(conn: Connection): void {
        this.borrowedConnections.delete(conn.id);
        this.pool.release(conn);
    }
    
    // Run periodically
    checkForLeaks(): void {
        const now = Date.now();
        
        for (const [id, info] of this.borrowedConnections) {
            const heldFor = now - info.borrowedAt;
            
            if (heldFor > this.leakThresholdMs) {
                console.warn(
                    `Potential connection leak: ${id} held for ${heldFor}ms\n` +
                    `Borrowed at: ${info.stack}`
                );
            }
        }
    }
}
 
// Stale connection handling
const pool = new Pool({
    // Validate connections before use
    testOnBorrow: true,
    
    // Custom validation query
    validationQuery: 'SELECT 1',
    
    // Timeout for validation
    validationTimeout: 5000,
    
    // Evict idle connections periodically
    evictionRunIntervalMillis: 30000,
    
    // Close connections after 30 minutes
    softIdleTimeoutMillis: 1800000,
    
    // Close connections after 1 hour regardless
    maxLifetimeMillis: 3600000,
});
 
// Graceful connection recycling with jitter
function getMaxLifetimeWithJitter(baseLifetimeMs: number): number {
    // Add ±10% jitter to avoid thundering herd
    const jitter = baseLifetimeMs * 0.1;
    return baseLifetimeMs + (Math.random() - 0.5) * 2 * jitter;
}
 
// Pool warm-up on startup
async function warmPool(pool: Pool, targetSize: number): Promise<void> {
    console.log(`Warming pool to ${targetSize} connections...`);
    
    const connections: Connection[] = [];
    
    for (let i = 0; i < targetSize; i++) {
        try {
            const conn = await pool.acquire();
            connections.push(conn);
            
            // Small delay to avoid connection storm
            await new Promise(r => setTimeout(r, 100));
        } catch (err) {
            console.error('Failed to warm connection:', err);
        }
    }
    
    // Release all warmed connections
    for (const conn of connections) {
        pool.release(conn);
    }
    
    console.log(`Pool warmed with ${connections.length} connections`);
}
 
// Call on application startup
warmPool(pool, pool.options.min || 5);

The Thundering Herd on Restart

Advanced Pooling Patterns

Beyond basic pooling, advanced patterns address specific challenges in production systems.

Separate Pools for Different Workloads:

Mix of quick queries and long-running analytics can cause head-of-line blocking. Solution: separate pools.

advanced-pooling.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
// Separate pools for different workloads
const transactionalPool = new Pool({
    ...baseConfig,
    max: 20,
    connectionTimeoutMillis: 5000,  // Fast timeout for OLTP
    idleTimeoutMillis: 60000,
});
 
const analyticsPool = new Pool({
    ...baseConfig,
    max: 5,                          // Fewer connections for heavy queries
    connectionTimeoutMillis: 30000,  // Longer timeout acceptable
    statement_timeout: '300000',     // 5 minute query timeout
});
 
// Read replica routing
interface PoolSet {
    primary: Pool;
    replicas: Pool[];
    currentReplica: number;
}
 
function createPoolSet(config: PoolConfig, replicaConfigs: PoolConfig[]): PoolSet {
    return {
        primary: new Pool(config),
        replicas: replicaConfigs.map(c => new Pool(c)),
        currentReplica: 0,
    };
}
 
function getReadPool(poolSet: PoolSet): Pool {
    // Round-robin across replicas
    if (poolSet.replicas.length === 0) {
        return poolSet.primary;
    }
    
    const pool = poolSet.replicas[poolSet.currentReplica];
    poolSet.currentReplica = (poolSet.currentReplica + 1) % poolSet.replicas.length;
    return pool;
}
 
function getWritePool(poolSet: PoolSet): Pool {
    return poolSet.primary;
}
 
const dbPools = createPoolSet(
    { host: 'primary.db.internal', ...baseConfig },
    [
        { host: 'replica-1.db.internal', ...baseConfig },
        { host: 'replica-2.db.internal', ...baseConfig },
    ]
);
 
// Usage
async function getUser(id: string) {
    const pool = getReadPool(dbPools);
    return pool.query('SELECT * FROM users WHERE id = $1', [id]);
}
 
async function updateUser(id: string, data: Partial<User>) {
    const pool = getWritePool(dbPools);
    return pool.query('UPDATE users SET name = $1 WHERE id = $2', [data.name, id]);
}
 
// Priority-based connection acquisition
class PriorityPool {
    private high: Pool;
    private low: Pool;
    
    constructor(config: PoolConfig) {
        // 70% for high priority, 30% for low
        const highMax = Math.ceil(config.max! * 0.7);
        const lowMax = Math.floor(config.max! * 0.3);
        
        this.high = new Pool({ ...config, max: highMax });
        this.low = new Pool({ ...config, max: lowMax });
    }
    
    acquire(priority: 'high' | 'low' = 'low'): Promise<PoolClient> {
        return priority === 'high' 
            ? this.high.connect()
            : this.low.connect();
    }
}
 
// Usage: critical path gets high priority
async function processPayment(data: PaymentData) {
    const client = await priorityPool.acquire('high');
    // ...
}
 
async function generateReport(params: ReportParams) {
    const client = await priorityPool.acquire('low');
    // ...
}

Monitor All Pools

Summary: Connection Pooling

Key Takeaways

•Understand connection cost — TCP handshake + TLS + auth + init can add 200ms+ to each new connection. Pooling amortizes this across many operations.
•Size pools appropriately — Start with (cores × 2) + spindles. Too small = request queuing; too large = backend overload.
•Use connection proxies at scale — PgBouncer, ProxySQL multiplex many app connections over fewer database connections.
•Pool HTTP clients too — Same TCP/TLS overhead applies. Use keep-alive agents, HTTP/2 for multiplexing.
•Redis needs less pooling — Single-threaded Redis benefits more from pipelining. Pool when blocking commands or true parallelism needed.
•Monitor pool health — Track wait time, active/idle counts, acquisition time. Alert on exhaustion before users feel it.
•Prevent common issues — Leak detection, connection validation, warm-up on startup, jittered max lifetime.
•Use separate pools for different workloads — OLTP vs analytics, reads vs writes, critical vs background.

Module Complete:

You've now completed the Latency Optimization module. You've learned to reduce latency through:

Network optimization — Protocol tuning, geographic distribution, payload compression
Database query optimization — Indexing, query rewriting, execution plan analysis
Caching — Multi-layer caching, access patterns, invalidation strategies
Async processing — Queues, event-driven architecture, saga patterns
Connection pooling — Pool mechanics, sizing, advanced patterns

These techniques, applied systematically, can reduce latency by orders of magnitude—transforming slow, frustrating systems into responsive, delightful user experiences.

Module Complete

5 / 5