CPU-Bound vs I/O-Bound: The Fundamental Bottleneck Dichotomy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

import math
import time
from concurrent.futures import ProcessPoolExecutor
import multiprocessing
 
def cpu_intensive_calculation(n: int) -> float:
    """
    Example CPU-bound operation: computing prime factors
    This is purely computational - no I/O, just CPU cycles
    """
    result = 0.0
    for i in range(1, n + 1):
        # Artificially CPU-intensive: lots of math operations
        result += math.sin(i) * math.cos(i) * math.sqrt(abs(math.tan(i) + 1))
    return result
 
def benchmark_cpu_bound_workload():
    """
    Demonstrates that CPU-bound work scales with CPU cores,
    and adding threads beyond core count provides no benefit.
    """
    iterations = 10_000_000
    num_cores = multiprocessing.cpu_count()
    
    # Single-threaded baseline
    start = time.time()
    cpu_intensive_calculation(iterations)
    single_thread_time = time.time() - start
    print(f"Single thread: {single_thread_time:.2f}s")
    
    # Scale with process pool (bypasses Python GIL)
    work_chunks = [iterations // num_cores] * num_cores
    
    start = time.time()
    with ProcessPoolExecutor(max_workers=num_cores) as executor:
        list(executor.map(cpu_intensive_calculation, work_chunks))
    multi_process_time = time.time() - start
    
    speedup = single_thread_time / multi_process_time
    print(f"{num_cores} processes: {multi_process_time:.2f}s (speedup: {speedup:.2f}x)")
    
    # OBSERVATION: Speedup approaches num_cores for truly CPU-bound work
    # Adding more processes than cores would NOT improve performance
 
# Key Insight: For CPU-bound work, threads > CPU cores yields diminishing returns
# The limiting factor is physical compute capacity

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

import asyncio
import time
import aiohttp
from concurrent.futures import ThreadPoolExecutor
import requests
 
async def fetch_url_async(session: aiohttp.ClientSession, url: str) -> int:
    """
    Example I/O-bound operation: HTTP request
    The CPU does almost nothing - just waits for network response
    """
    async with session.get(url) as response:
        content = await response.text()
        return len(content)
 
def fetch_url_sync(url: str) -> int:
    """Synchronous version for comparison"""
    response = requests.get(url)
    return len(response.text)
 
async def benchmark_io_bound_workload():
    """
    Demonstrates that I/O-bound work scales dramatically with concurrency,
    even on a single CPU core.
    """
    urls = ["https://httpbin.org/delay/1"] * 10  # Each takes 1 second
    
    # Sequential: 10 requests × 1 second = ~10 seconds
    start = time.time()
    for url in urls:
        fetch_url_sync(url)
    sequential_time = time.time() - start
    print(f"Sequential: {sequential_time:.2f}s")
    
    # Concurrent with async/await: all requests in parallel
    start = time.time()
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(*[fetch_url_async(session, url) for url in urls])
    async_time = time.time() - start
    print(f"Async concurrent: {async_time:.2f}s")
    
    speedup = sequential_time / async_time
    print(f"Speedup: {speedup:.1f}x")
    
    # OBSERVATION: 10x speedup on a single core!
    # For I/O-bound work, concurrency (not parallelism) is the key
 
# Key Insight: I/O-bound work benefits from concurrency within a single core
# The limiting factor is I/O latency, not CPU cycles
# Async programming, event loops, and thread pools are effective strategies

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

#!/bin/bash
# Quick diagnostic commands for bottleneck identification
 
# ---------------------------------------------
# CPU UTILIZATION
# ---------------------------------------------
 
# Overall and per-core CPU usage (Linux)
top -1           # Shows per-core breakdown
htop             # Interactive, shows per-core with graphs
mpstat -P ALL 1  # Per-CPU statistics every second
 
# Process-specific CPU usage
pidstat -u 1     # Per-process CPU every second
perf top         # Real-time function-level CPU profile
 
# ---------------------------------------------
# I/O WAIT AND BLOCKING
# ---------------------------------------------
 
# I/O wait indicator (high iowait = I/O-bound)
vmstat 1         # Look at 'wa' column (I/O wait %)
iostat -x 1      # Disk I/O statistics with queue depths
 
# Network I/O
ss -s            # Socket statistics summary
netstat -i       # Network interface statistics
iftop            # Interactive network traffic
 
# ---------------------------------------------
# THREAD STATE ANALYSIS
# ---------------------------------------------
 
# What are threads doing right now?
ps -eo pid,stat,cmd | grep <your_process>
# D = uninterruptible sleep (disk I/O)
# S = sleeping (often waiting for I/O)
# R = running (using CPU)
 
# Java-specific: thread dump
jstack <pid>     # Shows what each thread is doing
 
# System-wide: trace blocking operations
strace -c -p <pid>  # System call summary with time
 
# ---------------------------------------------
# QUICK DIAGNOSIS PATTERN
# ---------------------------------------------
# 
# High CPU + low iowait = CPU-bound
# Low CPU + high iowait = Disk I/O-bound
# Low CPU + low iowait + slow = Network I/O-bound (waiting on remote)
# Low CPU + high context switches = Lock contention

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

Investigation Steps:
 
1. Check CPU under load:
   $ top
   CPU: 8% user, 2% system, 90% idle
   
   → Low CPU utilization = NOT CPU-bound
 
2. Check application metrics:
   Average DB query time: 1.4 seconds
   Average HTTP downstream calls: 0.5 seconds
   Application logic: 0.1 seconds
   
   → Time spent in I/O: 1.9 seconds (95%)
 
3. Check thread pool:
   Pool size: 10 threads
   All threads frequently WAITING on database connection
   
   → I/O-bound, specifically database-bound
 
Diagnosis: I/O-bound (database queries)
 
Solutions:
1. Optimize slow queries (add indexes, rewrite)
2. Add caching for repeated queries
3. Increase connection pool size
4. Consider read replicas
 
What would NOT help:
- Faster CPU (would save 0.1 seconds, 5%)
- More powerful servers (same I/O wait time)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

Investigation Steps:
 
1. Check CPU under load:
   $ top
   CPU: 99% user on 1 core
   Other 7 cores: idle
   
   → Single-core saturation = CPU-bound, single-threaded
 
2. Profile the application:
   $ py-spy top --pid 12345
   
   85% time in: PIL.Image.resize()
   10% time in: jpeg_encode()
   5% time in: file_read/file_write
   
   → CPU time dominates (95%)
 
3. Check thread count:
   Single-threaded execution (only 1 worker)
 
Diagnosis: CPU-bound (image processing)
 
Solutions:
1. Parallelize: run multiple worker processes (one per core)
2. Use optimized libraries (Pillow-SIMD, libvips)
3. Offload to GPU if available
4. Consider worker pool sized to core count (8 workers)
 
Expected outcome after parallelization:
- 8 workers on 8 cores = ~40 images/minute (8x improvement)
 
What would NOT help:
- Async I/O (only 5% is I/O)
- Faster network
- More threads on single process (GIL limits to 1 core)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

Investigation Steps:
 
1. Check CPU and I/O:
   CPU: 25% user
   iowait: 2%
   
   → Neither classically CPU-bound nor I/O-bound
 
2. Check thread states:
   $ jstack 12345 | grep -c "BLOCKED"
   47 threads BLOCKED
   
   → Many threads blocked on locks!
 
3. Identify contention:
   $ jstack 12345 | grep -A 3 "BLOCKED"
   
   Threads waiting to acquire:
   - java.util.HashMap (not thread-safe!)
   - Custom cache object lock
   
4. Analyze lock duration:
   One thread holds lock for 200ms doing I/O
   47 threads wait for that single lock
 
Diagnosis: Lock contention (hidden bottleneck)
 
This looks I/O-bound from CPU metrics, but the actual
bottleneck is serialization due to coarse locking.
 
Solutions:
1. Use ConcurrentHashMap instead of synchronized HashMap
2. Reduce lock scope (don't hold locks during I/O)
3. Use lock-free data structures where possible
4. Consider read-write locks for read-heavy workloads
 
Key Insight:
Lock contention is often mistaken for I/O-bound behavior.
The symptom (low CPU, slow response) is similar, but the
cause (serialization) requires a different solution.