Asynchronous Programming - Learning Module

Loading content...

0/227

Callbacks

The First Async Pattern

When you ask the operating system to perform an asynchronous operation—reading a file, waiting for network data, or setting a timer—a fundamental question arises: how does the system tell you when the operation is complete?

The oldest and most fundamental answer is the callback: a function you provide that the system invokes when the operation finishes. The callback pattern predates modern programming languages, with roots in hardware interrupt handlers and early event-driven systems.

Callbacks are deceptively simple in concept: 'Call this function when you're done.' But as we'll discover, this simple idea leads to complex patterns, subtle bugs, and architectural challenges that shaped the evolution of all subsequent async abstractions.

What You Will Learn

By the end of this page, you will understand the mechanics of callback-based async programming, common callback patterns and conventions, error handling strategies, continuation-passing style (CPS), inversion of control, and the notorious 'callback hell' problem that motivated promises and async/await.

What Is a Callback?

A callback is a function passed as an argument to another function, intended to be invoked ('called back') at some later point—typically when an asynchronous operation completes.

Etymology and concept:

The term 'callback' reflects the control flow: you call a function, passing your callback; that function later calls back to your code via the function you provided. You're essentially saying: 'Here's my phone number—call me back when you have an answer.'

The fundamental signature:

async_operation(input_data, callback_function)

Where callback_function has a signature like:

callback_function(error, result)

This pattern appears in nearly every system that supports asynchronous operations, from kernel signal handlers to JavaScript event listeners.

callback_basics.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// The most basic callback pattern
 
// Synchronous version - blocks until file is read
const syncData = fs.readFileSync('/path/to/file', 'utf8');
console.log('Got data:', syncData);
console.log('This runs after read completes');
 
// Asynchronous callback version - continues immediately
fs.readFile('/path/to/file', 'utf8', function(error, data) {
    // This function is the CALLBACK
    // It runs LATER, when the file read completes
    if (error) {
        console.error('Read failed:', error);
        return;
    }
    console.log('Got data:', data);
});
 
console.log('This runs IMMEDIATELY, before read completes');
 
// Output order:
// 1. "This runs IMMEDIATELY, before read completes"
// 2. (... file read happens asynchronously ...)
// 3. "Got data: [file contents]"
 
/*
 * Key observations:
 * - readFile returns immediately, doesn't wait for I/O
 * - Callback is invoked LATER by the event loop
 * - Code after readFile() runs BEFORE callback
 * - This INVERTS normal top-to-bottom execution
 */

Callbacks at the OS level:

The callback concept originates from how hardware and operating systems handle events:

1. Hardware Interrupts

When hardware needs attention, it triggers an interrupt
CPU jumps to an Interrupt Service Routine (ISR)—essentially a hardware callback
ISR handles the event and returns to normal execution

2. Signal Handlers

UNIX signals use callbacks via signal() or sigaction()
You register a function; OS calls it when the signal arrives
Classic callback pattern at the system level

3. Event Notification

select(), poll(), epoll_wait() notify which descriptors are ready
Your code then 'callbacks' itself by dispatching to handlers
The callback dispatch is explicit rather than implicit

The Two Meanings of 'Callback'

The term 'callback' has two related but distinct meanings: (1) A function passed to async operations that's invoked on completion, and (2) Any function passed to another function for later invocation (e.g., map, filter). In this context, we focus on the async completion sense, but the concepts overlap.

Callback Mechanics: How It Actually Works

Understanding how callbacks work requires understanding what happens beneath the abstraction. Let's trace the complete lifecycle of an asynchronous callback operation.

Step-by-step breakdown of an async file read:

callback_lifecycle.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
/*
 * Tracing what happens during an async callback operation
 * Using Linux async I/O with signal notification as an example
 */
 
#include <signal.h>
#include <aio.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
 
char buffer[4096];
struct aiocb cb;
 
// STEP 4: OS invokes this callback when I/O completes
void read_complete_handler(int sig, siginfo_t *si, void *ctx) {
    // STEP 5: Verify which operation completed
    if (aio_error(&cb) == 0) {
        ssize_t bytes = aio_return(&cb);
        printf("CALLBACK INVOKED: Read %zd bytes\n", bytes);
        printf("Data: %.*s\n", (int)bytes, buffer);
    }
}
 
int main() {
    int fd = open("test.txt", O_RDONLY);
    
    // STEP 1: Register the callback (signal handler)
    struct sigaction sa;
    sa.sa_sigaction = read_complete_handler;
    sa.sa_flags = SA_SIGINFO;
    sigemptyset(&sa.sa_mask);
    sigaction(SIGIO, &sa, NULL);
    
    // STEP 2: Set up and submit async operation
    memset(&cb, 0, sizeof(cb));
    cb.aio_fildes = fd;
    cb.aio_buf = buffer;
    cb.aio_nbytes = sizeof(buffer);
    cb.aio_sigevent.sigev_notify = SIGEV_SIGNAL;
    cb.aio_sigevent.sigev_signo = SIGIO;
    
    printf("STEP 2: Submitting async read...\n");
    aio_read(&cb);  // Returns immediately!
    
    // STEP 3: Continue doing other work
    printf("STEP 3: Doing other work while read proceeds...\n");
    for (int i = 0; i < 5; i++) {
        printf("  Working... iteration %d\n", i);
        usleep(100000);  // 100ms
    }
    
    printf("Waiting for callback...\n");
    pause();  // Wait for signal (in production: proper event loop)
    
    close(fd);
    return 0;
}
 
/*
 * Execution Flow:
 * 
 * User Code                  Kernel                    Hardware
 * ---------                  ------                    --------
 *    |                          |                         |
 *    |-- aio_read() ----------->|                         |
 *    |<----- returns immediately|                         |
 *    |                          |-- queue read request -->|
 *    |  (doing other work)      |                         |
 *    |                          |                   [disk activity]
 *    |                          |<-- interrupt: complete--|
 *    |                          |                         |
 *    |<-- signal SIGIO ---------|                         |
 *    |                          |                         |
 *    +--> read_complete_handler |                         |
 *         (callback executes)   |                         |
 */

The callback registration and invocation phases:

Registration Phase:

User code prepares the callback function
User code passes callback to the async API
System stores the callback reference internally
Async operation begins in the background
Original call returns immediately

Invocation Phase:

Operation completes (I/O done, timer expires, etc.)
System retrieves stored callback reference
System prepares callback arguments (error/result)
System invokes the callback function
Callback executes in some execution context

Critical question: Where does the callback execute?

This varies by system:

System	Callback Context
Signal handlers	Interrupt context (limited operations allowed)
Node.js	Event loop thread (single-threaded)
Java AIO	Dedicated I/O thread pool
Windows IOCP	Thread pool thread
io_uring	User chooses (polling or kernel interrupt)

Callback Execution Context Matters

Understanding WHERE your callback executes is crucial for correctness. Signal handlers have severe restrictions (can only call async-signal-safe functions). Thread pool callbacks must handle synchronization. Event loop callbacks must not block. Always verify the execution context for your platform.

The Error-First Callback Convention

A critical challenge in callback-based programming is error handling. Unlike synchronous code where exceptions propagate up the call stack, callbacks execute in a different context—there's no caller to throw to.

The most influential solution is the error-first callback convention, popularized by Node.js but applicable to any callback-based system.

The convention:

callback(error, result)

First argument: Error object if operation failed, null or undefined if successful
Subsequent arguments: Result data if successful

This pattern ensures that error handling is impossible to forget—the first thing you must do is check the error parameter.

error_first_callback.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
// Error-first callback pattern in action
 
const fs = require('fs');
 
// CORRECT: Always handle error first
fs.readFile('/path/to/file', 'utf8', function(err, data) {
    // Error is ALWAYS the first parameter
    if (err) {
        // Handle the error appropriately
        console.error('Failed to read file:', err.message);
        console.error('Error code:', err.code);  // e.g., 'ENOENT', 'EACCES'
        
        // Common patterns:
        // 1. Log and return
        // 2. Call an error callback
        // 3. Use a default/fallback value
        // 4. Propagate to parent callback
        return;  // IMPORTANT: Stop execution after error
    }
    
    // Only reaches here if err is null/undefined
    console.log('File contents:', data);
    processData(data);
});
 
// WRONG: Ignoring potential errors
fs.readFile('/file', 'utf8', function(err, data) {
    // BUG: Not checking err - data might be undefined!
    console.log(data.length);  // CRASHES if file doesn't exist
});
 
// Pattern for propagating errors through callback chains
function readAndProcess(filename, callback) {
    fs.readFile(filename, 'utf8', function(err, data) {
        if (err) {
            // Propagate error to our caller's callback
            return callback(err, null);
        }
        
        // Process the data
        try {
            const result = JSON.parse(data);
            callback(null, result);  // Success: null error, result data
        } catch (parseError) {
            // Convert sync error to callback error
            callback(parseError, null);
        }
    });
}
 
// Using the function
readAndProcess('config.json', function(err, config) {
    if (err) {
        console.error('Failed:', err);
        return;
    }
    console.log('Config loaded:', config);
});

Why error-first became standard:

Consistency: Every callback has the same signature structure
Forced handling: You must actively ignore the error parameter
Early return: Errors are handled first, happy path follows
Composability: Error propagation follows a consistent pattern

Alternative conventions:

Some systems use different patterns:

Pattern	Signature	Example
Error-first	`callback(err, result)`	Node.js
Result-first	`callback(result, err)`	Some older APIs
Separate callbacks	`operation(onSuccess, onFailure)`	jQuery AJAX
Result wrapper	`callback({ error, data })`	Some RPC systems

The Golden Rule

Always handle the error case first and return immediately. Never let execution 'fall through' to the success logic when an error occurred. This pattern prevents subtle bugs where error state pollutes success handling.

Continuation-Passing Style (CPS)

Callbacks are an application of a broader programming pattern called Continuation-Passing Style (CPS). Understanding CPS illuminates why callback-based code looks the way it does.

What is a continuation?

A continuation represents 'the rest of the computation'—everything that happens after the current point in the program. When you pass a callback, you're essentially passing the continuation: 'After you finish, here's what to do next.'

Direct style vs CPS:

// DIRECT STYLE (normal code)
let x = compute_a();
let y = compute_b(x);
let z = compute_c(y);
return z;

// CONTINUATION-PASSING STYLE
compute_a((x) => {
    compute_b(x, (y) => {
        compute_c(y, (z) => {
            return_continuation(z);
        });
    });
});

In CPS, every function receives an extra argument: the continuation (callback) representing what to do with the result.

cps_transformation.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
// Transforming direct style to CPS
 
// ========================================
// DIRECT STYLE - Sequential, blocking
// ========================================
function processUserDirect(userId) {
    const user = database.getUser(userId);        // blocks
    const orders = database.getOrders(user.id);   // blocks
    const total = calculateTotal(orders);         // blocks
    return formatReport(user, orders, total);     // blocks
}
 
// Usage:
const report = processUserDirect(123);
console.log(report);
 
 
// ========================================
// CPS - Non-blocking, callback-based
// ========================================
function processUserCPS(userId, done) {
    // Each operation passes its continuation as a callback
    
    database.getUser(userId, function(err, user) {
        if (err) return done(err);
        
        // Continuation 1: After getting user, get orders
        database.getOrders(user.id, function(err, orders) {
            if (err) return done(err);
            
            // Continuation 2: After getting orders, calculate
            calculateTotal(orders, function(err, total) {
                if (err) return done(err);
                
                // Continuation 3: After calculation, format
                formatReport(user, orders, total, function(err, report) {
                    if (err) return done(err);
                    
                    // Final continuation: return result
                    done(null, report);
                });
            });
        });
    });
}
 
// Usage:
processUserCPS(123, function(err, report) {
    if (err) {
        console.error('Failed:', err);
        return;
    }
    console.log(report);
});
 
 
// ========================================
// CPS UTILITY: Converting sync to CPS
// ========================================
function syncToCPS(syncFn) {
    return function(...args) {
        const callback = args.pop();  // Last arg is continuation
        try {
            const result = syncFn(...args);
            // Use setImmediate to maintain async semantics
            setImmediate(() => callback(null, result));
        } catch (err) {
            setImmediate(() => callback(err));
        }
    };
}
 
// Convert a sync function to CPS
const parseCPS = syncToCPS(JSON.parse);
 
parseCPS('{"name": "Alice"}', function(err, obj) {
    if (err) return console.error(err);
    console.log(obj.name);  // "Alice"
});

Mathematical foundations of CPS:

CPS has deep theoretical roots in lambda calculus and programming language theory:

Every program can be mechanically transformed into CPS
Control flow becomes explicit: loops, conditionals, and exceptions become first-class values
Inversion of control: callee controls when/if to invoke continuation
Enables advanced features: coroutines, generators, and async/await can be desugared to CPS

The call stack in CPS:

In direct style, function calls build up the call stack, and returns pop it. In CPS, there are no returns—each function tail-calls its continuation. The 'stack' is instead represented as nested closures (callbacks within callbacks).

Direct: main() calls a() calls b() calls c()
        Stack: [main, a, b, c]
        c() returns, then b(), then a(), then main()

CPS:    main_cps(cont_main)
          └─→ a_cps(cont_a)  
               └─→ b_cps(cont_b)
                    └─→ c_cps(cont_c)
                         └─→ cont_c() → cont_b() → cont_a() → cont_main()
        No stack buildup - each is a tail call

CPS in Compilers

Many compilers (including those for ML, Scheme, and JavaScript) use CPS as an intermediate representation. It makes control flow explicit, simplifying optimization. When you write async/await, the compiler may transform your code through CPS internally.

Inversion of Control: The Callback Tradeoff

When you pass a callback to an async function, you surrender something important: control over when and how your code executes. This is called Inversion of Control (IoC).

In direct style code:

You call functions
You control the order of execution
You handle returns and exceptions
Execution flow is explicit and local

With callbacks:

The async function decides when to call you
You trust it to call you exactly once
You trust it to pass correct arguments
You can't easily cancel or timeout
Error handling becomes non-local

Trust Assumptions (Ideal)

•Callback invoked exactly once
•Called with correct argument types
•Called at the right time
•Error passed first if failure
•Execution context is safe

What Can Go Wrong

•Callback never called (lost)
•Callback called multiple times
•Called synchronously vs async
•Called with wrong arguments
•Called in wrong context (this)

callback_trust_issues.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
// Examples of callback trust violations
 
// ========================================
// PROBLEM 1: Callback never invoked
// ========================================
function riskyOperation(data, callback) {
    if (data.isValid) {
        process(data, callback);
    }
    // BUG: What if !data.isValid? Callback never called!
    // Caller waits forever
}
 
// ========================================
// PROBLEM 2: Callback invoked multiple times  
// ========================================
function leakyOperation(items, callback) {
    items.forEach(item => {
        validate(item, (err, result) => {
            if (err) {
                callback(err);  // BUG: Called for EACH error!
            }
        });
    });
    callback(null, 'done');  // And called here too!
}
 
// ========================================
// PROBLEM 3: Sync vs Async inconsistency
// ========================================
function cachingOperation(key, callback) {
    if (cache.has(key)) {
        // BUG: Calling sync! Violates async contract
        callback(null, cache.get(key));
        return;
    }
    // This path is async
    database.get(key, callback);
}
 
// The problem:
cachingOperation('key', (err, value) => {
    console.log('Callback executed');
});
console.log('After call');
 
// Uncached: "After call" then "Callback executed" (expected)
// Cached: "Callback executed" then "After call" (UNEXPECTED!)
 
// ========================================
// SOLUTION: Zalgo avoidance
// ========================================
function safeOperation(key, callback) {
    if (cache.has(key)) {
        // ALWAYS make callback async, even for cached
        setImmediate(() => callback(null, cache.get(key)));
        return;
    }
    database.get(key, callback);
}

The 'Zalgo' problem:

An infamous blog post by Isaac Schlueter (creator of npm) introduced 'Zalgo'—the demon of inconsistent async behavior. When a function sometimes calls its callback synchronously and sometimes asynchronously, it creates subtle bugs:

Event handlers might not be registered yet
Variables might not be initialized
Stack assumptions are violated
Race conditions appear and disappear

The rule: 'Don't release Zalgo'

If a function accepts a callback, it should ALWAYS invoke that callback asynchronously, even if the result is immediately available. Use setImmediate(), process.nextTick(), or setTimeout(cb, 0) to defer.

Defensive callback programming:

// Wrapper to protect against misbehaving APIs
function safeCallback(callback) {
    let called = false;
    return function(...args) {
        if (called) {
            console.error('Callback called more than once!');
            return;
        }
        called = true;
        callback.apply(this, args);
    };
}

The Trust Tax

Every callback-based API requires you to trust that it follows the conventions correctly. This trust tax increases cognitive load and debugging difficulty. Promises and async/await were designed partly to regain control and provide guarantees that callbacks cannot.

Callback Hell: When Async Gets Ugly

When multiple async operations depend on each other sequentially, callbacks nest within callbacks within callbacks. This pattern, known colloquially as 'callback hell' or the 'pyramid of doom', is the most notorious problem with callback-based programming.

The pyramid pattern:

callback_hell.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
// The classic callback hell pattern
 
function processOrder(orderId, finalCallback) {
    // Look up the order
    database.getOrder(orderId, function(err, order) {
        if (err) return finalCallback(err);
        
        // Look up the customer
        database.getCustomer(order.customerId, function(err, customer) {
            if (err) return finalCallback(err);
            
            // Check inventory for each item
            checkInventory(order.items, function(err, availability) {
                if (err) return finalCallback(err);
                
                // Calculate shipping
                calculateShipping(customer.address, order.items, function(err, shipping) {
                    if (err) return finalCallback(err);
                    
                    // Apply discounts
                    getDiscounts(customer.id, function(err, discounts) {
                        if (err) return finalCallback(err);
                        
                        // Calculate final price
                        calculateTotal(order, shipping, discounts, function(err, total) {
                            if (err) return finalCallback(err);
                            
                            // Charge the customer
                            chargeCard(customer.paymentMethod, total, function(err, charge) {
                                if (err) return finalCallback(err);
                                
                                // Update order status
                                database.updateOrder(orderId, { 
                                    status: 'paid', 
                                    chargeId: charge.id 
                                }, function(err, updated) {
                                    if (err) return finalCallback(err);
                                    
                                    // Send confirmation email
                                    sendEmail(customer.email, order, function(err, sent) {
                                        if (err) return finalCallback(err);
                                        
                                        // FINALLY done
                                        finalCallback(null, {
                                            order: updated,
                                            charge: charge,
                                            emailSent: sent
                                        });
                                    });
                                });
                            });
                        });
                    });
                });
            });
        });
    });
}
 
// Visual pattern - the dreaded pyramid:
//
//  database.getOrder(..., function(...) {
//      database.getCustomer(..., function(...) {
//          checkInventory(..., function(...) {
//              calculateShipping(..., function(...) {
//                  getDiscounts(..., function(...) {
//                      calculateTotal(..., function(...) {
//                          chargeCard(..., function(...) {
//                              updateOrder(..., function(...) {
//                                  sendEmail(..., function(...) {
//                                      // 9 levels deep!
//                                  });
//                              });
//                          });
//                      });
//                  });
//              });
//          });
//      });
//  });

Why callback hell is problematic:

Readability: The rightward drift makes code hard to follow. The visual structure obscures the logical structure.
Error handling: Every level needs error checking. Missing one creates silent failures.
Refactoring difficulty: Extracting or reordering steps requires careful callback rewiring.
Mental overhead: Understanding the state at each level requires tracking all outer closures.
Testing complexity: Each nested function needs its own test setup.
Debugging: Stack traces are unreadable, pointing to anonymous functions within anonymous functions.

It's Not Just Indentation

The 'pyramid' is a symptom, not the disease. The real problems are loss of control flow, scattered error handling, and cognitive overhead. Named functions and modularity help the visual problem but don't solve the fundamental inversion of control.

Taming Callback Hell: Patterns and Techniques

Before Promises and async/await, developers created patterns to manage callback complexity. Understanding these patterns illuminates both the problem and the evolution toward better solutions.

named_functions.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
// Pattern 1: Named functions (flatten the pyramid)
 
function processOrder(orderId, finalCallback) {
    // Each step is a named function
    getOrderStep(orderId);
    
    function getOrderStep(orderId) {
        database.getOrder(orderId, handleOrder);
    }
    
    function handleOrder(err, order) {
        if (err) return finalCallback(err);
        database.getCustomer(order.customerId, handleCustomer);
    }
    
    function handleCustomer(err, customer) {
        if (err) return finalCallback(err);
        calculateTotal(order, customer, handleTotal);
    }
    
    function handleTotal(err, total) {
        if (err) return finalCallback(err);
        chargeCard(customer.paymentMethod, total, handleCharge);
    }
    
    function handleCharge(err, charge) {
        if (err) return finalCallback(err);
        finalCallback(null, { order, customer, charge });
    }
}
 
// Benefits:
// - Flat structure, no pyramid
// - Named functions easier to debug
// - Can be unit tested individually
//
// Drawbacks:
// - Variables must be captured in closure or passed
// - Still manually threading callbacks
// - Error handling still repetitive

The fundamental limitation:

All these patterns help manage callbacks, but none solve the core problem: callbacks fundamentally invert control flow. The solution requires a new abstraction that restores linear reasoning about async code.

This leads us to the next async evolution: Futures and Promises—objects that represent eventual values and restore composability to async operations.

Historical Context

The async.js library was essential to Node.js development from 2010-2015. While largely superseded by Promises and async/await, its patterns influenced those later abstractions. Understanding async.js helps you appreciate why Promises are designed the way they are.

Summary: Callbacks

Callbacks are the foundational pattern for async programming, with roots in hardware interrupts and signal handlers. Despite their simplicity, they introduce significant complexity at scale.

Key Takeaways

•Callbacks are functions passed to async operations, invoked when the operation completes.
•Error-first convention (err, result) ensures errors are handled consistently.
•Continuation-Passing Style (CPS) is the theoretical foundation—passing 'what to do next' explicitly.
•Inversion of Control means you trust the async API to call you correctly, once, at the right time.
•'Zalgo' problems arise from inconsistent sync/async behavior—always be async.
•Callback hell emerges from sequential dependencies, creating the 'pyramid of doom'.
•Mitigation patterns (named functions, async.js, modularity) help but don't solve core issues.

What's next:

Callbacks represent 'eventual values' implicitly—you receive the value by having your callback invoked. Futures and Promises make this explicit: an object that represents a value that will be available in the future. This abstraction restores composability and leads to cleaner async code.

Page Complete

You now understand callback-based async programming in depth—the mechanics, conventions, problems, and patterns. While callbacks remain foundational (even Promises are built on them internally), their limitations motivated the evolution to Promises and async/await, which we explore next.