Problem: Algorithm with Varying Steps - Template Method Pattern | OneNoughtOne

0/246

System Design (LLD)Template Method Pattern

Template Method Pattern

LevelIntermediate

Duration50 mins

TopicTemplate Method Pattern

1 / 4

Problem: Algorithm with Varying Steps

The Reusable Algorithm Dilemma

Every experienced software engineer encounters a recurring challenge: you've built an algorithm that works beautifully for one scenario, and now you need to apply it to a similar but not identical situation. The overall structure of the algorithm remains the same, but certain steps must change.

This is the algorithm with varying steps problem—one of the most fundamental challenges in software design. How do you reuse the invariant parts of an algorithm while allowing flexibility in the parts that change? This seemingly simple question leads us directly to one of the most elegant and widely-used behavioral patterns: the Template Method Pattern.

What You Will Learn

By the end of this page, you will understand the core problem that the Template Method Pattern solves. You'll see concrete examples of algorithms with varying steps, explore why naive solutions create maintenance nightmares, and develop the intuition needed to recognize when this pattern is the right choice.

The Pattern of Patterns in Everyday Algorithms

Before we dive into code, let's recognize that algorithms with varying steps are everywhere—not just in software, but in any procedural domain.

Consider making a hot beverage:

Boil water
Prepare the vessel (cup, mug, teapot)
Add the beverage base (tea leaves, coffee grounds, cocoa powder)
Pour the hot water
Add extras (milk, sugar, honey, lemon)
Serve

This six-step algorithm works for tea, coffee, hot chocolate, and countless other beverages. Steps 1, 2, 4, and 6 are essentially identical regardless of what you're making. But steps 3 and 5 vary significantly—the kind of base you add and the kind of extras differ between beverages.

This is the essence of algorithms with varying steps: a fixed structure with customizable operations at specific points.

The Core Insight

An algorithm with varying steps isn't about having completely different algorithms—it's about having one algorithm with customization points. The structure (the 'template') is invariant. Only specific operations within that structure vary.

Software examples are abundant:

Domain	Fixed Algorithm Structure	Varying Steps
Data parsing	Read input → Parse → Validate → Transform → Output	Parsing logic, validation rules
Build systems	Fetch dependencies → Compile → Test → Package → Deploy	Compile commands, test frameworks
Report generation	Gather data → Format → Generate output → Deliver	Data sources, output format
Authentication	Accept credentials → Validate → Create session → Log event	Validation logic, session handling
Game AI	Observe state → Evaluate options → Select action → Execute	Evaluation criteria, action selection

In each case, there's a clear sequence that defines the shape of the algorithm. But within that shape, certain steps must adapt to specific requirements.

The Naive Approach: Copy-Paste Programming

When developers first encounter algorithms with varying steps, the instinctive response is often copy-paste programming. You have a working algorithm, you need a variation, so you copy the code and modify what needs to change.

Let's see this anti-pattern in action with a concrete example: document exporters for a report generation system.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
// PDF Exporter - works great in isolation
class PdfExporter {
    exportReport(data: ReportData): void {
        // Step 1: Gather data (shared logic)
        const enrichedData = this.gatherData(data);
        this.logProgress("Data gathered");
        
        // Step 2: Open output stream (shared logic)
        const stream = this.openOutputStream("report.pdf");
        
        // Step 3: Write header (PDF-specific)
        stream.write("%PDF-1.4
");
        stream.write("%\xE2\xE3\xCF\xD3
");
        this.logProgress("PDF header written");
        
        // Step 4: Write body (PDF-specific)
        const pdfBody = this.renderToPdf(enrichedData);
        stream.write(pdfBody);
        this.logProgress("PDF body written");
        
        // Step 5: Write footer (PDF-specific)
        stream.write("%%EOF
");
        
        // Step 6: Close stream (shared logic)
        stream.close();
        this.logProgress("Export complete");
        
        // Step 7: Notify completion (shared logic)
        this.notifyCompletion("PDF export finished");
    }
    
    private gatherData(data: ReportData): EnrichedData { /* ... */ }
    private openOutputStream(filename: string): OutputStream { /* ... */ }
    private renderToPdf(data: EnrichedData): string { /* ... */ }
    private logProgress(message: string): void { /* ... */ }
    private notifyCompletion(message: string): void { /* ... */ }
}
 
// HTML Exporter - 70% identical, 30% different
class HtmlExporter {
    exportReport(data: ReportData): void {
        // Step 1: Gather data (copy-pasted)
        const enrichedData = this.gatherData(data);
        this.logProgress("Data gathered");
        
        // Step 2: Open output stream (copy-pasted)
        const stream = this.openOutputStream("report.html");
        
        // Step 3: Write header (HTML-specific)
        stream.write("<!DOCTYPE html>
");
        stream.write("<html><head><title>Report</title></head>
");
        stream.write("<body>
");
        this.logProgress("HTML header written");
        
        // Step 4: Write body (HTML-specific)
        const htmlBody = this.renderToHtml(enrichedData);
        stream.write(htmlBody);
        this.logProgress("HTML body written");
        
        // Step 5: Write footer (HTML-specific)
        stream.write("</body></html>
");
        
        // Step 6: Close stream (copy-pasted)
        stream.close();
        this.logProgress("Export complete");
        
        // Step 7: Notify completion (copy-pasted)
        this.notifyCompletion("HTML export finished");
    }
    
    // Many methods are duplicated with only minor differences
    private gatherData(data: ReportData): EnrichedData { /* identical to PDF */ }
    private openOutputStream(filename: string): OutputStream { /* identical */ }
    private renderToHtml(data: EnrichedData): string { /* HTML-specific */ }
    private logProgress(message: string): void { /* identical */ }
    private notifyCompletion(message: string): void { /* identical */ }
}
 
// CSV Exporter - another 70% duplication
class CsvExporter {
    exportReport(data: ReportData): void {
        // Same pattern repeats: copy, modify specific parts
        // ...
    }
}

What's wrong with this approach?

At first glance, copy-paste seems pragmatic—it works, and you can see the entire algorithm in one place. But this approach creates severe problems as the system evolves:

Copy-Paste Anti-Pattern Problems

•Shotgun Surgery — When you need to fix a bug in gatherData() or logProgress(), you must find and update every copy. Miss one, and you have inconsistent behavior.
•Divergent Change — Over time, 'identical' copies drift apart. One developer adds error handling to the PDF version but forgets the others. Another optimizes logging in HTML but nowhere else.
•Code Bloat — The codebase grows linearly with each new exporter. Ten exporters means ten near-identical copies of the seven-step algorithm.
•Testing Overhead — Each exporter must be tested independently. There's no way to test 'the export algorithm' once—you test it N times across N exporters.
•Knowledge Fragmentation — New team members must understand that these classes are related. Nothing in the code explicitly documents the shared algorithm structure.

The Copy-Paste Trap

Copy-paste feels efficient in the moment but creates exponential maintenance burden over time. Every duplicate is a liability waiting to cause inconsistency. The 'quick solution' becomes the expensive legacy.

The Conditional Approach: Type Flags and Switches

A more 'sophisticated' approach that developers often try is consolidating the algorithm into a single class and using type flags or switch statements to handle the variations. This eliminates duplication but introduces its own severe problems.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
enum ExportFormat {
    PDF = 'pdf',
    HTML = 'html',
    CSV = 'csv',
    XML = 'xml',
    JSON = 'json',
}
 
class GenericExporter {
    exportReport(data: ReportData, format: ExportFormat): void {
        // Step 1: Gather data (shared)
        const enrichedData = this.gatherData(data);
        this.logProgress("Data gathered");
        
        // Step 2: Open output stream (shared, but filename varies)
        const extension = this.getExtension(format);
        const stream = this.openOutputStream(`report.${extension}`);
        
        // Step 3: Write header (varies by format)
        switch (format) {
            case ExportFormat.PDF:
                stream.write("%PDF-1.4
%\xE2\xE3\xCF\xD3
");
                break;
            case ExportFormat.HTML:
                stream.write("<!DOCTYPE html><html><head><title>Report</title>
");
                stream.write("</head><body>
");
                break;
            case ExportFormat.CSV:
                // CSV might have headers as first row
                stream.write(this.getCsvHeaders(enrichedData));
                break;
            case ExportFormat.XML:
                stream.write('<?xml version="1.0" encoding="UTF-8"?>
');
                stream.write("<report>
");
                break;
            case ExportFormat.JSON:
                stream.write("{
  \"report\": {
");
                break;
        }
        this.logProgress(`${format} header written`);
        
        // Step 4: Write body (varies by format)
        switch (format) {
            case ExportFormat.PDF:
                stream.write(this.renderToPdf(enrichedData));
                break;
            case ExportFormat.HTML:
                stream.write(this.renderToHtml(enrichedData));
                break;
            case ExportFormat.CSV:
                stream.write(this.renderToCsv(enrichedData));
                break;
            case ExportFormat.XML:
                stream.write(this.renderToXml(enrichedData));
                break;
            case ExportFormat.JSON:
                stream.write(this.renderToJson(enrichedData));
                break;
        }
        this.logProgress(`${format} body written`);
        
        // Step 5: Write footer (varies by format)
        switch (format) {
            case ExportFormat.PDF:
                stream.write("%%EOF
");
                break;
            case ExportFormat.HTML:
                stream.write("</body></html>
");
                break;
            case ExportFormat.CSV:
                // No footer for CSV
                break;
            case ExportFormat.XML:
                stream.write("</report>
");
                break;
            case ExportFormat.JSON:
                stream.write("  }
}");
                break;
        }
        
        // Step 6: Close stream (shared)
        stream.close();
        this.logProgress("Export complete");
        
        // Step 7: Notify completion (shared)
        this.notifyCompletion(`${format} export finished`);
    }
    
    // Helper methods for each format...
    private renderToPdf(data: EnrichedData): string { /* ... */ }
    private renderToHtml(data: EnrichedData): string { /* ... */ }
    private renderToCsv(data: EnrichedData): string { /* ... */ }
    private renderToXml(data: EnrichedData): string { /* ... */ }
    private renderToJson(data: EnrichedData): string { /* ... */ }
    private getCsvHeaders(data: EnrichedData): string { /* ... */ }
    // ... more format-specific helpers
}

Why this is problematic:

The conditional approach trades duplication for a different set of problems—and in many ways, they're worse:

Open/Closed Principle Violation

•Adding a new format (Markdown, YAML) requires modifying the existing class
•Every switch statement must be extended
•The class never stops growing
•Risk of breaking existing formats when adding new ones

Single Responsibility Violation

•One class knows about PDF, HTML, CSV, XML, JSON...
•Changes to any format affect this class
•The class has multiple reasons to change
•Format-specific logic is scattered across switches

Additional Problems

•Complexity Growth — The class becomes a 'god class' that knows too much. A class with 5 formats × 5 switch statements has 25 branches to test.
•Hidden Dependencies — Format-specific logic is spread across multiple switch statements. Understanding the complete HTML export path requires reading the entire method.
•Team Bottleneck — Only one file exists for all export logic. If Developer A is adding Markdown support while Developer B is fixing a CSV bug, merge conflicts are inevitable.
•Testing Difficulty — You can't easily test 'PDF export' in isolation. The test must navigate through all the conditional branches.
•Forgotten Cases — When adding a new format, it's easy to miss one of the switch statements. The compiler might not catch this depending on the language.

The Switch Statement Smell

When the same switch statement (or the same set of type checks) appears in multiple places in your code, that's a strong signal that you have a behavioral variation that should be modeled with polymorphism—not conditionals.

Identifying the Invariant Structure

Before we can solve the algorithm-with-varying-steps problem properly, we must clearly identify what parts are invariant (same for all variations) and what parts are variant (different for each variation).

Let's analyze our document export algorithm:

Algorithm Step Analysis
Step	Operation	Nature	Reasoning
1	Gather data	INVARIANT	All exporters need the same enriched data
2	Open output stream	INVARIANT	All exporters write to a file stream (extension varies)
3	Write header	VARIANT	Each format has a unique header structure
4	Write body	VARIANT	Each format renders content differently
5	Write footer	VARIANT	Each format has a unique footer or closing
6	Close stream	INVARIANT	All exporters must close the stream
7	Notify completion	INVARIANT	All exporters trigger the same notification

The key observation:

When we separate invariants from variants, we see that the sequence of steps is invariant. Every exporter will always:

Gather → 2. Open → 3. Header → 4. Body → 5. Footer → 6. Close → 7. Notify

This sequence never changes. What changes is how certain steps are executed.

This is the crucial insight that leads to the Template Method Pattern:

The algorithm's structure (the sequence of steps and their relationships) should be defined in one place. The implementation of specific steps should be customizable without affecting the structure.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// The algorithm structure (INVARIANT)
// ──────────────────────────────────────────────────────
//   exportReport():
//       gatherData()           ← INVARIANT (same for all)
//       openOutputStream()     ← INVARIANT (same for all)
//       writeHeader()          ← VARIANT   (differs per format)
//       writeBody()            ← VARIANT   (differs per format)
//       writeFooter()          ← VARIANT   (differs per format)
//       closeStream()          ← INVARIANT (same for all)
//       notifyCompletion()     ← INVARIANT (same for all)
// ──────────────────────────────────────────────────────
 
// The invariant parts define the "skeleton" of the algorithm
// The variant parts are "hot spots" that need customization

Finding the Template

When analyzing any algorithm for this pattern, ask: 'If I listed the steps at a high level of abstraction, would they be the same across all variations?' If yes, you have a template. The steps that implement those abstractions differently are your customization points.

Why the Order Matters: Algorithm Contracts

An important aspect of algorithms with varying steps is that the order and relationships between steps are often critical. The algorithm's correctness depends not just on what steps happen, but on their sequence.

Consider what happens if we violate our export algorithm's order:

Order Violations and Their Consequences

•Write body before opening stream → Runtime error: no stream to write to
•Skip gathering data → Body has nothing to render, output is empty
•Close stream before writing footer → Truncated, invalid file
•Write header after body → Malformed document (header appears mid-content)
•Forget to close stream → Resource leak, file possibly corrupted

This is exactly why the copy-paste and conditional approaches are so dangerous. Each copy or branch can accidentally violate the algorithm's implicit contract. A developer modifying the HTML exporter might reorder steps in a way that 'works' for HTML but violates assumptions other parts of the system depend on.

The algorithm's sequence is not just a guideline—it's a contract.

This contract has several components:

Algorithm Contract Components

•Preconditions — What must be true before each step? (e.g., stream must be open before writing)
•Postconditions — What must be true after each step? (e.g., after writing header, header is complete)
•Invariants — What must remain true throughout? (e.g., data consistency, resource validity)
•Ordering constraints — What must come before what? (e.g., gather data before render)

Enforcing the Contract

The ideal solution to algorithms with varying steps would enforce the algorithm's contract while allowing customization. Variant steps should be implementable without any ability to violate the invariant structure. This is precisely what the Template Method Pattern achieves.

Real-World Manifestations of This Problem

The algorithm-with-varying-steps problem appears constantly in production software. Recognizing these situations is crucial for applying the Template Method Pattern effectively. Let's examine several common scenarios:

1. Data Processing Pipelines

Almost every ETL (Extract-Transform-Load) system follows a common structure:

1
2
3
4
5
6
7
8
9
10
11
12
13
// Invariant structure:
// 1. Connect to source
// 2. Extract data (VARIANT: SQL, API, file, etc.)
// 3. Validate data (common validations)
// 4. Transform data (VARIANT: format conversions, calculations)
// 5. Connect to destination
// 6. Load data (VARIANT: bulk insert, upsert, etc.)
// 7. Log metrics and cleanup
 
// Without Template Method, you'd have:
class SqlToWarehouseETL { /* full algorithm, SQL-specific parts */ }
class ApiToWarehouseETL { /* 80% duplicate, API-specific parts */ }
class FileToWarehouseETL { /* 80% duplicate, file-specific parts */ }

2. Web Framework Request Handling

Every web request follows a predictable lifecycle:

1
2
3
4
5
6
7
8
9
10
11
12
13
// Invariant structure:
// 1. Receive request
// 2. Parse request data
// 3. Authenticate user (common middleware)
// 4. Authorize action (VARIANT: permission checks per endpoint)
// 5. Validate input (VARIANT: validation rules per endpoint)
// 6. Handle request (VARIANT: business logic per endpoint)
// 7. Format response (VARIANT: JSON, XML, HTML per endpoint)
// 8. Send response
// 9. Log request metrics
 
// The framework defines the structure
// Each endpoint implements specific steps

3. Test Frameworks

Testing frameworks embody this pattern explicitly:

1
2
3
4
5
6
7
8
9
10
11
12
// Invariant structure (the "test runner template"):
// 1. Setup test suite (beforeAll)
// 2. For each test:
//    a. Setup test (beforeEach)           ← VARIANT per test class
//    b. Execute test (the test itself)   ← VARIANT per test method
//    c. Verify results (assertions)      ← VARIANT per test method
//    d. Teardown test (afterEach)         ← VARIANT per test class
// 3. Teardown test suite (afterAll)
// 4. Report results
 
// JUnit, pytest, Jest—all implement this template
// Tests customize the variant parts

4. Compilers and Interpreters

Language processing follows consistent phases:

1
2
3
4
5
6
7
8
9
10
11
// Invariant structure:
// 1. Read source file
// 2. Lexical analysis (VARIANT: language-specific tokens)
// 3. Parsing (VARIANT: language-specific grammar)
// 4. Semantic analysis (VARIANT: type systems, rules)
// 5. Optimization (common patterns, plus language-specific)
// 6. Code generation (VARIANT: target architecture)
// 7. Output result
 
// GCC, Clang, Rust compiler—all follow this template
// Each language implements specific compilation steps

5. Game Development

Game loops are perhaps the most explicit example:

1
2
3
4
5
6
7
8
9
10
11
12
13
// Invariant structure (every game, every frame):
// 1. Process input (common: read controllers, keyboard, mouse)
// 2. Update game state (VARIANT: game-specific logic)
//    - Update AI
//    - Update physics
//    - Update animations
// 3. Render scene (VARIANT: 2D vs 3D, engine-specific)
// 4. Present frame (common: buffer swap)
// 5. Calculate delta time (common)
// 6. Repeat
 
// Unity, Unreal, custom engines—all follow this template
// Games customize Update() and Render() methods

Pattern Recognition Exercise

Look at frameworks and libraries you use. Almost every one defines a lifecycle or processing pipeline that you customize. React's component lifecycle, Spring's request handling, Express middleware—they're all manifestations of algorithms with varying steps.

Requirements for a Proper Solution

Now that we understand the problem deeply, let's articulate what an ideal solution must achieve. These requirements will serve as our design criteria:

Solution Requirements

•Define the algorithm structure once — The skeleton of the algorithm (the sequence of steps and their relationships) should be defined in exactly one place.
•Allow customization of specific steps — Variant steps should be implementable by different classes without duplicating the invariant structure.
•Enforce the algorithm contract — It should be impossible (or at least difficult) for customizations to violate the algorithm's order and relationships.
•Support the Open/Closed Principle — Adding a new variation (a new exporter format) should not require modifying existing code.
•Maximize code reuse — Shared logic should be written once and inherited by all variations automatically.
•Make the structure explicit — It should be immediately apparent what the algorithm's template is and where the customization points are.
•Provide sensible defaults — For steps that have a 'common' implementation, provide a default that variations can override if needed.

How Naive Approaches Fail These Requirements
Requirement	Copy-Paste	Conditionals
Define structure once	❌ Structure copied N times	✅ One method, but mixed with variants
Allow customization	✅ Full customization per copy	❌ Modify class to customize
Enforce contract	❌ Each copy can violate	⚠️ Sequence fixed but not enforced
Open/Closed Principle	❌ Must copy entire algorithm	❌ Must modify class
Maximize reuse	❌ No reuse; full duplication	⚠️ Helpers reused, but scattered
Make structure explicit	❌ Implicit in duplication	❌ Hidden in conditionals
Provide defaults	⚠️ Copy provides 'defaults'	❌ No default concept

Setting Up for the Solution

With these requirements clearly defined, we're ready to see how the Template Method Pattern addresses each one. The pattern provides an elegant solution that leverages inheritance to define the algorithm structure while allowing polymorphism at customization points.

Summary: The Problem Space

We've thoroughly explored the problem that the Template Method Pattern solves. Let's consolidate our understanding:

Key Takeaways

•Algorithms with varying steps are ubiquitous — From beverage preparation to ETL pipelines to game loops, many algorithms share a common structure with customization points.
•Copy-paste creates maintenance nightmares — Duplicating the entire algorithm for each variation leads to shotgun surgery, inconsistency, and code bloat.
•Conditionals violate core principles — Using type flags and switch statements concentrates all variations in one class, violating Open/Closed and Single Responsibility.
•The structure and the steps must be separated — The invariant sequence of the algorithm should be defined once; the variant implementations should be customizable independently.
•Algorithm order is a contract — The sequence of steps often encodes critical preconditions, postconditions, and invariants that must not be violated.
•The solution must enforce the contract — An ideal solution makes it impossible for customizations to accidentally violate the algorithm's structure.

What's Next:

In the next page, we'll discover how the Template Method Pattern solves this problem elegantly using inheritance and abstract methods. We'll see how a base class can define the algorithm's skeleton while delegating specific steps to subclasses—achieving all the requirements we've outlined while maintaining clean, maintainable code.

Page Complete

You now understand the fundamental problem that motivates the Template Method Pattern. You can identify algorithms with varying steps in real-world code and articulate why naive solutions create long-term problems. Next, we'll explore the elegant solution: abstract classes with hook methods.