von Neumann Architecture

Every program you've ever run—from a simple "Hello, World" to a complex 3D game engine—executes through the same fundamental process, repeated billions of times per second:

1. Fetch an instruction from memory
2. Decode the instruction to understand what it means
3. Execute the operation specified
4. Repeat

This is the instruction cycle (also called the fetch-decode-execute cycle or machine cycle), and it is the heartbeat of every von Neumann computer. Understanding this cycle in depth is essential for anyone who wants to:

• Understand how compilers translate code to machine instructions
• Reason about CPU performance and optimization
• Comprehend context switching in operating systems
• Debug low-level issues and understand error conditions
• Appreciate modern CPU optimizations (pipelining, out-of-order execution)

This page will trace through the instruction cycle step by step, examine what happens at each stage, and explore the sophisticated optimizations that modern CPUs use to execute instructions faster.

At its core, the instruction cycle is conceptually simple. The CPU is a state machine that endlessly repeats a fixed sequence of operations:

The Canonical Three Steps:

1. FETCH

• Read the instruction at the memory address stored in the Program Counter (PC)
• Place the instruction bits into the Instruction Register (IR)
• Increment the PC to point to the next instruction

2. DECODE

• Examine the bits in the IR
• Determine the operation type (ADD, LOAD, JUMP, etc.)
• Identify the operands (which registers, what memory address)
• Generate control signals for the execution phase

3. EXECUTE

• Perform the operation specified
• This might involve the ALU (for arithmetic/logic), memory (for load/store), or control flow (for jumps/branches)
• Write results to destination (register or memory)
• Update status flags if applicable

The Cycle in Action: A Concrete Example

Let's trace through a specific instruction: ADD R1, R2, R3 (R1 = R2 + R3)

FETCH:

• PC contains 0x1000 (current instruction address)
• CPU places 0x1000 on address bus, asserts read signal
• Memory returns the bit pattern for ADD R1, R2, R3 (say, 0x02123000)
• This pattern is loaded into IR
• PC is incremented to 0x1004 (assuming 4-byte instructions)

DECODE:

• Decoder examines IR content: 0x02123000
• Bits 28-31 (0x0) specify the opcode class: arithmetic
• Bits 24-27 (0x2) specify ADD operation
• Bits 20-23 (0x1) specify destination: R1
• Bits 16-19 (0x2) specify source 1: R2
• Bits 12-15 (0x3) specify source 2: R3
• Control unit generates signals to:
  • Read R2 value to ALU input A
  • Read R3 value to ALU input B
  • Set ALU to ADD mode
  • Route ALU output to R1

EXECUTE:

• ALU performs addition of R2 and R3 contents
• Result is written to R1
• Flags updated: Zero flag if result is 0, Negative flag if result < 0, Carry and Overflow as appropriate

This entire sequence happens in nanoseconds—a 3 GHz CPU completes ~3 billion such cycles per second.

Before the CPU can decode an instruction, we must understand how instructions are encoded—how abstract operations like "add" or "load" are represented as bit patterns.

Components of an Instruction:

Every machine instruction contains:

1. Opcode (Operation Code): Specifies what operation to perform
2. Operand Specifiers: Identify the data to operate on (registers, memory addresses, immediate values)
3. Addressing Mode: How to interpret the operand specifiers

Instruction Formats

Different ISAs (Instruction Set Architectures) use different formats:

Addressing Modes

The operand specifier doesn't always mean the same thing. The addressing mode determines how to interpret it:

The addressing mode affects decode complexity. RISC architectures keep it simple (only a few modes); CISC architectures (x86) support many modes, making decode elaborate.

The fetch phase seems simple—just read the instruction from memory. In practice, substantial complexity exists:

The Fetch Process:

1. Address Generation: The PC value is placed on the internal address bus
2. Cache Lookup: The instruction cache (I-cache) is checked for that address
   • Cache Hit: Instruction returned in ~1 cycle
   • Cache Miss: Must fetch from L2, L3, or main memory (10-300 cycles!)
3. Instruction Delivery: Bits are placed in the IR
4. PC Update: PC is incremented by instruction size

The Role of the Instruction Cache (I-Cache)

Modern CPUs have dedicated instruction caches, separate from data caches. This separation (Modified Harvard architecture at cache level) allows instruction fetch and data access to happen simultaneously.

Typical I-cache characteristics:

• Size: 32-64 KB per core
• Line size: 64 bytes (fetches a block of instructions)
• Associativity: 4-8 way set associative
• Latency: 3-4 cycles

Branch Prediction During Fetch

A critical complication: what happens when the fetched instruction is a branch?

If we wait until we know whether the branch is taken, we waste cycles. Instead, modern CPUs use branch prediction to guess ahead:

1. BTB (Branch Target Buffer): Caches recent branch targets
2. Branch Predictor: Guesses taken/not-taken based on history
3. Speculative Fetch: CPU fetches from predicted path
4. Rollback if Wrong: If prediction was wrong, discard fetched instructions and restart

Branch prediction accuracy exceeds 95% on typical code, but mispredictions cost 10-20 cycles.

The decode phase translates the raw instruction bits into control signals that orchestrate execution. This is where the CPU "understands" what to do.

Decode Responsibilities:

1. Opcode Recognition: Determine the instruction type
2. Operand Extraction: Parse register numbers, immediate values, addressing modes
3. Control Signal Generation: Activate the correct functional units
4. Register Read Initiation: Start fetching operand values from register file
5. Hazard Detection: Identify dependencies that could cause problems

RISC vs CISC Decode Complexity

The x86 Decode Challenge

Modern x86 CPUs face a formidable decode challenge: hundreds of instruction variants, complex addressing modes, and variable length. The solution? Micro-operations (μops).

x86 CPUs internally decode complex CISC instructions into simpler RISC-like micro-operations:

ADD [memory], register  →  μop1: LOAD temp, [memory]
                            μop2: ADD temp, register
                            μop3: STORE [memory], temp

This decouples the complex ISA from execution—the back-end of the CPU is essentially a RISC machine executing μops.

Pre-Decode and Micro-op Cache

To accelerate decode:

1. Pre-Decode: As cache lines are filled, mark instruction boundaries (x86 variable length problem)
2. Micro-op Cache (μop Cache): Cache decoded μops. If the same instruction is executed again, skip decode entirely.

Intel's μop cache ("Decoded Stream Buffer") can hold ~2000 μops, covering most hot loops. This effectively transforms x86 into RISC internally.

Execute is where the actual work happens. Depending on the instruction type, different resources are used:

Instruction Categories and Execution:

Execution for Different Instruction Types:

Write-Back and Commit

The final sub-phase of execute is writing results:

• Register Write: ALU result or loaded data → destination register
• Memory Write: Data → cache/memory (via store buffer)
• PC Update: For branches, PC may or may not be modified
• Flag Update: Condition codes reflect operation result

Instruction Commit and Exceptions

Modern out-of-order CPUs can have many instructions "in flight" simultaneously. Results are committed (made architecturally visible) only when an instruction is definitively correct:

1. All earlier instructions have committed
2. The instruction didn't cause an exception
3. If speculative, the speculation was correct

If an exception occurs (divide by zero, page fault, illegal instruction), instructions after the faulting one must be discarded. This is where the OS takes over—we'll discuss this more in the exceptions section.

If each instruction goes through F-D-E sequentially, and each phase takes a cycle, every instruction takes 3 cycles. But different phases use different hardware—why not overlap them?

The Pipelining Insight:

While instruction N is executing, instruction N+1 can decode, and instruction N+2 can fetch:

Cycle:  1    2    3    4    5    6    7
        ─────────────────────────────────
Inst 1: [F]  [D]  [E]
Inst 2:      [F]  [D]  [E]
Inst 3:           [F]  [D]  [E]
Inst 4:                [F]  [D]  [E]
        ─────────────────────────────────
                    ↑
                 3 instructions in flight
                 After initial fill, 1 instruction 
                 completes per cycle!

Throughput vs Latency:

• Latency: Each instruction still takes 3 cycles from start to finish
• Throughput: One instruction completes per cycle (3× improvement!)

This is the magic of pipelining—the same hardware, used more efficiently.

Modern Pipeline Depths

Pipeline Hazards:

Pipelining isn't free. Several hazards can stall the pipeline:

1. Data Hazards (RAW - Read After Write)

ADD R1, R2, R3    ; R1 written in cycle 5
SUB R4, R1, R5    ; R1 needed in cycle 3 - not ready!

Solution: Forwarding (bypass result directly) or stall.

2. Control Hazards (Branches)

BEQ target        ; Branch outcome known in cycle 3
ADD R1, R2, R3    ; Already fetched - is it correct?

Solution: Branch prediction (guess) + flush on misprediction.

3. Structural Hazards (Resource Conflicts)

LOAD R1, [mem]    ; Uses memory port
STORE R2, [mem]   ; Also needs memory port

Solution: Duplicate resources or stall.

The instruction cycle isn't always allowed to proceed uninterrupted. Interrupts and exceptions can forcibly redirect the CPU:

Types of Interruptions:

1. Hardware Interrupts: External signals from devices

   • Disk completed a transfer
   • Keyboard key pressed
   • Timer expired (critical for OS scheduling!)
   • Network packet arrived

2. Exceptions (Traps): Synchronous, caused by instruction execution

   • Division by zero
   • Invalid opcode
   • Page fault (memory not mapped)
   • System call (deliberate trap to kernel)

3. Faults vs Traps:

   • Fault: Can be fixed; restart faulting instruction (page fault)
   • Trap: Continue after trapping instruction (syscall)
   • Abort: Cannot recover; terminate process (machine check)

The Interrupt Response Sequence:

Precise vs Imprecise Exceptions

In a pipelined, out-of-order CPU, defining "where" an exception occurred is complex:

• Precise Exception: When an exception is taken, all earlier instructions have completed, none later have visible effects. The saved PC points exactly at the faulting instruction. This is essential for OS correctness (page fault handlers need to know which load failed).

• Imprecise Exception: Some earlier instructions may not have completed; some later may have. The state is ambiguous. Processing is difficult.

Modern CPUs guarantee precise exceptions despite out-of-order execution by using a Reorder Buffer (ROB) that commits instructions in program order.

We've traced through the instruction cycle in depth—the core mechanism that brings stored programs to life. Let's consolidate:

What's Next:

We've now covered most of the von Neumann architecture: stored programs, CPU-memory-I/O organization, bus architecture, and the instruction cycle. But no discussion of von Neumann architecture is complete without addressing its fundamental limitation—the von Neumann Bottleneck. The final page in this module explores this critical constraint and the techniques modern systems use to mitigate it.