Skip to main content

CPU Pipeline Architecture

Follow instructions through a five-stage pipeline, distinguish latency from throughput, and connect issue width to out-of-order execution.

An instruction has several jobs to do: fetch its encoding, read its operands, compute a result, access memory if needed, and write back. A pipeline lets different instructions occupy those stages together. The first question is what this overlap buys before dependencies and branches complicate the picture.

Why overlap instructions?

Consider an authored teaching model with five stages, each taking one cycle. Without overlap, one instruction finishes all five stages before the next starts. Three instructions take 15 cycles. With overlap, a new independent instruction enters each cycle, and the same three finish in seven.

Overlap the same three instructions

Without pipelining: one instruction at a time

Without pipelining: three instructions execute sequentially across 15 cycles
InstructionCycle 1Cycle 2Cycle 3Cycle 4Cycle 5Cycle 6Cycle 7Cycle 8Cycle 9Cycle 10Cycle 11Cycle 12Cycle 13Cycle 14Cycle 15
i1IFIDEXMEMWBNot activeNot activeNot activeNot activeNot activeNot activeNot activeNot activeNot activeNot active
i2Not activeNot activeNot activeNot activeNot activeIFIDEXMEMWBNot activeNot activeNot activeNot activeNot active
i3Not activeNot activeNot activeNot activeNot activeNot activeNot activeNot activeNot activeNot activeIFIDEXMEMWB
15cycles0.20Full-run IPC

With pipelining: stages overlap

With pipelining: three instructions overlap across 7 cycles
InstructionCycle 1Cycle 2Cycle 3Cycle 4Cycle 5Cycle 6Cycle 7Cycle 8Cycle 9Cycle 10Cycle 11Cycle 12Cycle 13Cycle 14Cycle 15
i1IFIDEXMEMWBNot activeNot activeNot activeNot activeNot activeNot activeNot activeNot activeNot activeNot active
i2Not activeIFIDEXMEMWBNot activeNot activeNot activeNot activeNot activeNot activeNot activeNot activeNot active
i3Not activeNot activeIFIDEXMEMWBNot activeNot activeNot activeNot activeNot activeNot activeNot activeNot active
7cycles0.43Full-run IPC2.14×speedup

Each instruction still takes 5 cycles. Overlap reduces the time for all three from 15 to 7 cycles. The pipelined rate approaches 1 instruction per cycle for a long stream; this short run averages only 0.43 because it includes filling and draining.

Calculated teaching schedules with five equal one-cycle stages and no hazards. Completion is counted at the end of WB. Both rows use the same cycle scale.

This comparison keeps the stage time and instruction work fixed. It illustrates scheduling, not a measured processor speedup. A real single-cycle implementation may instead use one longer clock cycle for the whole instruction.

The five-stage model

Fetch (IF), Decode (ID), Execute (EX), Memory (MEM), and Write Back (WB) divide the work. Pipeline registers carry each stage's output across a clock edge so the next stage can consume it.

The five duties of the teaching pipeline
IF
Instruction Fetch
PC → Memory. Fetch the next instruction word.
ID
Instruction Decode
Decode the instruction. Read source registers.
EX
Execute
ALU operates on inputs. Compute addresses for loads/stores.
MEM
Memory Access
Read or write data memory (for loads / stores).
WB
Write Back
Write the ALU result or loaded value back to the register file.

Between each pair of stages sits a pipeline register that latches the previous stage's output for the next stage to consume on the next clock edge.

In this model, even an arithmetic instruction passes through MEM, although it has no data-memory operation there. Completion is counted at the end of WB. These conventions make the timelines comparable; they are not a claim that every processor has exactly five stages.

Latency, throughput, fill and drain

Latency is how long one instruction spends in the pipeline: five cycles here. Throughput is the rate of completed instructions: at most one per cycle for this scalar model once it fills. Overlap improves the second quantity without reducing the first.

For nn independent instructions and kk one-cycle stages, the first instruction completes after kk cycles. Each remaining instruction completes one cycle later:

Tpipeline=k+(n−1),Tserial=nk.T_{\mathrm{pipeline}}=k+(n-1), \qquad T_{\mathrm{serial}}=nk.

The ideal cycle-count speedup is therefore:

S=nkn+k−1.S=\frac{nk}{n+k-1}.

It approaches kk for a long stream under these assumptions. A short run pays for the initial fill and final drain, so its average IPC, n/(n+k−1)n/(n+k-1), is below the steady-state limit. The three-instruction example has 3/7≈0.433/7\approx0.43 IPC even though its steady-state limit is 1.

Depth alone also says little about elapsed time. Runtime is cycle count multiplied by clock period. Splitting work into more stages may allow a shorter period, but pipeline registers and uneven stage work add overhead. The explorer below holds cycle time abstract, so it cannot predict a frequency or runtime improvement.

Width: completing more work per cycle

A superscalar design can accept and execute more than one instruction per cycle when resources and dependencies permit. In the idealized comparison below, every stage has the same width and all eight instructions are independent.

One or two instructions per cycle, with equal work

Both schedules run the same eight independent instructions through five one-cycle stages. The wider machine has capacity for two instructions at every stage.

Scalar: one instruction starts per cycle

1-wide pipeline: eight instructions complete in 12 cycles, averaging 0.67 IPC.
InstructionCycle 1Cycle 2Cycle 3Cycle 4Cycle 5Cycle 6Cycle 7Cycle 8Cycle 9Cycle 10Cycle 11Cycle 12
i1IFIDEXMEMWBNot activeNot activeNot activeNot activeNot activeNot activeNot active
i2Not activeIFIDEXMEMWBNot activeNot activeNot activeNot activeNot activeNot active
i3Not activeNot activeIFIDEXMEMWBNot activeNot activeNot activeNot activeNot active
i4Not activeNot activeNot activeIFIDEXMEMWBNot activeNot activeNot activeNot active
i5Not activeNot activeNot activeNot activeIFIDEXMEMWBNot activeNot activeNot active
i6Not activeNot activeNot activeNot activeNot activeIFIDEXMEMWBNot activeNot active
i7Not activeNot activeNot activeNot activeNot activeNot activeIFIDEXMEMWBNot active
i8Not activeNot activeNot activeNot activeNot activeNot activeNot activeIFIDEXMEMWB

Two-wide: two instructions start per cycle

2-wide pipeline: eight instructions complete in 8 cycles, averaging 1.00 IPC.
InstructionCycle 1Cycle 2Cycle 3Cycle 4Cycle 5Cycle 6Cycle 7Cycle 8Cycle 9Cycle 10Cycle 11Cycle 12
i1IFIDEXMEMWBNot activeNot activeNot activeNot activeNot activeNot activeNot active
i2IFIDEXMEMWBNot activeNot activeNot activeNot activeNot activeNot activeNot active
i3Not activeIFIDEXMEMWBNot activeNot activeNot activeNot activeNot activeNot active
i4Not activeIFIDEXMEMWBNot activeNot activeNot activeNot activeNot activeNot active
i5Not activeNot activeIFIDEXMEMWBNot activeNot activeNot activeNot activeNot active
i6Not activeNot activeIFIDEXMEMWBNot activeNot activeNot activeNot activeNot active
i7Not activeNot activeNot activeIFIDEXMEMWBNot activeNot activeNot activeNot active
i8Not activeNot activeNot activeIFIDEXMEMWBNot activeNot activeNot activeNot active
12
Scalar cycles (8 instructions)
8
Two-wide cycles (8 instructions)
0.67
Scalar full-run IPC
1.00
Two-wide full-run IPC

The same eight instructions finish in 12 versus 8 cycles: a 1.50× speedup at the same clock period. Each instruction still takes 5 cycles. For a long independent stream, the ideal rates approach 1 and 2 instructions per cycle; filling and draining lower the averages for this short run.

Calculated ideal schedules with no dependencies or resource conflicts, shown on the same cycle scale. Real superscalar machines share and schedule execution resources; this diagram does not model a particular processor.

At depth five, width one finishes in 12 cycles and width two finishes in eight. Those finite runs average approximately 0.67 and 1.00 IPC respectively. The steady-state limits are 1 and 2. Two-wide capacity does not make this short run twice as fast because both runs still fill and drain.

For a depth kk and width ww with no conflicts, the same reasoning gives:

T=k+⌈nw⌉−1.T=k+\left\lceil\frac{n}{w}\right\rceil-1.

Try depth and width together

Keep the eight instructions fixed. Change depth between 5, 7 and 10, then change width between 1 and 2. Scrub to the first completion and the last completion: which changes increase individual latency, and which admit more work each cycle?

How depth and width change a finite run

Follow the same eight independent instructions from an empty pipeline to completion. Change depth or width, then scrub to the first and last retirements.

Pipeline depth

Pipeline width

Step 1 / 13

8 independent instructions are ready. No cycles have elapsed.

5-stage, 1-wide pipeline: eight instructions complete in 12 cycles. Highlighted cycle 0.
InstructionCycle 1Cycle 2Cycle 3Cycle 4Cycle 5Cycle 6Cycle 7Cycle 8Cycle 9Cycle 10Cycle 11Cycle 12
i1IFIDEXMEMWBNot activeNot activeNot activeNot activeNot activeNot activeNot active
i2Not activeIFIDEXMEMWBNot activeNot activeNot activeNot activeNot activeNot active
i3Not activeNot activeIFIDEXMEMWBNot activeNot activeNot activeNot activeNot active
i4Not activeNot activeNot activeIFIDEXMEMWBNot activeNot activeNot activeNot active
i5Not activeNot activeNot activeNot activeIFIDEXMEMWBNot activeNot activeNot active
i6Not activeNot activeNot activeNot activeNot activeIFIDEXMEMWBNot activeNot active
i7Not activeNot activeNot activeNot activeNot activeNot activeIFIDEXMEMWBNot active
i8Not activeNot activeNot activeNot activeNot activeNot activeNot activeIFIDEXMEMWB
5
Instruction latency (cycles)
12
Cycles for all 8 instructions
0.67
Full-run average IPC
1
Steady-state IPC limit

This run takes ⌈8 / 1⌉ + 5 − 1 = 12 cycles. Its average is 8 / 12 = 0.67 instructions per cycle, including filling and draining. With a long stream of independent work, the rate approaches 1 per cycle. More stages increase latency in cycles; width sets the ideal steady-state rate.

Calculated teaching schedule: each stage takes one cycle, every stage has enough lanes for the selected width, and all operands and memory accesses are ready. Retirement is counted at the end of the final stage. Clock frequency and wall-clock time are not modeled.

The schedules are calculated from the ideal model above. They contain no data dependencies, cache misses, branches or resource conflicts. This establishes a capacity baseline before the next lesson asks why real instruction streams sometimes cannot fill it.

Out-of-order execution: finding ready work

More width helps only when there is ready work to use it. Suppose instruction 2 waits for instruction 1, while instruction 3 has independent operands. An out-of-order scheduler may run instruction 3 first. It still has to preserve the program's required results, exceptions and memory ordering.

Execute when ready, retire in order

A conceptual out-of-order core. Instructions enter the issue queue and reorder buffer at dispatch; execution later reports completion to the buffer.

Fetch
Read instruction stream
↓
Decode
Parse opcodes, operands
↓
Rename
Map architectural → physical regs
Removes false dependencies
↓
Issue Queue
Hold instructions until operands ready
Wakes up ready instructions
↓
Execution Units
ALU · FPU · Load/Store
Run in parallel, out of program order
↓
Reorder Buffer (ROB)
Track in-flight instructions in program order
Bookkeeping for in-order retire
↓
Commit
Retire instructions in program order
State changes become architecturally visible

Renaming and queueing let execution proceed when operands and units are ready. The ROB records completion; retirement makes architectural state permanent in program order. Branch recovery discards younger work when a prediction is wrong. Speculative work can still affect microarchitectural state such as caches.

The diagram groups responsibilities rather than assigning one cycle to each box. A physical-register-file design keeps register results outside the reorder buffer.

The blocks divide that responsibility:

  • Register renaming assigns physical destinations to register writes, separating successive uses of the same architectural name.
  • Issue queues hold operations until their operands are ready and a suitable execution unit is available.
  • The reorder buffer (ROB) tracks instructions in program order so completed work can commit in order, including precise exception handling.
  • Load/store queues track pending memory operations and possible address conflicts. Register independence alone does not prove two memory accesses are independent.

The BOOM pipeline documentation provides a concrete RISC-V implementation of these stages. Its ROB documentation distinguishes execution completion from architectural commit, and its load/store unit documentation explains store-to-load forwarding and recovery from ordering mistakes. The diagram here is an architectural illustration, not a simulation of BOOM.

A different scheduling choice: VLIW

Very Long Instruction Word designs expose groups of operations for the compiler to schedule together. A teaching bundle might contain independent arithmetic, load and store slots. This shifts scheduling work toward the compiler; it does not remove dependencies or make every slot useful. Code size, available parallelism and compatibility across implementations remain tradeoffs. MIT's architecture course treats VLIW as a separate scheduling approach.

From capacity to stalls

The ideal schedules answer how much overlap is possible. They do not yet answer when an operand is ready, which instruction wins a shared resource, or what happens when the next address depends on a branch. Those are the questions for Hazard Detection, the next lesson.

Power provides a separate limit: even useful parallel hardware must fit an operating budget. Dennard Scaling explains that context without being a prerequisite for the dependency walkthrough.

Further reading

  • Patterson & Hennessy: Computer Organization and Design
  • Shen & Lipasti: Modern Processor Design
  • Intel / AMD / ARM optimization guides
  • Agner Fog's optimization manuals

If you found this explanation helpful, consider sharing it with others.

Mastodon