An instruction has several jobs to do: fetch its encoding, read its operands, compute a result, access memory if needed, and write back. A pipeline lets different instructions occupy those stages together. The first question is what this overlap buys before dependencies and branches complicate the picture.
Why overlap instructions?
Consider an authored teaching model with five stages, each taking one cycle. Without overlap, one instruction finishes all five stages before the next starts. Three instructions take 15 cycles. With overlap, a new independent instruction enters each cycle, and the same three finish in seven.
Overlap the same three instructions
Without pipelining: one instruction at a time
| Instruction | Cycle 1 | Cycle 2 | Cycle 3 | Cycle 4 | Cycle 5 | Cycle 6 | Cycle 7 | Cycle 8 | Cycle 9 | Cycle 10 | Cycle 11 | Cycle 12 | Cycle 13 | Cycle 14 | Cycle 15 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| i1 | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active | Not active | Not active | Not active | Not active | Not active |
| i2 | Not active | Not active | Not active | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active |
| i3 | Not active | Not active | Not active | Not active | Not active | Not active | Not active | Not active | Not active | Not active | IF | ID | EX | MEM | WB |
With pipelining: stages overlap
| Instruction | Cycle 1 | Cycle 2 | Cycle 3 | Cycle 4 | Cycle 5 | Cycle 6 | Cycle 7 | Cycle 8 | Cycle 9 | Cycle 10 | Cycle 11 | Cycle 12 | Cycle 13 | Cycle 14 | Cycle 15 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| i1 | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active | Not active | Not active | Not active | Not active | Not active |
| i2 | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active | Not active | Not active | Not active | Not active |
| i3 | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active | Not active | Not active | Not active |
Each instruction still takes 5 cycles. Overlap reduces the time for all three from 15 to 7 cycles. The pipelined rate approaches 1 instruction per cycle for a long stream; this short run averages only 0.43 because it includes filling and draining.
This comparison keeps the stage time and instruction work fixed. It illustrates scheduling, not a measured processor speedup. A real single-cycle implementation may instead use one longer clock cycle for the whole instruction.
The five-stage model
Fetch (IF), Decode (ID), Execute (EX), Memory (MEM), and Write Back (WB) divide the work. Pipeline registers carry each stage's output across a clock edge so the next stage can consume it.
Between each pair of stages sits a pipeline register that latches the previous stage's output for the next stage to consume on the next clock edge.
In this model, even an arithmetic instruction passes through MEM, although it has no data-memory operation there. Completion is counted at the end of WB. These conventions make the timelines comparable; they are not a claim that every processor has exactly five stages.
Latency, throughput, fill and drain
Latency is how long one instruction spends in the pipeline: five cycles here. Throughput is the rate of completed instructions: at most one per cycle for this scalar model once it fills. Overlap improves the second quantity without reducing the first.
For independent instructions and one-cycle stages, the first instruction completes after cycles. Each remaining instruction completes one cycle later:
The ideal cycle-count speedup is therefore:
It approaches for a long stream under these assumptions. A short run pays for the initial fill and final drain, so its average IPC, , is below the steady-state limit. The three-instruction example has IPC even though its steady-state limit is 1.
Depth alone also says little about elapsed time. Runtime is cycle count multiplied by clock period. Splitting work into more stages may allow a shorter period, but pipeline registers and uneven stage work add overhead. The explorer below holds cycle time abstract, so it cannot predict a frequency or runtime improvement.
Width: completing more work per cycle
A superscalar design can accept and execute more than one instruction per cycle when resources and dependencies permit. In the idealized comparison below, every stage has the same width and all eight instructions are independent.
One or two instructions per cycle, with equal work
Both schedules run the same eight independent instructions through five one-cycle stages. The wider machine has capacity for two instructions at every stage.
Scalar: one instruction starts per cycle
| Instruction | Cycle 1 | Cycle 2 | Cycle 3 | Cycle 4 | Cycle 5 | Cycle 6 | Cycle 7 | Cycle 8 | Cycle 9 | Cycle 10 | Cycle 11 | Cycle 12 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| i1 | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active | Not active | Not active |
| i2 | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active | Not active |
| i3 | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active |
| i4 | Not active | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active |
| i5 | Not active | Not active | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active |
| i6 | Not active | Not active | Not active | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active |
| i7 | Not active | Not active | Not active | Not active | Not active | Not active | IF | ID | EX | MEM | WB | Not active |
| i8 | Not active | Not active | Not active | Not active | Not active | Not active | Not active | IF | ID | EX | MEM | WB |
Two-wide: two instructions start per cycle
| Instruction | Cycle 1 | Cycle 2 | Cycle 3 | Cycle 4 | Cycle 5 | Cycle 6 | Cycle 7 | Cycle 8 | Cycle 9 | Cycle 10 | Cycle 11 | Cycle 12 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| i1 | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active | Not active | Not active |
| i2 | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active | Not active | Not active |
| i3 | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active | Not active |
| i4 | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active | Not active |
| i5 | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active |
| i6 | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active |
| i7 | Not active | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active |
| i8 | Not active | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active |
The same eight instructions finish in 12 versus 8 cycles: a 1.50× speedup at the same clock period. Each instruction still takes 5 cycles. For a long independent stream, the ideal rates approach 1 and 2 instructions per cycle; filling and draining lower the averages for this short run.
At depth five, width one finishes in 12 cycles and width two finishes in eight. Those finite runs average approximately 0.67 and 1.00 IPC respectively. The steady-state limits are 1 and 2. Two-wide capacity does not make this short run twice as fast because both runs still fill and drain.
For a depth and width with no conflicts, the same reasoning gives:
Try depth and width together
Keep the eight instructions fixed. Change depth between 5, 7 and 10, then change width between 1 and 2. Scrub to the first completion and the last completion: which changes increase individual latency, and which admit more work each cycle?
How depth and width change a finite run
Follow the same eight independent instructions from an empty pipeline to completion. Change depth or width, then scrub to the first and last retirements.
Pipeline depth
Pipeline width
8 independent instructions are ready. No cycles have elapsed.
| Instruction | Cycle 1 | Cycle 2 | Cycle 3 | Cycle 4 | Cycle 5 | Cycle 6 | Cycle 7 | Cycle 8 | Cycle 9 | Cycle 10 | Cycle 11 | Cycle 12 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| i1 | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active | Not active | Not active |
| i2 | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active | Not active |
| i3 | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active | Not active |
| i4 | Not active | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active | Not active |
| i5 | Not active | Not active | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active | Not active |
| i6 | Not active | Not active | Not active | Not active | Not active | IF | ID | EX | MEM | WB | Not active | Not active |
| i7 | Not active | Not active | Not active | Not active | Not active | Not active | IF | ID | EX | MEM | WB | Not active |
| i8 | Not active | Not active | Not active | Not active | Not active | Not active | Not active | IF | ID | EX | MEM | WB |
This run takes ⌈8 / 1⌉ + 5 − 1 = 12 cycles. Its average is 8 / 12 = 0.67 instructions per cycle, including filling and draining. With a long stream of independent work, the rate approaches 1 per cycle. More stages increase latency in cycles; width sets the ideal steady-state rate.
The schedules are calculated from the ideal model above. They contain no data dependencies, cache misses, branches or resource conflicts. This establishes a capacity baseline before the next lesson asks why real instruction streams sometimes cannot fill it.
Out-of-order execution: finding ready work
More width helps only when there is ready work to use it. Suppose instruction 2 waits for instruction 1, while instruction 3 has independent operands. An out-of-order scheduler may run instruction 3 first. It still has to preserve the program's required results, exceptions and memory ordering.
Execute when ready, retire in order
A conceptual out-of-order core. Instructions enter the issue queue and reorder buffer at dispatch; execution later reports completion to the buffer.
Renaming and queueing let execution proceed when operands and units are ready. The ROB records completion; retirement makes architectural state permanent in program order. Branch recovery discards younger work when a prediction is wrong. Speculative work can still affect microarchitectural state such as caches.
The blocks divide that responsibility:
- Register renaming assigns physical destinations to register writes, separating successive uses of the same architectural name.
- Issue queues hold operations until their operands are ready and a suitable execution unit is available.
- The reorder buffer (ROB) tracks instructions in program order so completed work can commit in order, including precise exception handling.
- Load/store queues track pending memory operations and possible address conflicts. Register independence alone does not prove two memory accesses are independent.
The BOOM pipeline documentation provides a concrete RISC-V implementation of these stages. Its ROB documentation distinguishes execution completion from architectural commit, and its load/store unit documentation explains store-to-load forwarding and recovery from ordering mistakes. The diagram here is an architectural illustration, not a simulation of BOOM.
A different scheduling choice: VLIW
Very Long Instruction Word designs expose groups of operations for the compiler to schedule together. A teaching bundle might contain independent arithmetic, load and store slots. This shifts scheduling work toward the compiler; it does not remove dependencies or make every slot useful. Code size, available parallelism and compatibility across implementations remain tradeoffs. MIT's architecture course treats VLIW as a separate scheduling approach.
From capacity to stalls
The ideal schedules answer how much overlap is possible. They do not yet answer when an operand is ready, which instruction wins a shared resource, or what happens when the next address depends on a branch. Those are the questions for Hazard Detection, the next lesson.
Power provides a separate limit: even useful parallel hardware must fit an operating budget. Dennard Scaling explains that context without being a prerequisite for the dependency walkthrough.
Further reading
- Patterson & Hennessy: Computer Organization and Design
- Shen & Lipasti: Modern Processor Design
- Intel / AMD / ARM optimization guides
- Agner Fog's optimization manuals
Related concepts
Deep dive into CPU cache lines — interactive cache simulator with configurable associativity and replacement policies, false sharing MESI protocol visualization, access pattern benchmarks, and optimization techniques.
Replay branch outcomes to compare predictors, then connect local and global history to speculative execution, recovery and profiling.
Explore how smaller transistors promised more speed at the same power, why voltage limits broke that bargain, and how chip design adapted.
Trace data, structural and control hazards, understand forwarding and load-use stalls, and connect register dependencies to scheduling.
Master sequential vs strided memory access patterns. Learn how cache efficiency and hardware prefetching affect application performance.
Discover how memory interleaving distributes addresses across banks for parallel access. Boost memory bandwidth in DDR5 and GPU systems.
