Understanding num_workers

The GPU is fine. The queue isn’t.

You raised batch size, mixed precision is on, the kernel looks healthy — and utilization still dips between steps. The gap is often not FLOPs. It is who loads the next batch while the current one trains.

num_workers is how many subprocess loaders DataLoader keeps ahead of the training loop. Zero means the main process loads, then trains, then loads. Greater than zero means other processes fill a prefetch queue so main mostly trains.

Two numbers

1. Sync path · GPU busy ≈ 50% of the window
With num_workers=0, every step is load-then-train on the critical path. The device waits for host work half the time (or more if decode is heavy).

2. Sweet spot · often ~4 on one GPU, then plateaus
Two workers start real overlap. Four is the usual first good setting. Eight and sixteen buy RAM and scheduler noise long before they buy another 2×.

Flip the tapes until those claims feel mechanical.

Pipeline: who works each step

Same three roles — main, workers, GPU. Change n and watch the hollow gaps fill.

Worker pipeline

num_workers

Step 1 / 12

50%

GPU busy in this authored schedule

main load

t0 · nw=0

Who works each step · amber = load on main (GPU waits)

main

workers

GPU

worker_pipeline.py

loader = DataLoader(ds, num_workers=0)for batch in loader:          # load on main    train(batch)              # GPU waits first

Main process loads the next batch. GPU has nothing — this is the hollow half of the step.

num_workers=0 loads on the main process. Every batch pays serial load→train. The GPU tape stays half empty.

Processes, not threads

Python’s GIL would serialize heavy transforms in a thread pool. DataLoader workers are separate interpreters via multiprocessing. That is why num_workers > 0 actually parallelizes decode and augmentation.

python

DataLoader(
    dataset,
    batch_size=32,
    num_workers=4,
    prefetch_factor=2,       # batches buffered per worker
    persistent_workers=True, # keep pool across epochs
)
# queue depth ≈ num_workers × prefetch_factor

Throughput vs workers

The curve climbs, flattens, then can fall. Memory keeps climbing either way.

Throughput vs workers

num_workers

3.5×

iterate speed vs num_workers=0

~3.2× RAM

worker footprint (rough)

climb / sweet starve / overhead

n=4 — Usual first try on a single GPU: queue stays full, RAM still sane.

Start at 4 on one GPU, sweep 0→2→4→8 while timing pure iteration (no train). Stop when the curve flattens — more workers past that mostly buy RAM and scheduler noise.

How to measure your own curve

Time pure iteration — no model — while sweeping workers. Stop when elapsed time stops dropping.

python

import time
from torch.utils.data import DataLoader

for nw in [0, 2, 4, 8, 16]:
    loader = DataLoader(dataset, batch_size=32, num_workers=nw)
    t0 = time.time()
    for _ in loader:
        pass
    print(f"num_workers={nw}: {time.time() - t0:.2f}s")

What workers cost

Same knob. Three different taxes.

What workers cost

cost topic

Tax

Dataset in __init__

Each worker process re-holds what you loaded at construct time. 10 GB × 4 workers ≈ 40 GB.

×N copies

Fix

Lazy __getitem__

Keep paths/handles light. Load the sample when asked. Workers share the recipe, not the bytes.

Workers buy parallel load. They also buy process copies, startup, and RAM. The wrong Dataset shape turns that into a silent memory bomb.

Dataset shape matters

python

# Bad: 10 GB in __init__ → × num_workers copies after fork/spawn
class BadDataset(Dataset):
    def __init__(self):
        self.data = load_huge_file()

# Good: paths only; load in __getitem__
class GoodDataset(Dataset):
    def __init__(self):
        self.paths = list_paths()

    def __getitem__(self, idx):
        return load_file(self.paths[idx])

Never call .cuda() inside __getitem__ when num_workers > 0 — CUDA contexts belong in the main process after collation.

On Windows, start under if __name__ == "__main__": (spawn re-imports your module). Prefer persistent_workers=True so you do not pay that startup every epoch.

What to do

Start at 4 on a single GPU; use 0 only for debugging or tiny CPU runs.
Sweep 0 → 2 → 4 → 8 with a pure-iteration timer; pick the elbow.
Lazy datasets — paths in __init__, load in __getitem__.
persistent_workers=True when num_workers > 0 and you train many epochs.
Pair with pin_memory=True and .to(device, non_blocking=True) so H2D does not re-serialize the path you just opened.

python

train_loader = DataLoader(
    train_dataset,
    batch_size=64,
    num_workers=4,
    pin_memory=True,
    persistent_workers=True,
    prefetch_factor=2,
)

for batch in train_loader:
    x = batch['images'].to('cuda', non_blocking=True)
    y = batch['labels'].to('cuda', non_blocking=True)
    ...