Understanding num_workers

The GPU is fine. The queue isn’t.

You raised batch size, mixed precision is on, the kernel looks healthy — and utilization still dips between steps. The gap is often not FLOPs. It is who loads the next batch while the current one trains.

num_workers is how many subprocess loaders DataLoader keeps ahead of the training loop. Zero means the main process loads, then trains, then loads. Greater than zero means other processes fill a prefetch queue so main mostly trains.

Two numbers

1. Sync path · GPU busy ≈ 50% of the window
With num_workers=0, every step is load-then-train on the critical path. The device waits for host work half the time (or more if decode is heavy).

2. Sweet spot · often ~4 on one GPU, then plateaus
Two workers start real overlap. Four is the usual first good setting. Eight and sixteen buy RAM and scheduler noise long before they buy another 2×.

Flip the tapes until those claims feel mechanical.

Pipeline: who works each step

Same three roles — main, workers, GPU. Change n and watch the hollow gaps fill.

DataLoader(
    dataset,
    batch_size=32,
    num_workers=4,
    prefetch_factor=2,       # batches buffered per worker
    persistent_workers=True, # keep pool across epochs
)
# queue depth ≈ num_workers × prefetch_factor

Throughput vs workers

The curve climbs, flattens, then can fall. Memory keeps climbing either way.

import time
from torch.utils.data import DataLoader

for nw in [0, 2, 4, 8, 16]:
    loader = DataLoader(dataset, batch_size=32, num_workers=nw)
    t0 = time.time()
    for _ in loader:
        pass
    print(f"num_workers={nw}: {time.time() - t0:.2f}s")

What workers cost

Same knob. Three different taxes.

Dataset shape matters

# Bad: 10 GB in __init__ → × num_workers copies after fork/spawn
class BadDataset(Dataset):
    def __init__(self):
        self.data = load_huge_file()

# Good: paths only; load in __getitem__
class GoodDataset(Dataset):
    def __init__(self):
        self.paths = list_paths()

    def __getitem__(self, idx):
        return load_file(self.paths[idx])

Never call .cuda() inside __getitem__ when num_workers > 0 — CUDA contexts belong in the main process after collation.

On Windows, start under if __name__ == "__main__": (spawn re-imports your module). Prefer persistent_workers=True so you do not pay that startup every epoch.

What to do

1. Start at 4 on a single GPU; use 0 only for debugging or tiny CPU runs.
2. Sweep 0 → 2 → 4 → 8 with a pure-iteration timer; pick the elbow.
3. Lazy datasets — paths in __init__, load in __getitem__.
4. persistent_workers=True when num_workers > 0 and you train many epochs.
5. Pair with pin_memory=True and .to(device, non_blocking=True) so H2D does not re-serialize the path you just opened.

train_loader = DataLoader(
    train_dataset,
    batch_size=64,
    num_workers=4,
    pin_memory=True,
    persistent_workers=True,
    prefetch_factor=2,
)

for batch in train_loader:
    x = batch['images'].to('cuda', non_blocking=True)
    y = batch['labels'].to('cuda', non_blocking=True)
    ...

Further Reading

• PyTorch DataLoader — workers, prefetch, persistent workers
• PyTorch Performance Tuning Guide — data pipeline knobs
• Related: pin_memory — H2D path once the queue is full

Related concepts