Skip to main content

Tensor Cores: Mixed Precision & Matrix Acceleration

How NVIDIA Tensor Cores accelerate GEMM: warp-level MMA, precision dials (TF32/FP16/BF16/INT8/FP8), shape alignment and padding cost, AMP vs model.half(), and when the speedup actually shows up.

The matmul is fine. The util is not.

You enabled AMP. Loss looks stable. Step time barely moved. The profiler still shows a fat GEMM — but not the Tensor Core kernel you expected. Dimensions are “almost” multiples of 8. Or someone called model.half() six months ago and nobody re-checked.

Tensor Cores are not “CUDA cores but faster.” They are a separate unit that eats warp-level matrix multiply-accumulate when precision, shape, layout, and library kernel all agree. Unsupported operations or kernels can use CUDA cores; dimension alignment in modern libraries is an efficiency recommendation, not a universal eligibility gate.

This page is those gates — not a catalog of every WMMA fragment shape.

Two numbers

1. ~16× dense peak (FP16/BF16 vs FP32 on A100)
Marketing math for aligned, compute-bound GEMMs. Your layer may be memory-bound and never touch it.

2. Alignment affects efficiency

cuBLAS 11.0+ and cuDNN 7.6.3+ can use Tensor Cores without the older dimension restrictions. Multiples still improve efficiency; A100 guidance favors 64 FP16 or 128 INT8 elements in the relevant contiguous dimensions. K=4095 alone does not establish fallback. NVIDIA’s matrix multiplication guide documents the version and layout scope.

Same GEMM, two engines

CUDA cores do scalar and vector MACs. Tensor Cores do tile MMA: one warp, one fragment product, far fewer cycles for the same math when the path is open.

CUDA cores vs Tensor Cores
16×
FP16 dense peak vs FP32 CUDA cores · A100 teaching model

Same job: a small GEMM tile. CUDA cores do scalar MACs; Tensor Cores swallow a warp-level matrix fragment per instruction.

CUDA cores · scalar MAC0 / 4096 cycles
Tensor Cores · tile MMA0 / 256

Press Race — simplified teaching model, not a profiler number.

Tensor Cores are not faster CUDA cores. They are a different unit for D = A·B + C on warp fragments.

Precision is a dial

Lower width → higher peak math rate, tighter range or fewer mantissa bits. TF32 keeps FP32 storage but truncates mantissa in the multiply. BF16 keeps FP32 exponent range (training-friendly). INT8/FP8 are inference (or specialized training) after calibration.

Precision dial
312TFLOPS
dense peak · A100 · sparse 624 TFLOPS · 16×
Tensor Core
E5
M10
signexponentmantissa / int16-bit math

Half the memory of FP32. Narrow exponent → needs loss scaling to avoid gradient underflow.

Lower precision is not free accuracy. BF16 keeps FP32 range (good training default on Ampere/Hopper); FP16 needs loss scaling; INT8/FP8 need calibration.

Alignment and padding cost

Inspect an analytical padding calculation for the selected shapes. The bar is useful GEMM work divided by work at rounded dimensions; it predicts neither Tensor Core eligibility nor throughput. Benchmark edge handling versus padding with your library and layout.

Alignment and padding
100.00%useful work
analytical useful / padded GEMM work · multiple 64
M=4096 · N=4096 · K=4096
Rounded dimensions: 4096 × 4096 × 4096100.00%
M
4096
÷64
N
4096
÷64
K
4096
÷64

All three dimensions meet this A100 efficiency recommendation. This does not predict achieved throughput.

cuBLAS 11.0+ and cuDNN 7.6.3+ remove the old dimension eligibility restriction. These A100 recommendations use 128-byte dimension multiples (64 FP16/BF16 or 128 INT8 elements). The bar calculates padding overhead only; layouts, chosen kernels and edge handling determine actual performance.

Inside one Tensor Core op

Public APIs expose warp-collective steps (load → mma_sync → store). Fragment sizes vary by architecture and precision; the story is the same: 32 lanes cooperate on one tile.

One Tensor Core op
Step 1 / 5
0/4
warp steps · stage: ready
A frag
×
B frag
→
D = A·B+C
WMMA setup excerpt
// Include <mma.h> and <cuda_fp16.h>.
// Inside a kernel, all 32 lanes participate uniformly.
namespace wmma = nvcuda::wmma;
wmma::fragment<wmma::matrix_a, 16, 16, 16, half, wmma::row_major> a_frag;
wmma::fragment<wmma::matrix_b, 16, 16, 16, half, wmma::row_major> b_frag;
wmma::fragment<wmma::accumulator, 16, 16, 16, float> c_frag, d_frag;
wmma::fill_fragment(c_frag, 0.0f); // or load a valid C tile
// A/B/D: 256-bit aligned. lda/ldb: multiples of 8 half elements.
// ldd: multiple of 4 float elements. All leading dimensions >= 16.
// Pointer setup and bounds checks are omitted.
WMMA collective calls
wmma::load_matrix_sync(a_frag, A, lda);wmma::load_matrix_sync(b_frag, B, ldb);wmma::mma_sync(d_frag, a_frag, b_frag, c_frag);wmma::store_matrix_sync(D, d_frag, ldd, wmma::mem_row_major);

After declarations and C initialization below, inspect four warp-collective calls.

Real kernels pipeline loads under mma with cp.async / TMA so the cores never starve — the serial film is the mechanism, not the production schedule.

In production you rarely write WMMA by hand — cuBLAS, cuDNN, CUTLASS, and framework kernels do. The film is so you know what “Tensor Core path” means when the profiler names it.

How you actually turn them on

You do not flip a “Tensor Cores” switch. You put eligible math on a dtype and shape the hardware accelerates. In PyTorch that is almost always AMP, not a global .half().

AMP path
~16×
same peak class · no loss-scale dance
master w
FP32 copy
autocast
GEMM → BF16
Tensor Core
MMA fire
optimizer
plain step
# no GradScaler required for BF16
with autocast("cuda", dtype=torch.bfloat16):
loss = criterion(model(x), y)
loss.backward()
opt.step()

Ampere/Hopper default for many teams: FP32 range, fewer mantissa bits, Tensor Core path without GradScaler.

Prefer autocast + (optional) GradScaler over model.half(). Let the framework keep soft ops in FP32 while GEMMs hit Tensor Cores.

Enable mixed precision when:

  • You train or fine-tune large models on Volta or newer.
  • Matmul shapes can hit the alignment multiples (or padding is measured-cheap).
  • You can use BF16 on Ampere/Hopper — skip the GradScaler dance when numerics allow.

Stay in FP32 when:

  • The model is tiny and launch overhead dominates.
  • Numerics are still broken in full precision — fix that first.
  • Your accuracy requirement needs full FP32 inputs. Set tolerances using actual numerical error; TF32 has no universal 1-ULP output bound. Repeatability is separate: use deterministic algorithms and the documented cuBLAS workspace configuration when required. See PyTorch reproducibility and TF32 precision controls.
  • Hardware is pre-Volta.
python
import torch
from torch.amp import autocast, GradScaler

device = "cuda"
model = model.to(device)
# BF16 path — common Ampere/Hopper default
for batch in dataloader:
    optimizer.zero_grad(set_to_none=True)
    with autocast("cuda", dtype=torch.bfloat16):
        loss = criterion(model(batch["input"].to(device)), batch["target"].to(device))
    loss.backward()
    optimizer.step()

# FP16 needs GradScaler
scaler = GradScaler("cuda")
with autocast("cuda", dtype=torch.float16):
    loss = criterion(model(x), y)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()

Traps

Same four failure modes show up in every “AMP didn’t help” review.

Tensor Core traps
edge tiles
Almost-right shapes

K=4095 or odd channels. Modern libraries may still use Tensor Cores, but edge tiles and layout can cost efficiency.

align or pad on evidence

Check architecture-specific efficiency recommendations and the selected kernel. Pad only when profiler shows the cliff is real and padding is cheap.

Tensor Cores fire when precision, shape, layout, and kernel agree. The trap is assuming any one of those is enough.

What to do

  1. Start with BF16 autocast on Ampere/Hopper; use FP16 + GradScaler when you must.
  2. Align M, N, K (and channels) to the precision’s multiple — pad only with profiler evidence.
  3. Prefer library kernels (cuBLAS / cuDNN / framework) over hand WMMA unless you are writing a kernel library.
  4. Profile Tensor Core utilization — Nsight Compute or the PyTorch profiler — not just “AMP is imported.”
  5. Validate numerics after every precision or quantization change; never publish peak TFLOPS as a product SLA.

Further reading

GPU & High-Performance Computing
NVIDIA vs AMD for Deep Learning: CUDA vs ROCm and the Datacenter Accelerators

NVIDIA vs AMD for deep learning compared at both layers: the CUDA vs ROCm software moat, the microarchitecture (warp vs wavefront, SM vs CU, Tensor vs Matrix Cores), and the datacenter accelerators (H100/H200/B200 vs MI300X/MI325X).

GPU & High-Performance Computing
CUDA Contexts: Ownership, Current Stack, Isolation

Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.

GPU & High-Performance Computing
CUDA Context vs Streams vs MPS: Which Layer Fixes What

Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.

GPU & High-Performance Computing
CUDA Multi-Process Service (MPS): Concurrent Client Contexts

Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.

GPU & High-Performance Computing
CUDA Streams: Asynchronous Execution and Concurrency

A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the legacy default-stream fences between blocking streams in one context.

GPU & High-Performance Computing
GPU Pipeline Hazards: Scoreboard Stalls and Warp Divergence

A fine kernel can still profile poorly: warp divergence taxes throughput ~1/N; alone on a long scoreboard wait, SM issue util can hit 0%. Interactive demos.

If you found this explanation helpful, consider sharing it with others.

Mastodon