Quantization Effects Simulator

A float32 embedding index is mostly empty precision. Ten million 768-dim vectors take ~30 GB at full precision; store them as int8 and that drops to ~7.5 GB with almost no loss in search quality. The only question worth asking is the one the simulator answers: how much recall does each bit of compression actually cost?

Interactive Quantization Simulator

The simulator is real — it quantizes genuine GloVe vectors live and re-runs nearest-neighbor search at each precision, scoring recall@10 against the float32 baseline. Watch the neighbor list survive int8 intact and start to scramble at binary.

Scalar quantization

The workhorse is scalar quantization: map each floating-point dimension to a small integer. With a per-vector symmetric scheme, the scale is set by the largest magnitude in the vector, so the value range maps onto the available integer levels:

with b the bit width (8 for int8, 4 for int4). Binary is the degenerate case — keep only the sign, x̂_i = \mathrm{sign}(x_i).

def quantize(v, bits):
    if bits == 1:                                 # binary: sign only
        return np.sign(v)
    qmax = 2 ** (bits - 1) - 1                     # 127 for int8, 7 for int4
    s = np.abs(v).max() / qmax                     # per-vector scale
    q = np.clip(np.round(v / s), -qmax - 1, qmax)  # to integers
    return q * s                                   # dequantize for scoring

Memory follows directly from the bit width — a d-dimensional vector costs d · b / 8 bytes, so int8 is 4× smaller than float32 and binary is 32× smaller.

The precision ladder

The metric that matters for retrieval is not reconstruction error but recall@k — do the quantized nearest neighbors still match the float32 ones?

Measured live on the GloVe vectors in the simulator above:

int8 is nearly free — a 4× saving for a fraction of a percent of recall. The drop steepens fast after that: binary buys another 8× over int8 but surrenders a third of the recall.

int8 or binary?

The standard production recipe is int8 as a near-lossless default, and binary only as a coarse first pass — retrieve a large candidate set by fast Hamming distance, then re-rank those few with int8 or float32 vectors.

Beyond scalar quantization

Scalar quantization treats every dimension independently. Two refinements go further:

• Product quantization splits the vector into sub-vectors and replaces each with the nearest entry in a learned codebook — far higher compression than scalar at the same recall. It gets its own treatment in vector quantization.
• Binary embeddings lean into the 1-bit extreme, training the model so the sign pattern itself is informative and search runs on Hamming distance. See binary embeddings.

Best practices

1. Default to int8 — it is near-lossless and supported everywhere; reach for lower precision only when memory is the binding constraint.
2. Measure recall, not MSE — reconstruction error can look fine while neighbor rankings quietly degrade.
3. Two-stage with binary — use 1-bit vectors to fetch candidates fast, then re-rank with higher precision rather than trusting binary scores directly.
4. Calibrate the scale — clip extreme outliers (e.g. at the 99.9th percentile) so they do not stretch the scale and waste levels on the bulk of the distribution.

References

• Jégou et al. "Product Quantization for Nearest Neighbor Search"
• Shakir et al. "Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval"
• Dettmers et al. "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale"

Related concepts