Skip to main content

Quantization Effects Simulator

Embedding quantization simulator: explore memory-accuracy trade-offs from float32 to int8 and binary representations for retrieval.

A float32 embedding index is mostly empty precision. Ten million 768-dim vectors take ~30 GB at full precision; store them as int8 and that drops to ~7.5 GB with almost no loss in search quality. The only question worth asking is the one the simulator answers: how much recall does each bit of compression actually cost?

Interactive Quantization Simulator

The simulator is real — it quantizes genuine GloVe vectors live and re-runs nearest-neighbor search at each precision, scoring recall@10 against the float32 baseline. Watch the neighbor list survive int8 intact and start to scramble at binary.

Scalar quantization

The workhorse is scalar quantization: map each floating-point dimension to a small integer. With a per-vector symmetric scheme, the scale is set by the largest magnitude in the vector, so the value range maps onto the available integer levels:

s = \frac{\max_i |x_i|}{2^{\,b-1} - 1}, \qquad \hat{x}_i = \mathrm{round}\!\left(\frac{x_i}{s}\right) \cdot s

with b the bit width (8 for int8, 4 for int4). Binary is the degenerate case — keep only the sign, \hat{x}_i = \mathrm{sign}(x_i).

python
def quantize(v, bits):
    if bits == 1:                                 # binary: sign only
        return np.sign(v)
    qmax = 2 ** (bits - 1) - 1                     # 127 for int8, 7 for int4
    s = np.abs(v).max() / qmax                     # per-vector scale
    q = np.clip(np.round(v / s), -qmax - 1, qmax)  # to integers
    return q * s                                   # dequantize for scoring

Memory follows directly from the bit width — a d-dimensional vector costs d \cdot b / 8 bytes, so int8 is 4× smaller than float32 and binary is 32× smaller.

The precision ladder

The metric that matters for retrieval is not reconstruction error but recall@k — do the quantized nearest neighbors still match the float32 ones?

\text{recall@}k = \frac{\lvert\, \mathrm{NN}_k^{\text{quant}}(q) \,\cap\, \mathrm{NN}_k^{\text{float32}}(q) \,\rvert}{k}

Measured live on the GloVe vectors in the simulator above:

PrecisionBits/dimCompressionBytes/vec (50d)recall@10
float32321×2001.00
int884×50≈ 0.99
int448×25≈ 0.92
binary132×6.25≈ 0.61

int8 is nearly free — a 4× saving for a fraction of a percent of recall. The drop steepens fast after that: binary buys another 8× over int8 but surrenders a third of the recall.

int8 or binary?

int8 (scalar)
binary (sign)
Bits per dimension
8
1
Compression
4×
32×
Recall@10 (this demo)
≈ 99%
≈ 61%
Distance
Integer dot product
Hamming (XOR + popcount)
Role
Drop-in index replacement
First-stage filter, then re-rank

The standard production recipe is int8 as a near-lossless default, and binary only as a coarse first pass — retrieve a large candidate set by fast Hamming distance, then re-rank those few with int8 or float32 vectors.

Beyond scalar quantization

Scalar quantization treats every dimension independently. Two refinements go further:

  • Product quantization splits the vector into sub-vectors and replaces each with the nearest entry in a learned codebook — far higher compression than scalar at the same recall. It gets its own treatment in vector quantization.
  • Binary embeddings lean into the 1-bit extreme, training the model so the sign pattern itself is informative and search runs on Hamming distance. See binary embeddings.

Best practices

  1. Default to int8 — it is near-lossless and supported everywhere; reach for lower precision only when memory is the binding constraint.
  2. Measure recall, not MSE — reconstruction error can look fine while neighbor rankings quietly degrade.
  3. Two-stage with binary — use 1-bit vectors to fetch candidates fast, then re-rank with higher precision rather than trusting binary scores directly.
  4. Calibrate the scale — clip extreme outliers (e.g. at the 99.9th percentile) so they do not stretch the scale and waste levels on the bulk of the distribution.

References

  • Jégou et al. "Product Quantization for Nearest Neighbor Search"
  • Shakir et al. "Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval"
  • Dettmers et al. "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale"

If you found this explanation helpful, consider sharing it with others.

Mastodon