2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
How FlashAttention computes exact attention with linear memory by tiling Q, K, and V into SRAM-resident blocks and fusing the softmax, avoiding the quadratic HBM cost of materializing the full attention matrix.