Attention Sinks: Stable Streaming LLMs
See how preserving initial attention sinks stabilizes a rolling KV cache, why it costs recent slots, and why stable streaming does not mean infinite recall.
7 min readConcept
Explore machine learning concepts related to Inference. Clear explanations and practical insights.
See how preserving initial attention sinks stabilizes a rolling KV cache, why it costs recent slots, and why stable streaming does not mean infinite recall.
Trace a cached decode step, separate saved projections from growing attention work, and calculate KV-cache memory with explicit dimensions.
A small draft model proposes tokens, the target model checks them all in one pass, and a rejection rule keeps the output distribution exactly the target's.