Search Results
2 results for “KV cache”
Infrastructure
KV Cache
A KV cache (key-value cache) is a memory optimisation used in transformer inference that stores pre-computed key and value tensors from the attention mechanism, eliminating redundant recomputation when generating tokens sequentially.
6 min readUpdated June 2026
Infrastructure
Prompt Caching
Prompt caching is an inference optimisation technique that stores precomputed key-value representations of repeated prompt prefixes, reducing latency and token processing costs for applications with stable system prompts or long shared contexts.
6 min readUpdated June 2026