AIStackInsights

The KV Cache Is a Database. You're Probably Treating It Like a Buffer.

Every modern inference engine has a database hiding inside it — a content-addressable, multi-tenant, hierarchically-tiered store with eviction policies, locality of reference, and a 100× cost gap between hits and misses. Most teams ignore it. The ones who don't run inference at a fraction of the cost.

May 10, 202619 min read

inference kv-cache paged-attention

Large Language Models

Speculative Decoding in Production: How a 1B Draft Model Cuts 70B Latency by 3-5×

The largest single inference speedup of the last three years is also the most invisible to application developers. A small draft model proposes tokens; a big model verifies them in parallel; the math guarantees the output distribution is unchanged. Here is how it actually works — and why your stack probably has it on already.

April 29, 202618 min read

inference speculative-decoding llm-serving

Tag: sglang

The KV Cache Is a Database. You're Probably Treating It Like a Buffer.

Speculative Decoding in Production: How a 1B Draft Model Cuts 70B Latency by 3-5×