Lei Mao | AI Inference & Markethttps://maomao123321.github.io/Recent content on Lei Mao | AI Inference & MarketHugo -- 0.146.0enSun, 07 Dec 2025 13:06:53 -0500The KV Cache Dilemma: Why LLM Inference Needs to "Forget" to Scale?https://maomao123321.github.io/inference/kv-cache-dilemma/Sun, 07 Dec 2025 13:06:53 -0500https://maomao123321.github.io/inference/kv-cache-dilemma/<p>Have you ever wondered why the 10th turn of a conversation with an LLM feels just as fast as the first? Mathematically, this shouldn&rsquo;t happen. As the context grows (History + New Question), the computation required to generate the next token should increase linearly. The secret sauce is KV Cache (Key-Value Cache). It is the single most critical component in LLM Inference optimization.</p><link>https://maomao123321.github.io/about/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://maomao123321.github.io/about/</guid><description/></item></channel></rss>