Hi there 👋

I am Lei Mao, a ML engineer specializing in AI inference optimization and architecture design. Also focused on market analysis in the inference and computing power.

The KV Cache Dilemma: Why LLM Inference Needs to "Forget" to Scale?

Have you ever wondered why the 10th turn of a conversation with an LLM feels just as fast as the first? Mathematically, this shouldn’t happen. As the context grows (History + New Question), the computation required to generate the next token should increase linearly. The secret sauce is KV Cache (Key-Value Cache). It is the single most critical component in LLM Inference optimization. ...