- JavaScript 56.3%
- HTML 24.1%
- Python 8.5%
- C++ 4.9%
- CSS 3.2%
- Other 2.9%
7d449d1fe1* KV Cache Calculator: recurrent state precision for every linear-attention model The recurrent-state precision selector applied to the Qwen formula only; the two Kimi KDA formulas fixed the recurrent state at 4 bytes per element. Engines make this a choice: SGLang --mamba-ssm-dtype (bf16 is required for FlashInfer GDN decode on SM100+), vLLM --mamba-ssm-cache-dtype. - both KDA formulas read the selected precision profile; the model default stays fp32, or bf16 when kda_recurrent_state_bytes_per_element says 2 - the control shows for every model with a linear-attention state; the help text names both engine flags; a "KDA recurrent-state precision" row joins the result components - Python port: calculate_cache_size(recurrent_state_precision=...) with the same defaults and labels; the Qwen convolution term now counts kernel - 1 past inputs as the web calculator does - tests: Kimi K3 checkpoint 449,372,160 B at fp32 and 232,316,928 B at bf16 in both suites; GLM-5.3-Flash follows the same selector; Qwen3.8 state bytes match the web values Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * KV Cache Calculator: recurrent-state tooltip only on its ? button Point the label at the select so hovering the label or dropdown no longer opens the help, and shorten the help text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|---|---|---|
| assets | ||
| config/_default | ||
| content | ||
| data | ||
| hugo-blox | ||
| images | ||
| layouts | ||
| packages/kvcache-simulator | ||
| scripts | ||
| static | ||
| tests | ||
| wasm/kvcache-sim | ||
| .editorconfig | ||
| .gitattributes | ||
| .gitignore | ||
| .hugo_build.lock | ||
| go.mod | ||
| go.sum | ||
| hugoblox.yaml | ||
| LICENSE.md | ||
| netlify.toml | ||
| README.md | ||
| test.md | ||
| theme.toml | ||
Official website source for KVCache.AI — the home of open-source projects and research on KV cache management and LLM serving optimization.
About KVCache.AI
KVCache.AI advances the state of the art in Large Language Model (LLM) inference optimization. In decoder-only Transformer models, data from diverse modalities can ultimately be transformed into KV cache, making it a central component of modern LLM serving systems. As a result, KV cache has become a key focus for improving inference efficiency through techniques such as caching, scheduling, compression, offloading, and disaggregated serving architectures.
Through open-source projects and academic research, KVCache.ai develops effective, practical, and high-performance solutions for KV cache management and LLM serving optimization. The goal is to make LLM deployment more accessible, efficient, and cost-effective for organizations of all sizes.
Featured Projects
- Mooncake — A KV cache-centric disaggregated architecture for LLM serving.
- KTransformers — A CPU/GPU heterogeneous LLM inference and fine-tuning framework for running and tuning 100B+ models on accessible workstation hardware.
- AgentENV — An open-source runtime platform designed for AI Agent applications.
Tools
- KV Cache Size Calculator — Estimate KV cache capacity for common production LLM families, including DeepSeek, GLM, Kimi, Qwen, MiniMax, MiMo, Dots3, Inkling and others.
- KV Cache Hit Rate Simulator — Calculate KV cache hit rate of preset or your own trace, under different memory budgets.
Tech Stack
This site is built with:
- Hugo (v0.126.3) — static site generator
- Hugo Blox — theme and page builder (Tailwind CSS)
- Pagefind — static search (used in production builds on Netlify)
Content is written in Markdown with YAML front matter. Custom layouts and shortcodes live under layouts/.
Links
- Website: https://kvcache.ai
- GitHub Organization: https://github.com/kvcache-ai
- X (Twitter): @KVCache_AI