[PR #49] [MERGED] KV Cache Calculator: recurrent state precision for every linear-attention model #49

New issue

Closed

opened 2026-10-29 14:32:14 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:32:14 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/kvcache-blog/pull/49
Author: @Venkat2811
Created: 9/25/2026
Status: ✅ Merged
Merged: 10/20/2026
Merged by: @ykwd

Base: main ← Head: feat/recurrent-state-precision-all-linear

📝 Commits (2)

bd146dc KV Cache Calculator: recurrent state precision for every linear-attention model
ca33c9f KV Cache Calculator: recurrent-state tooltip only on its ? button

📊 Changes

5 files changed (+161 additions, -24 deletions)

View changed files

📝 assets/js/kv-cache-calculator.js (+58 -18)
📝 layouts/shortcodes/kv-cache-calculator.html (+3 -3)
📝 packages/kvcache-simulator/src/kvcache_sim/calculator.py (+51 -3)
📝 packages/kvcache-simulator/tests/test_kvcache_sim.py (+25 -0)
📝 tests/kv-cache-calculator.test.mjs (+24 -0)

📄 Description

Summary

The recurrent-state precision selector applies to the Qwen formula only; the two Kimi KDA formulas fix the recurrent state at 4 bytes per element. Engines make this a choice: SGLang --mamba-ssm-dtype (bf16 is required for FlashInfer GDN decode on SM100+), vLLM --mamba-ssm-cache-dtype.

Changes

both KDA formulas read the selected precision profile; the model default stays fp32, or bf16 when kda_recurrent_state_bytes_per_element says 2
the control shows for every model with a linear-attention state; the help text names both engine flags; a "KDA recurrent-state precision" row joins the result components
Python port: calculate_cache_size(recurrent_state_precision=...) with the same defaults and labels; its Qwen convolution term now counts kernel - 1 past inputs, as the web calculator does (it counted the full kernel)
no catalog change; default numbers are unchanged

Validation

node --test tests/*.test.mjs: 156 of 157, the failing catalog test predates this change; new tests: Kimi K3 checkpoint 449,372,160 B at fp32 and 232,316,928 B at bf16, GLM-5.3-Flash follows the selector
pytest packages/kvcache-simulator: 26 passed, including Qwen3.8 state bytes equal to the web values (74.8125 MiB bf16, 146.8125 MiB fp32)
Hugo build not run here; only the calculator JS, its shortcode help text and the package changed

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/kvcache-blog/pull/49 **Author:** [@Venkat2811](https://github.com/Venkat2811) **Created:** 9/25/2026 **Status:** ✅ Merged **Merged:** 10/20/2026 **Merged by:** [@ykwd](https://github.com/ykwd) **Base:** `main` ← **Head:** `feat/recurrent-state-precision-all-linear` --- ### 📝 Commits (2) - [`bd146dc`](https://github.com/kvcache-ai/kvcache-blog/commit/bd146dc9d7894a859f36222ff78a0952183eeaf5) KV Cache Calculator: recurrent state precision for every linear-attention model - [`ca33c9f`](https://github.com/kvcache-ai/kvcache-blog/commit/ca33c9f39918889306f2f3c8c1be21a150ae2dda) KV Cache Calculator: recurrent-state tooltip only on its ? button ### 📊 Changes **5 files changed** (+161 additions, -24 deletions) <details> <summary>View changed files</summary> 📝 `assets/js/kv-cache-calculator.js` (+58 -18) 📝 `layouts/shortcodes/kv-cache-calculator.html` (+3 -3) 📝 `packages/kvcache-simulator/src/kvcache_sim/calculator.py` (+51 -3) 📝 `packages/kvcache-simulator/tests/test_kvcache_sim.py` (+25 -0) 📝 `tests/kv-cache-calculator.test.mjs` (+24 -0) </details> ### 📄 Description ## Summary The recurrent-state precision selector applies to the Qwen formula only; the two Kimi KDA formulas fix the recurrent state at 4 bytes per element. Engines make this a choice: SGLang `--mamba-ssm-dtype` (bf16 is required for FlashInfer GDN decode on SM100+), vLLM `--mamba-ssm-cache-dtype`. ## Changes - both KDA formulas read the selected precision profile; the model default stays fp32, or bf16 when `kda_recurrent_state_bytes_per_element` says 2 - the control shows for every model with a linear-attention state; the help text names both engine flags; a "KDA recurrent-state precision" row joins the result components - Python port: `calculate_cache_size(recurrent_state_precision=...)` with the same defaults and labels; its Qwen convolution term now counts kernel - 1 past inputs, as the web calculator does (it counted the full kernel) - no catalog change; default numbers are unchanged ## Validation - `node --test tests/*.test.mjs`: 156 of 157, the failing catalog test predates this change; new tests: Kimi K3 checkpoint 449,372,160 B at fp32 and 232,316,928 B at bf16, GLM-5.3-Flash follows the selector - `pytest packages/kvcache-simulator`: 26 passed, including Qwen3.8 state bytes equal to the web values (74.8125 MiB bf16, 146.8125 MiB fp32) - Hugo build not run here; only the calculator JS, its shortcode help text and the package changed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:32:14 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/kvcache-blog#49

No description provided.

Rows
Columns