[PR #83] [MERGED] fix(dsv4): honor --max-total-tokens for the compressed-attention pool #82

New issue

Closed

opened 2026-10-29 21:07:04 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:07:04 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/83
Author: @05yuki
Created: 8/27/2026
Status: ✅ Merged
Merged: 9/10/2026
Merged by: @yyj6666667

Base: main ← Head: fix/dsv4-compressed-kv-pool-sizing

📝 Commits (1)

206600e fix(dsv4): honor --max-total-tokens for the compressed-attention pool

📊 Changes

1 file changed (+15 additions, -1 deletions)

View changed files

📝 python/sglang/srt/model_executor/model_runner_kv_cache_mixin.py (+15 -1)

📄 Description

For DeepSeek V4, init_memory_pool() seeds self.max_total_num_tokens with
profile_max_num_token(), which estimates capacity from the generic KV
cell size. set_num_tokens_hybrid_swa_compress() then profiles the real
compressed layout via DSv4MemoryCalculator, but when --max-total-tokens
was passed explicitly it also treats the already-seeded
self.max_total_num_tokens as a ceiling and falls back to
get_pool_sizes_by_configuration() with it.

The ceiling therefore comes from a layout the model never uses. On a
2x16 GB SM120 setup with KT CPU experts, a configured 1M-token context
was clamped to roughly 143K tokens even though the compressed pool fits
the full 1M.

Seed the ceiling with the configured value instead when the model uses
compressed attention and --max-total-tokens was given. The dedicated
calculator still profiles available VRAM and still reduces the pool when
it does not fit, and the configuration path is only ever taken to shrink
below the profiled size, so a large configured value cannot inflate the
pool past what profiling allows.

Runs that do not pass --max-total-tokens are unaffected: the ceiling is
never consulted there, and profile_max_num_token() keeps seeding the
value that max_running_requests is derived from.

Verified on DeepSeek-V4-Flash MXFP4: full_max_total_num_tokens reaches
1,048,576 with mem_fraction_static 0.985 and decoding stays coherent.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/83 **Author:** [@05yuki](https://github.com/05yuki) **Created:** 8/27/2026 **Status:** ✅ Merged **Merged:** 9/10/2026 **Merged by:** [@yyj6666667](https://github.com/yyj6666667) **Base:** `main` ← **Head:** `fix/dsv4-compressed-kv-pool-sizing` --- ### 📝 Commits (1) - [`206600e`](https://github.com/kvcache-ai/sglang/commit/206600e9addd49dfbc428d47876f143a8613aafb) fix(dsv4): honor --max-total-tokens for the compressed-attention pool ### 📊 Changes **1 file changed** (+15 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `python/sglang/srt/model_executor/model_runner_kv_cache_mixin.py` (+15 -1) </details> ### 📄 Description For DeepSeek V4, init_memory_pool() seeds self.max_total_num_tokens with profile_max_num_token(), which estimates capacity from the generic KV cell size. set_num_tokens_hybrid_swa_compress() then profiles the real compressed layout via DSv4MemoryCalculator, but when --max-total-tokens was passed explicitly it also treats the already-seeded self.max_total_num_tokens as a ceiling and falls back to get_pool_sizes_by_configuration() with it. The ceiling therefore comes from a layout the model never uses. On a 2x16 GB SM120 setup with KT CPU experts, a configured 1M-token context was clamped to roughly 143K tokens even though the compressed pool fits the full 1M. Seed the ceiling with the configured value instead when the model uses compressed attention and --max-total-tokens was given. The dedicated calculator still profiles available VRAM and still reduces the pool when it does not fit, and the configuration path is only ever taken to shrink below the profiled size, so a large configured value cannot inflate the pool past what profiling allows. Runs that do not pass --max-total-tokens are unaffected: the ceiling is never consulted there, and profile_max_num_token() keeps seeding the value that max_running_requests is derived from. Verified on DeepSeek-V4-Flash MXFP4: full_max_total_num_tokens reaches 1,048,576 with mem_fraction_static 0.985 and decoding stays coherent. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:07:04 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/sglang#82

No description provided.

Rows
Columns