kvcache-ai/sglang

Fork 0

mirror of https://github.com/kvcache-ai/sglang.git synced 2026-11-05 01:07:39 +00:00

[PR #95] [MERGED] fix(kv-cache): guard compressed pool initialization for non-SWA models #95

New issue

Closed

opened 2026-10-29 21:07:09 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:07:09 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/95
Author: @yyj6666667
Created: 9/11/2026
Status: ✅ Merged
Merged: 9/11/2026
Merged by: @yyj6666667

Base: main ← Head: fix/guard-compressed-kv-pool-init

📝 Commits (2)

4f22278 fix(kv-cache): guard compressed pool initialization for non-SWA models
defb49b chore: remove test changes from KV guard PR

📊 Changes

1 file changed (+2 additions, -1 deletions)

View changed files

📝 python/sglang/srt/model_executor/model_runner_kv_cache_mixin.py (+2 -1)

📄 Description

Summary

Scope the compressed-attention capacity branch introduced by #83 to hybrid-SWA models. Kimi and other non-SWA models retain the original profile_max_num_token() path, without accessing an inapplicable compressed-attention flag.

Why

#83 added a direct read of model_config.is_swa_with_compressed_attention at the shared KV-pool initialization entry point. Historically, _derive_hybrid_model() returned early for non-SWA models before defining that field. This caused Kimi's initialization to fail with AttributeError, independently of the compressed-tensors package.

Current main already initializes the flags to False in 4afb5b541. This PR preserves that fix and additionally guards the consumer with is_hybrid_swa, so correctness does not rely solely on the new default. It does not claim that current main still reproduces the historical missing-field error.

Changes and scope

Add model_config.is_hybrid_swa as the first short-circuit condition before reading is_swa_with_compressed_attention.
Preserve #83's configured capacity seed for DeepSeek V4 / NextN when hybrid SWA is active and --max-total-tokens is supplied.
Preserve generic profiling for non-SWA models, other hybrid models, disabled hybrid memory, and DeepSeek V4 without an explicit token limit.
No test-file changes are included; the existing test file has been restored exactly to the base revision.

Only one runtime condition changes. No Kimi whitelist, model/config-file modifications, kernel changes, dependency/version changes, tokenizer changes, or unrelated experimental patches are included. The later GPU-memory profiling and capacity clamping code is untouched.

Validation

python -m pytest -q -c /dev/null test/registered/unit/test_model_config_hybrid_flags.py
# 12 passed

The unchanged 12 hybrid-configuration tests pass after the cleanup. Before removing the additional test changes, 42 CPU cases passed, including the capacity-selection branches and legacy configurations without the compressed-attention field. An in-memory negative control failed the 12 missing-field cases without the guard. Those additional tests are not part of the final PR diff.

Ruff (F401,F821), isort, git diff --check, and Black for the changed runtime lines passed. Existing formatting elsewhere is untouched.

Validation boundary: these are CPU configuration/branch regressions, not GPU inference or Adapter end-to-end acceptance. This PR addresses the KV initialization regression only; it does not claim to resolve every Kimi loading issue. No PyPI release is part of this PR.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/95 **Author:** [@yyj6666667](https://github.com/yyj6666667) **Created:** 9/11/2026 **Status:** ✅ Merged **Merged:** 9/11/2026 **Merged by:** [@yyj6666667](https://github.com/yyj6666667) **Base:** `main` ← **Head:** `fix/guard-compressed-kv-pool-init` --- ### 📝 Commits (2) - [`4f22278`](https://github.com/kvcache-ai/sglang/commit/4f2227805b7fc8d58b04cf11e6d550cc79d11f59) fix(kv-cache): guard compressed pool initialization for non-SWA models - [`defb49b`](https://github.com/kvcache-ai/sglang/commit/defb49bde1d27f6bedca747c59eabe671de20a5e) chore: remove test changes from KV guard PR ### 📊 Changes **1 file changed** (+2 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `python/sglang/srt/model_executor/model_runner_kv_cache_mixin.py` (+2 -1) </details> ### 📄 Description ## Summary Scope the compressed-attention capacity branch introduced by #83 to hybrid-SWA models. Kimi and other non-SWA models retain the original `profile_max_num_token()` path, without accessing an inapplicable compressed-attention flag. ## Why #83 added a direct read of `model_config.is_swa_with_compressed_attention` at the shared KV-pool initialization entry point. Historically, `_derive_hybrid_model()` returned early for non-SWA models before defining that field. This caused Kimi's initialization to fail with `AttributeError`, independently of the `compressed-tensors` package. Current main already initializes the flags to `False` in `4afb5b541`. This PR preserves that fix and additionally guards the consumer with `is_hybrid_swa`, so correctness does not rely solely on the new default. It does not claim that current main still reproduces the historical missing-field error. ## Changes and scope - Add `model_config.is_hybrid_swa` as the first short-circuit condition before reading `is_swa_with_compressed_attention`. - Preserve #83's configured capacity seed for DeepSeek V4 / NextN when hybrid SWA is active and `--max-total-tokens` is supplied. - Preserve generic profiling for non-SWA models, other hybrid models, disabled hybrid memory, and DeepSeek V4 without an explicit token limit. - No test-file changes are included; the existing test file has been restored exactly to the base revision. Only one runtime condition changes. No Kimi whitelist, model/config-file modifications, kernel changes, dependency/version changes, tokenizer changes, or unrelated experimental patches are included. The later GPU-memory profiling and capacity clamping code is untouched. ## Validation ```bash python -m pytest -q -c /dev/null test/registered/unit/test_model_config_hybrid_flags.py # 12 passed ``` The unchanged 12 hybrid-configuration tests pass after the cleanup. Before removing the additional test changes, 42 CPU cases passed, including the capacity-selection branches and legacy configurations without the compressed-attention field. An in-memory negative control failed the 12 missing-field cases without the guard. Those additional tests are not part of the final PR diff. Ruff (`F401,F821`), isort, `git diff --check`, and Black for the changed runtime lines passed. Existing formatting elsewhere is untouched. **Validation boundary:** these are CPU configuration/branch regressions, not GPU inference or Adapter end-to-end acceptance. This PR addresses the KV initialization regression only; it does not claim to resolve every Kimi loading issue. No PyPI release is part of this PR. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:07:09 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/sglang#95

No description provided.

Rows
Columns