[PR #73] [MERGED] fix(dsv4): lazily allocate layerwise prefill slots #72

New issue

Closed

opened 2026-10-29 21:06:57 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:06:57 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/73
Author: @yyj6666667
Created: 8/5/2026
Status: ✅ Merged
Merged: 8/6/2026
Merged by: @yyj6666667

Base: main ← Head: feat/dsv4-lazy-layerwise-prefill

📝 Commits (1)

91c9513 fix(dsv4): lazily allocate layerwise prefill slots

📊 Changes

5 files changed (+296 additions, -72 deletions)

View changed files

📝 python/sglang/srt/layers/moe/kt_ep_wrapper.py (+119 -12)
📝 python/sglang/srt/layers/quantization/v4_marlin_moe.py (+31 -0)
📝 python/sglang/srt/model_executor/memory_profiler.py (+12 -1)
📝 python/sglang/srt/model_executor/model_runner.py (+7 -13)
📝 test/unit/test_mxfp4_layerwise_prefill.py (+127 -46)

📄 Description

Summary

Defer MXFP4 layerwise-prefill raw/prepared slot allocation until the first request that reaches the configured GPU-prefill threshold.
Reserve the corresponding capacity during profiling without allocating the slot tensors at server startup.
Keep a coordinated fallback to hybrid CPU/GPU inference if lazy initialization cannot fit on any TP rank.
Add unit coverage for initialization, capacity accounting, retry/fallback, and request-trigger behavior.

Validation

python3 test/unit/test_mxfp4_layerwise_prefill.py (27 tests)
End-to-end DeepSeek-V4-Flash runs with KT_GPU_PREFILL_TOKEN_THRESHOLD=2048 on qj5090 and sap4.

Scope

This PR intentionally does not include the independent CPU-group barrier cleanup in #27.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/73 **Author:** [@yyj6666667](https://github.com/yyj6666667) **Created:** 8/5/2026 **Status:** ✅ Merged **Merged:** 8/6/2026 **Merged by:** [@yyj6666667](https://github.com/yyj6666667) **Base:** `main` ← **Head:** `feat/dsv4-lazy-layerwise-prefill` --- ### 📝 Commits (1) - [`91c9513`](https://github.com/kvcache-ai/sglang/commit/91c95138743b31ab1bca8568e6d1285530797ed3) fix(dsv4): lazily allocate layerwise prefill slots ### 📊 Changes **5 files changed** (+296 additions, -72 deletions) <details> <summary>View changed files</summary> 📝 `python/sglang/srt/layers/moe/kt_ep_wrapper.py` (+119 -12) 📝 `python/sglang/srt/layers/quantization/v4_marlin_moe.py` (+31 -0) 📝 `python/sglang/srt/model_executor/memory_profiler.py` (+12 -1) 📝 `python/sglang/srt/model_executor/model_runner.py` (+7 -13) 📝 `test/unit/test_mxfp4_layerwise_prefill.py` (+127 -46) </details> ### 📄 Description ## Summary - Defer MXFP4 layerwise-prefill raw/prepared slot allocation until the first request that reaches the configured GPU-prefill threshold. - Reserve the corresponding capacity during profiling without allocating the slot tensors at server startup. - Keep a coordinated fallback to hybrid CPU/GPU inference if lazy initialization cannot fit on any TP rank. - Add unit coverage for initialization, capacity accounting, retry/fallback, and request-trigger behavior. ## Validation - `python3 test/unit/test_mxfp4_layerwise_prefill.py` (27 tests) - End-to-end DeepSeek-V4-Flash runs with `KT_GPU_PREFILL_TOKEN_THRESHOLD=2048` on qj5090 and sap4. ## Scope This PR intentionally does not include the independent CPU-group barrier cleanup in #27. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:06:57 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/sglang#72

No description provided.

Rows
Columns