kvcache-ai/sglang

Fork 0

mirror of https://github.com/kvcache-ai/sglang.git synced 2026-11-05 01:07:39 +00:00

[PR #91] kt: plumb --kt-mmap-experts-dir through to the KT MoE wrapper #93

New issue

Open

opened 2026-10-29 21:07:08 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:07:08 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/91
Author: @tim-odonnell
Created: 9/7/2026
Status: 🔄 Open

Base: main ← Head: kt-mmap-experts-dir

📝 Commits (1)

f8bf78b kt: plumb --kt-mmap-experts-dir through to the KT MoE wrapper

📊 Changes

2 files changed (+15 additions, -0 deletions)

View changed files

📝 python/sglang/srt/layers/moe/kt_ep_wrapper.py (+3 -0)
📝 python/sglang/srt/server_args.py (+12 -0)

📄 Description

What

Companion to kvcache-ai/ktransformers#2190 — plumbs --kt-mmap-experts-dir
through to the KT MoE wrapper.

When set, the kt-kernel CPU MoE kernels keep each CPU-resident expert's BufferB
weight block in a MAP_SHARED file under this directory instead of anonymous RAM,
so a host whose CPU expert working set is close to total RAM can serve the model
(cold experts reclaim without swap, refault from the file). Unset (default) →
unchanged anonymous behaviour.

How

server_args.py: --kt-mmap-experts-dir / ServerArgs.kt_mmap_experts_dir.
kt_ep_wrapper.py: carried on KTConfig, passed to the inference
KTMoEWrapper (the SFT wrapper path is unchanged).

Requires the matching kt-kernel change (KTMoEWrapper mmap_experts_dir kwarg →
GeneralMOEConfig.mmap_weights_dir, kvcache-ai/ktransformers#2190). A kt_kernel
build without it simply ignores the flag.

Testing

End-to-end with kvcache-ai/ktransformers#2190 on 1× CMP-170HX (sm_80) + 128 GB
DDR5, DeepSeek-V4-Flash-0731 native MXFP4/FP8: --kt-mmap-experts-dir flows to
KTConfig → KTMoEWrapper → MOEConfig.mmap_weights_dir; scheduler process
anonymous RSS drops from ~105 GiB to ~2.4 GiB; decode throughput, VRAM, and
greedy outputs unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KW36fR7syvcTaG7Yqqq8pP

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/91 **Author:** [@tim-odonnell](https://github.com/tim-odonnell) **Created:** 9/7/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `kt-mmap-experts-dir` --- ### 📝 Commits (1) - [`f8bf78b`](https://github.com/kvcache-ai/sglang/commit/f8bf78b57d673fdbcf453bef489719bdf2ecd382) kt: plumb --kt-mmap-experts-dir through to the KT MoE wrapper ### 📊 Changes **2 files changed** (+15 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `python/sglang/srt/layers/moe/kt_ep_wrapper.py` (+3 -0) 📝 `python/sglang/srt/server_args.py` (+12 -0) </details> ### 📄 Description ## What Companion to **kvcache-ai/ktransformers#2190** — plumbs `--kt-mmap-experts-dir` through to the KT MoE wrapper. When set, the kt-kernel CPU MoE kernels keep each CPU-resident expert's `BufferB` weight block in a `MAP_SHARED` file under this directory instead of anonymous RAM, so a host whose CPU expert working set is close to total RAM can serve the model (cold experts reclaim without swap, refault from the file). Unset (default) → unchanged anonymous behaviour. ## How - `server_args.py`: `--kt-mmap-experts-dir` / `ServerArgs.kt_mmap_experts_dir`. - `kt_ep_wrapper.py`: carried on `KTConfig`, passed to the inference `KTMoEWrapper` (the SFT wrapper path is unchanged). Requires the matching kt-kernel change (`KTMoEWrapper` `mmap_experts_dir` kwarg → `GeneralMOEConfig.mmap_weights_dir`, kvcache-ai/ktransformers#2190). A `kt_kernel` build without it simply ignores the flag. ## Testing End-to-end with kvcache-ai/ktransformers#2190 on 1× CMP-170HX (sm_80) + 128 GB DDR5, DeepSeek-V4-Flash-0731 native MXFP4/FP8: `--kt-mmap-experts-dir` flows to `KTConfig` → `KTMoEWrapper` → `MOEConfig.mmap_weights_dir`; scheduler process anonymous RSS drops from ~105 GiB to ~2.4 GiB; decode throughput, VRAM, and greedy outputs unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KW36fR7syvcTaG7Yqqq8pP --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 21:07:08 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/sglang#93

No description provided.

Rows
Columns