[PR #64] feat(kt): add --kt-skip-gpu-expert-cpu-copy to reclaim host RAM for GPU experts #64

New issue

Open

opened 2026-10-29 21:06:53 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:06:53 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/64
Author: @gvr13n
Created: 7/12/2026
Status: 🔄 Open

Base: main ← Head: feat/kt-skip-gpu-expert-cpu-copy

📝 Commits (1)

d9a8b1c feat(kt): add --kt-skip-gpu-expert-cpu-copy to reclaim host RAM for GPU experts

📊 Changes

2 files changed (+29 additions, -0 deletions)

View changed files

📝 python/sglang/srt/layers/moe/kt_ep_wrapper.py (+19 -0)
📝 python/sglang/srt/server_args.py (+10 -0)

📄 Description

Companion to the kt-kernel PR (kvcache-ai/ktransformers#2085); see kvcache-ai/ktransformers#2084 for the full rationale and numbers.

Adds the user-facing --kt-skip-gpu-expert-cpu-copy server arg and resolves it into an effective flag passed to KTMoEWrapper:

_mask_static = method == "MXFP4" or (method == "MXFP8" and not kt_enable_dynamic_expert_update)
skip = kt_skip_gpu_expert_cpu_copy and _mask_static

Only paths whose full-GPU prefill fallback is mask-aware may skip: MXFP4's _prepare_weight_mxfp4 → _prepare_weight_fp8 copies GPU experts device-to-device from original_layer and never reads the CPU copies; MXFP4's dynamic-update mask rewrite is unconditionally skipped, so its mask is static even with --kt-enable-dynamic-expert-update on. If the flag cannot be honored (e.g. MXFP8 + dynamic update), it warns once at layer 0 and falls back to stock behavior — safe by construction.

Measured on DeepSeek-V4-Flash (MXFP4, 100–110 GPU experts, RTX PRO 6000 + 123GB RAM): scheduler RSS ~172GB → ~99GB, decode swap io → 0, identical output.

Diff is pure additions (+29, 2 files: server_args.py, kt_ep_wrapper.py), rebased on current main.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/64 **Author:** [@gvr13n](https://github.com/gvr13n) **Created:** 7/12/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/kt-skip-gpu-expert-cpu-copy` --- ### 📝 Commits (1) - [`d9a8b1c`](https://github.com/kvcache-ai/sglang/commit/d9a8b1c40ad3026fe938691f3eb382488322faa7) feat(kt): add --kt-skip-gpu-expert-cpu-copy to reclaim host RAM for GPU experts ### 📊 Changes **2 files changed** (+29 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `python/sglang/srt/layers/moe/kt_ep_wrapper.py` (+19 -0) 📝 `python/sglang/srt/server_args.py` (+10 -0) </details> ### 📄 Description Companion to the kt-kernel PR (kvcache-ai/ktransformers#2085); see kvcache-ai/ktransformers#2084 for the full rationale and numbers. Adds the user-facing `--kt-skip-gpu-expert-cpu-copy` server arg and resolves it into an effective flag passed to `KTMoEWrapper`: ``` _mask_static = method == "MXFP4" or (method == "MXFP8" and not kt_enable_dynamic_expert_update) skip = kt_skip_gpu_expert_cpu_copy and _mask_static ``` Only paths whose full-GPU prefill fallback is mask-aware may skip: MXFP4's `_prepare_weight_mxfp4` → `_prepare_weight_fp8` copies GPU experts device-to-device from `original_layer` and never reads the CPU copies; MXFP4's dynamic-update mask rewrite is unconditionally skipped, so its mask is static even with `--kt-enable-dynamic-expert-update` on. If the flag cannot be honored (e.g. MXFP8 + dynamic update), it warns once at layer 0 and falls back to stock behavior — safe by construction. Measured on DeepSeek-V4-Flash (MXFP4, 100–110 GPU experts, RTX PRO 6000 + 123GB RAM): scheduler RSS ~172GB → ~99GB, decode swap io → 0, identical output. Diff is pure additions (+29, 2 files: `server_args.py`, `kt_ep_wrapper.py`), rebased on current main. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 21:06:53 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/sglang#64

No description provided.

Rows
Columns