[PR #27] [fix](kt): Fix barrier call to use cpu_group instead of device_group #28

New issue

Open

opened 2026-10-29 21:06:36 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:06:36 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/27
Author: @SCDESPERTATE
Created: 3/12/2026
Status: 🔄 Open

Base: main ← Head: fix/layerwise_prefill_barrier_usage

📝 Commits (1)

00bdcd7 fix: change the device_group to cpu_group in barrier call

📊 Changes

1 file changed (+7 additions, -11 deletions)

View changed files

📝 python/sglang/srt/layers/moe/kt_ep_wrapper.py (+7 -11)

📄 Description

Summary

The comment in the GroupCoordinator.barrier method in parallel_state.py explicitly states: "don't use device_group here." However, the current implementation frequently uses device_group in dist.barrier calls, which introduces unnecessary latency during the expert transfer process in layerwise-prefill. This latency scales with both the number of layers and the number of experts per layer, degrading performance.

Profling

On a system with Quadro RTX 5000x1 and PyTorch 2.7.1, using device_group in dist.barrier adds around 75 microseconds of latency per expert transfer compare with using cpu_group. For a model like Qwen3-30B-A3B (128 experts per layer, 48 layers), this accumulates to nearly 460.8ms of additional overhead.

Modifications

Replace the explicit dist.barrier(group=get_tp_group().device_group) call with the already wrapped GroupCoordinator.barrier method which use cpu_group inside.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/27 **Author:** [@SCDESPERTATE](https://github.com/SCDESPERTATE) **Created:** 3/12/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/layerwise_prefill_barrier_usage` --- ### 📝 Commits (1) - [`00bdcd7`](https://github.com/kvcache-ai/sglang/commit/00bdcd7889c297e9664d1eee09468fb6b5f9d918) [fix](kt_ep_wrapper): change the device_group to cpu_group in `barrier` call ### 📊 Changes **1 file changed** (+7 additions, -11 deletions) <details> <summary>View changed files</summary> 📝 `python/sglang/srt/layers/moe/kt_ep_wrapper.py` (+7 -11) </details> ### 📄 Description ## Summary The comment in the `GroupCoordinator.barrier` method in `parallel_state.py` explicitly states: "don't use device_group here." However, the current implementation frequently uses `device_group` in `dist.barrier` calls, which introduces unnecessary latency during the expert transfer process in layerwise-prefill. This latency scales with both the number of layers and the number of experts per layer, degrading performance. ### Profling On a system with `Quadro RTX 5000`x1 and `PyTorch 2.7.1`, using `device_group` in `dist.barrier` adds around 75 microseconds of latency per expert transfer compare with using `cpu_group`. For a model like Qwen3-30B-A3B (128 experts per layer, 48 layers), this accumulates to nearly 460.8ms of additional overhead. ### Modifications Replace the explicit `dist.barrier(group=get_tp_group().device_group)` call with the already wrapped `GroupCoordinator.barrier` method which use `cpu_group` inside. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 21:06:36 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/sglang#28

No description provided.

Rows
Columns