kvcache-ai/sglang

Fork 0

mirror of https://github.com/kvcache-ai/sglang.git synced 2026-11-05 01:07:39 +00:00

[PR #67] [CLOSED] [Elastic EP] Fix recovery lifecycle and add manual coverage #70

New issue

Closed

opened 2026-10-29 21:06:55 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:06:55 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/67
Author: @UNIDY2002
Created: 7/20/2026
Status: ❌ Closed

Base: main ← Head: sunxun/recovery-fix

📝 Commits (10+)

205a2f2 Move LoRA cuda-graph buffers and logging into LoRAManager (#31151)
caa85ea Extract init_torch_distributed and refactor into functions (#31152)
0f20f52 Introduce RemoteInstanceWeightTransporter component (#31153)
d15f6a9 Introduce NgramEmbeddingManager component (#31154)
5b540b1 Extract load_model helpers into a load_model_utils module (#31155)
39e508b Extract layer-index setup into a module (#31156)
17c0460 Extract spec aux-hidden-state resolution into a module (#31157)
440aebd Extract small single-function helpers into modules (#31158)
08798db Extract MoE/EP setup into a moe_ep_setup module (#31159)
1dc48c2 Absorb capturer setup and extract the shared-mooncake gate (#31160)

📊 Changes

6548 files changed (+1274413 additions, -248942 deletions)

View changed files

➕ .claude/rules/forward-batch-init-new-purity.md (+17 -0)
➕ .claude/rules/general-code-style.md (+19 -0)
➕ .claude/rules/modify-component-must-read.md (+9 -0)
➕ .claude/rules/no-dataclasses.md (+23 -0)
➕ .claude/rules/no-getattr-defensive.md (+39 -0)
➕ .claude/rules/schedule-batch-out-of-place-mutation.md (+30 -0)
➕ .claude/rules/unit-test-admission.md (+72 -0)
📝 .claude/skills/add-jit-kernel/SKILL.md (+225 -129)
📝 .claude/skills/add-sgl-kernel/SKILL.md (+46 -37)
➕ .claude/skills/ci-workflow-guide/SKILL.md (+406 -0)
➕ .claude/skills/clean-startup-log/SKILL.md (+282 -0)
➕ .claude/skills/cookbook-add-model/SKILL.md (+226 -0)
➕ .claude/skills/cookbook-add-model/references/authoring-reference.md (+278 -0)
➕ .claude/skills/cookbook-add-model/references/engine-axis.md (+274 -0)
➕ .claude/skills/cookbook-add-model/references/mintlify-authoring.md (+116 -0)
➕ .claude/skills/cookbook-add-model/references/vendor-logo.md (+76 -0)
➕ .claude/skills/cookbook-add-model/templates/benchmarks.jsx.tmpl (+32 -0)
➕ .claude/skills/cookbook-add-model/templates/config.jsx.tmpl (+407 -0)
➕ .claude/skills/cookbook-add-model/templates/page.mdx.tmpl (+155 -0)
➕ .claude/skills/cookbook-migrate-model/SKILL.md (+240 -0)

...and 80 more files

📄 Description

Motivation

An initialization lifecycle refactor moved recover-joiner work away from the
ordering required by Elastic EP recovery. The recovery path must complete local
memory-pool setup, attention-backend initialization, warmup, and CUDA graph
capture before joining the recovered process groups. This PR restores that
ordering and adds a manual 4+4 recovery test that exercises the full path.

CC @yoray @zack: the changes preserve the scale-up path. Please run the
scale-up validation if you would like to confirm it against your workflow.

Modifications

Run recover-joiner process-group admission after CUDA graph capture and the
post-capture KV-pool resize.
Complete the recovery-side broadcasts and treat recover workers as joiners
during launch-time setup.
Add test/manual/ep/test_elastic_recover.py, an 8-GPU single-host 4+4
recovery test. It verifies initial serving, degraded serving after killing
one logical node, four-rank recovery, and post-recovery serving. The test
keeps CUDA graph enabled and uses a common explicit random seed for the
initial service and recover joiner.

Accuracy Tests

Validated on one H20 host with 8 GPUs, Mooncake Elastic EP, CUDA graph, and
RDMA. The manual test completed initial serving, degraded serving, recovery of
ranks [4, 5, 6, 7], and three post-recovery requests successfully:

python -m pytest test/manual/ep/test_elastic_recover.py::TestElasticRecover4To4 -v -s
1 passed in 423.97s

Speed Tests and Profiling

This change does not add work to the steady-state inference path. CUDA graph
capture remains enabled in the recovery validation.

Checklist

Format code according to the project code style.
Add a manual recovery test.
Update documentation according to Write documentations.
Provide recovery validation with CUDA graph enabled.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/67 **Author:** [@UNIDY2002](https://github.com/UNIDY2002) **Created:** 7/20/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `sunxun/recovery-fix` --- ### 📝 Commits (10+) - [`205a2f2`](https://github.com/kvcache-ai/sglang/commit/205a2f2de41eccc35929242586a9e8663a1ee734) Move LoRA cuda-graph buffers and logging into LoRAManager (#31151) - [`caa85ea`](https://github.com/kvcache-ai/sglang/commit/caa85ea022dc58c89ae5782f1f606d70ae55954d) Extract init_torch_distributed and refactor into functions (#31152) - [`0f20f52`](https://github.com/kvcache-ai/sglang/commit/0f20f52e5e0f5067791af38aac6062596fa1fafc) Introduce RemoteInstanceWeightTransporter component (#31153) - [`d15f6a9`](https://github.com/kvcache-ai/sglang/commit/d15f6a9ac33a2878a53390976fb1a62286899e76) Introduce NgramEmbeddingManager component (#31154) - [`5b540b1`](https://github.com/kvcache-ai/sglang/commit/5b540b16de8eb33dd24ad46c056b2c35640efe18) Extract load_model helpers into a load_model_utils module (#31155) - [`39e508b`](https://github.com/kvcache-ai/sglang/commit/39e508b7fc30bf1792360a8dc9de12431d57f278) Extract layer-index setup into a module (#31156) - [`17c0460`](https://github.com/kvcache-ai/sglang/commit/17c04602c645c71d22a81651818e2935ea4d3f5c) Extract spec aux-hidden-state resolution into a module (#31157) - [`440aebd`](https://github.com/kvcache-ai/sglang/commit/440aebdfe0f650bf61f9fb5c7a6570c37e2effea) Extract small single-function helpers into modules (#31158) - [`08798db`](https://github.com/kvcache-ai/sglang/commit/08798dba0d8129d595e16b9132b42bc96ab69508) Extract MoE/EP setup into a moe_ep_setup module (#31159) - [`1dc48c2`](https://github.com/kvcache-ai/sglang/commit/1dc48c2c3bd345e3caa72befac3d8ffeadc1c5ab) Absorb capturer setup and extract the shared-mooncake gate (#31160) ### 📊 Changes **6548 files changed** (+1274413 additions, -248942 deletions) <details> <summary>View changed files</summary> ➕ `.claude/rules/forward-batch-init-new-purity.md` (+17 -0) ➕ `.claude/rules/general-code-style.md` (+19 -0) ➕ `.claude/rules/modify-component-must-read.md` (+9 -0) ➕ `.claude/rules/no-dataclasses.md` (+23 -0) ➕ `.claude/rules/no-getattr-defensive.md` (+39 -0) ➕ `.claude/rules/schedule-batch-out-of-place-mutation.md` (+30 -0) ➕ `.claude/rules/unit-test-admission.md` (+72 -0) 📝 `.claude/skills/add-jit-kernel/SKILL.md` (+225 -129) 📝 `.claude/skills/add-sgl-kernel/SKILL.md` (+46 -37) ➕ `.claude/skills/ci-workflow-guide/SKILL.md` (+406 -0) ➕ `.claude/skills/clean-startup-log/SKILL.md` (+282 -0) ➕ `.claude/skills/cookbook-add-model/SKILL.md` (+226 -0) ➕ `.claude/skills/cookbook-add-model/references/authoring-reference.md` (+278 -0) ➕ `.claude/skills/cookbook-add-model/references/engine-axis.md` (+274 -0) ➕ `.claude/skills/cookbook-add-model/references/mintlify-authoring.md` (+116 -0) ➕ `.claude/skills/cookbook-add-model/references/vendor-logo.md` (+76 -0) ➕ `.claude/skills/cookbook-add-model/templates/benchmarks.jsx.tmpl` (+32 -0) ➕ `.claude/skills/cookbook-add-model/templates/config.jsx.tmpl` (+407 -0) ➕ `.claude/skills/cookbook-add-model/templates/page.mdx.tmpl` (+155 -0) ➕ `.claude/skills/cookbook-migrate-model/SKILL.md` (+240 -0) _...and 80 more files_ </details> ### 📄 Description ## Motivation An initialization lifecycle refactor moved recover-joiner work away from the ordering required by Elastic EP recovery. The recovery path must complete local memory-pool setup, attention-backend initialization, warmup, and CUDA graph capture before joining the recovered process groups. This PR restores that ordering and adds a manual 4+4 recovery test that exercises the full path. CC @yoray @zack: the changes preserve the scale-up path. Please run the scale-up validation if you would like to confirm it against your workflow. ## Modifications - Run recover-joiner process-group admission after CUDA graph capture and the post-capture KV-pool resize. - Complete the recovery-side broadcasts and treat recover workers as joiners during launch-time setup. - Add `test/manual/ep/test_elastic_recover.py`, an 8-GPU single-host 4+4 recovery test. It verifies initial serving, degraded serving after killing one logical node, four-rank recovery, and post-recovery serving. The test keeps CUDA graph enabled and uses a common explicit random seed for the initial service and recover joiner. ## Accuracy Tests Validated on one H20 host with 8 GPUs, Mooncake Elastic EP, CUDA graph, and RDMA. The manual test completed initial serving, degraded serving, recovery of ranks `[4, 5, 6, 7]`, and three post-recovery requests successfully: ```text python -m pytest test/manual/ep/test_elastic_recover.py::TestElasticRecover4To4 -v -s 1 passed in 423.97s ``` ## Speed Tests and Profiling This change does not add work to the steady-state inference path. CUDA graph capture remains enabled in the recovery validation. ## Checklist - [ ] Format code according to the project code style. - [x] Add a manual recovery test. - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide recovery validation with CUDA graph enabled. - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:06:55 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/sglang#70

No description provided.

Rows
Columns