[PR #2] [CLOSED] Support different tp_size - Part 1 p/d dynamic connection #3

New issue

Closed

opened 2026-10-29 21:06:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:06:23 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/2
Author: @yuan-luo
Created: 4/7/2025
Status: ❌ Closed

Base: mooncake_transfer_engine ← Head: sgl_different_tp_size

📝 Commits (5)

4f305bf Support bootstrap server rpc with prefill and decode
ac3e904 Refactor prefill and decode to support dynamic rank ports
53aa51d Support bootstrap server rpc with prefill and decode
0664822 Address review comments
0f33b49 Refactor prefill/decode bootstrap register

📊 Changes

3 files changed (+193 additions, -32 deletions)

View changed files

📝 python/sglang/srt/disaggregation/conn.py (+191 -32)
📝 python/sglang/srt/disaggregation/decode.py (+1 -0)
📝 python/sglang/srt/disaggregation/prefill.py (+1 -0)

📄 Description

Motivation

To support different tensor parallel size on prefill and decode. This PR is Part 1, supporting prefill / decode connection establishment and register to bootstrap server.

Modifications

Bootstrap server support REGISTER http route service with prefill and decode
KVManager add connection pool cache
Refactor decode's connection establishment with bootstrap server
Refactor prefill's connection establishment with bootstrap server

Checklist

Format your code according to the Code Formatting with Pre-Commit.
Add unit tests as outlined in the Running Unit Tests.
Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/2 **Author:** [@yuan-luo](https://github.com/yuan-luo) **Created:** 4/7/2025 **Status:** ❌ Closed **Base:** `mooncake_transfer_engine` ← **Head:** `sgl_different_tp_size` --- ### 📝 Commits (5) - [`4f305bf`](https://github.com/kvcache-ai/sglang/commit/4f305bfe528ccaeb3c45f12c4d49f7d644e8fcc7) Support bootstrap server rpc with prefill and decode - [`ac3e904`](https://github.com/kvcache-ai/sglang/commit/ac3e904cbb720872c3f19b744b184c35129e8051) Refactor prefill and decode to support dynamic rank ports - [`53aa51d`](https://github.com/kvcache-ai/sglang/commit/53aa51d68733eb062899a99eac11f3f18fd59279) Support bootstrap server rpc with prefill and decode - [`0664822`](https://github.com/kvcache-ai/sglang/commit/066482295e2071cffaca598e7acb453235b16937) Address review comments - [`0f33b49`](https://github.com/kvcache-ai/sglang/commit/0f33b49b90b181ca028125d759107fbb3c031239) Refactor prefill/decode bootstrap register ### 📊 Changes **3 files changed** (+193 additions, -32 deletions) <details> <summary>View changed files</summary> 📝 `python/sglang/srt/disaggregation/conn.py` (+191 -32) 📝 `python/sglang/srt/disaggregation/decode.py` (+1 -0) 📝 `python/sglang/srt/disaggregation/prefill.py` (+1 -0) </details> ### 📄 Description  ## Motivation To support different tensor parallel size on prefill and decode. This PR is Part 1, supporting prefill / decode connection establishment and register to bootstrap server.  ## Modifications 1. Bootstrap server support REGISTER http route service with prefill and decode 2. KVManager add connection pool cache 3. Refactor decode's connection establishment with bootstrap server 4. Refactor prefill's connection establishment with bootstrap server  ## Checklist - [ ] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/references/contribution_guide.html#code-formatting-with-pre-commit). - [ ] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/references/contribution_guide.html#running-unit-tests-adding-to-ci). - [ ] Update documentation / docstrings / example tutorials as needed, according to [Writing Documentation](https://docs.sglang.ai/references/contribution_guide.html#writing-documentation-running-docs-ci). - [ ] Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to [Benchmark and Profiling](https://docs.sglang.ai/references/benchmark_and_profiling.html) and [Accuracy Results](https://docs.sglang.ai/references/accuracy_evaluation.html). - [ ] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR. - [ ] Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:06:23 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/sglang#3

No description provided.

Rows
Columns