[GH-ISSUE #1933] Pipeline Parallel for ktransformers for multi low-vram GPUs #1200

New issue

Open

opened 2026-10-29 14:49:52 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:49:52 +00:00

Owner

Originally created by @NukePeace on GitHub (Apr 16, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1933

Reminder

I have read the above rules and searched the existing issues.

Description

Looks like the pipeline parallel for sglang is not well implemented and we are a bit away from sglang main branch, I think we can use well-maintained vllm integrating with sglang parameter to have literal full control over this entire project.

Pull Request

No response

Originally created by @NukePeace on GitHub (Apr 16, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1933 ### Reminder - [x] I have read the above rules and searched the existing issues. ### Description Looks like the pipeline parallel for sglang is not well implemented and we are a bit away from sglang main branch, I think we can use well-maintained vllm integrating with sglang parameter to have literal full control over this entire project. ### Pull Request _No response_

bsakaguchi added the

enhancement

label

2026-10-29 14:49:52 +00:00

bsakaguchi commented

2026-10-29 14:49:52 +00:00

Author

Owner

@ErvinXie commented on GitHub (Apr 17, 2026):

Thanks for the suggestion. We agree that pipeline parallelism is a valuable technique, especially for multi low-VRAM GPU setups, and that SGLang's current PP support is not sufficient for our needs.

However, implementing proper PP on top of SGLang would require a significant amount of work. We may consider exploring this direction in the future, but it's not on the near-term roadmap at this point.

This reply was co-authored by an AI agent and a human.

@ErvinXie commented on GitHub (Apr 17, 2026): Thanks for the suggestion. We agree that pipeline parallelism is a valuable technique, especially for multi low-VRAM GPU setups, and that SGLang's current PP support is not sufficient for our needs. However, implementing proper PP on top of SGLang would require a significant amount of work. We may consider exploring this direction in the future, but it's not on the near-term roadmap at this point. --- *This reply was co-authored by an AI agent and a human.*