[GH-ISSUE #2693] [Bug]: Uneven multi-GPU transfer performance under heavy data fragmentation #700

New issue

Closed

opened 2026-10-29 21:18:23 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:18:23 +00:00

Owner

Originally created by @c-guo16 on GitHub (Jul 1, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2693

Bug Report

Internode GDR transfer performance tests reveal significant discrepancies in Mooncake transfer speeds across different GPUs under small message sizes and large batch sizes.

Our test case is <message size 4096, batch size 160000>, performance is shown bellow:

With a single QP, slow GPUs achieve approximately 0.9 GB/s bandwidth, while fast GPUs reach around 2.7 GB/s.

The fast/slow GPU performance pattern varies drastically, yet it is consistently tied to the receiver side. Pinning the receiver to a specific topology or switching to a different receiver host alters the fast/slow GPU pattern. Modifications on the sender side barely affect this pattern at all.

Whether the receiver is topology-pinned or not changes the fast/slow GPU pattern. However, pinning to an incorrect topology yields identical performance patterns as pinning to the correct affinity topology. We suspect this fast/slow GPU imbalance stems from Mooncake filtering out unrelated network adapters during topology binding.

We notice that this imbalance occurs on different version of Mooncake, including the newest v0.3.11.post1.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @c-guo16 on GitHub (Jul 1, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2693 ### Bug Report Internode GDR transfer performance tests reveal significant discrepancies in Mooncake transfer speeds across different GPUs under small message sizes and large batch sizes. Our test case is <message size 4096, batch size 160000>, performance is shown bellow: <img width="840" height="334" alt="Image" src="https://github.com/user-attachments/assets/997f2569-c1e0-4244-a793-bd0978009cf4" /> With a single QP, slow GPUs achieve approximately 0.9 GB/s bandwidth, while fast GPUs reach around 2.7 GB/s. The fast/slow GPU performance pattern varies drastically, yet it is consistently tied to the receiver side. Pinning the receiver to a specific topology or switching to a different receiver host alters the fast/slow GPU pattern. Modifications on the sender side barely affect this pattern at all. Whether the receiver is topology-pinned or not changes the fast/slow GPU pattern. However, pinning to an incorrect topology yields identical performance patterns as pinning to the correct affinity topology. We suspect this fast/slow GPU imbalance stems from Mooncake filtering out unrelated network adapters during topology binding. We notice that this imbalance occurs on different version of Mooncake, including the newest v0.3.11.post1. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:18:23 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:18:24 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 1, 2026):

Thanks for opening this issue, @c-guo16!

Field	Value
Issue	#2693
GitHub user ID	`35829535`
Reporter	@c-guo16

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 1, 2026): Thanks for opening this issue, @c-guo16! | Field | Value | |-------|-------| | **Issue** | #2693 | | **GitHub user ID** | `35829535` | | **Reporter** | @c-guo16 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:30:03 +00:00

[PR #700] [MERGED] refactor(store): move client buffer implementation to store module #1754