[PR #499] [MERGED] add support for batch transfer to accelerate transfer operation #1605

New issue

Closed

opened 2026-10-29 21:28:51 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:28:51 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/499
Author: @ssssnow
Created: 6/16/2025
Status: ✅ Merged
Merged: 6/20/2025
Merged by: @ShangmingCai

Base: main ← Head: snow/add_batch_transfer

📝 Commits (7)

63e0a0a add support for batch transfer to accelerate transfer operation
35bc940 fix tcp port exhausted issue
8d3a63d add more info and fix double free
cbd748d rm unused freeBatchID
79f5b35 Merge branch 'main' into snow/add_batch_transfer
d21d566 [chore] add TODO comment for batchTransfer
49b3274 [Fix] reset 0 to transfer bytes

📊 Changes

6 files changed (+237 additions, -1 deletions)

View changed files

📝 mooncake-integration/transfer_engine/transfer_engine_py.cpp (+109 -1)
📝 mooncake-integration/transfer_engine/transfer_engine_py.h (+16 -0)
📝 mooncake-transfer-engine/include/multi_transport.h (+2 -0)
📝 mooncake-transfer-engine/include/transfer_engine.h (+12 -0)
📝 mooncake-transfer-engine/src/multi_transport.cpp (+35 -0)
📝 mooncake-wheel/tests/transfer_engine_initiator_test.py (+63 -0)

📄 Description

Add Support for Batch Transfer Interface

For transports such as RDMA, there is inherent overhead associated with initiating each transfer operation, which cannot be avoided. To achieve optimal performance—such as in MLA KVCache transfers—batch transfers can significantly outperform single-step transfers by reducing this overhead.

Below are the batch transfer speed results based on a 200*2Gbps RDMA setup:

Related test has been added to mooncake-wheel/tests/transfer_engine_initiator_test.py

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/499 **Author:** [@ssssnow](https://github.com/ssssnow) **Created:** 6/16/2025 **Status:** ✅ Merged **Merged:** 6/20/2025 **Merged by:** [@ShangmingCai](https://github.com/ShangmingCai) **Base:** `main` ← **Head:** `snow/add_batch_transfer` --- ### 📝 Commits (7) - [`63e0a0a`](https://github.com/kvcache-ai/Mooncake/commit/63e0a0a4c69c7f0a44e3abc4425db6d01af7a480) add support for batch transfer to accelerate transfer operation - [`35bc940`](https://github.com/kvcache-ai/Mooncake/commit/35bc940a3332e4de72cb03da6a299ce181fda873) fix tcp port exhausted issue - [`8d3a63d`](https://github.com/kvcache-ai/Mooncake/commit/8d3a63dd0999c6bae16941535c4f4beeb95ecf7e) add more info and fix double free - [`cbd748d`](https://github.com/kvcache-ai/Mooncake/commit/cbd748d241be57670b6733ec5adff2c584eebeb4) rm unused freeBatchID - [`79f5b35`](https://github.com/kvcache-ai/Mooncake/commit/79f5b354779947c2b922ef2bae6c669dc7862b5a) Merge branch 'main' into snow/add_batch_transfer - [`d21d566`](https://github.com/kvcache-ai/Mooncake/commit/d21d566b25bd33c0cce69d1bb8bdee76351e7470) [chore] add TODO comment for batchTransfer - [`49b3274`](https://github.com/kvcache-ai/Mooncake/commit/49b32748a42ba12407fee989bc164896f095a7f4) [Fix] reset 0 to transfer bytes ### 📊 Changes **6 files changed** (+237 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-integration/transfer_engine/transfer_engine_py.cpp` (+109 -1) 📝 `mooncake-integration/transfer_engine/transfer_engine_py.h` (+16 -0) 📝 `mooncake-transfer-engine/include/multi_transport.h` (+2 -0) 📝 `mooncake-transfer-engine/include/transfer_engine.h` (+12 -0) 📝 `mooncake-transfer-engine/src/multi_transport.cpp` (+35 -0) 📝 `mooncake-wheel/tests/transfer_engine_initiator_test.py` (+63 -0) </details> ### 📄 Description # Add Support for Batch Transfer Interface For transports such as RDMA, there is inherent overhead associated with initiating each transfer operation, which cannot be avoided. To achieve optimal performance—such as in MLA KVCache transfers—batch transfers can significantly outperform single-step transfers by reducing this overhead. Below are the batch transfer speed results based on a 200*2Gbps RDMA setup: ![image](https://github.com/user-attachments/assets/da4295c9-fb11-4c98-a3a7-73b85195aabb) Related test has been added to `mooncake-wheel/tests/transfer_engine_initiator_test.py` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:28:51 +00:00