[PR #564] [MERGED] add support for asynchronous batch transfer to accelerate transfer operation #1657

New issue

Closed

opened 2026-10-29 21:29:15 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:29:15 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/564
Author: @SCDESPERTATE
Created: 6/26/2025
Status: ✅ Merged
Merged: 7/7/2025
Merged by: @alogfans

Base: main ← Head: add_async_transfer_support

📝 Commits (2)

e1a5786 add support for asynchronously a transferring batch (without waiting for the result to return)
a9f7438 fix the return type in read/write wrappers of batchTransferAsync

📊 Changes

4 files changed (+231 additions, -0 deletions)

View changed files

📝 mooncake-integration/transfer_engine/transfer_engine_py.cpp (+140 -0)
📝 mooncake-integration/transfer_engine/transfer_engine_py.h (+21 -0)
📝 mooncake-transfer-engine/include/transport/transport.h (+1 -0)
📝 mooncake-wheel/tests/transfer_engine_initiator_test.py (+69 -0)

📄 Description

Add Support for Asynchronous Batch Transfer Interface

Motivation
As mentioned in #499 , the batch transfer operation can greatly accelerate the transfer process. With batch sizes exceeding thousands, Pybind interface overhead becomes substantial, reducing the overall throughput. One intuitive method in resolving this is dividing a large batch of transfer requests into micro-batches and posted them separately. But with current implementation of batch transfer, each micro-batch have to wait for all its tasks to complete, then return, making the division pointless. So this pull request aims to decouple task submission from result polling, allowing Pybind calling overhead to overlap with actual task transfer time.
Modification
Introduce an asynchronous batch transfer Pybind interface for task submission without immediate result retrieval. A separate querying interface ensures batch completion. These modifications,based on #557, aim to enhance transfer efficiency and throughput.
Result
Below are the batch transfer speed results based on a 200*2Gbps RDMA setup,

====================================================================================================
SUMMARY
====================================================================================================
Test Case            Sync(s)        Sync(GB/s)    Async(GB/s)     Async(GB/s)    Speedup   
----------------------------------------------------------------------------------------------------
200MB/5000chunks     0.006          32.587         0.005          39.418         17.33%    
200MB/10000chunks    0.007          26.612         0.007          28.794         7.575%    
300MB/8000chunks     0.009          33.172         0.007          39.978         17.02%    
400MB/10000chunks    0.011          34.076         0.010          41.097         17.08%    
500MB/15000chunks    0.015          31.949         0.011          42.585         24.97%    
600MB/12000chunks    0.016          35.754         0.014          42.936         16.72%    
700MB/20000chunks    0.021          32.688         0.016          42.880         23.76%    
700MB/10000chunks    0.018          38.163         0.016          43.102         11.46%    

Average Speedup: 16.99%
Maximum Speedup: 24.97%
Average Aync-Batch Throughput: 40.099 GB/s
Average Sync-Batch Throughput: 33.125 GB/s

Related test has been added to mooncake-wheel/tests/transfer_engine_initiator_test.py

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/564 **Author:** [@SCDESPERTATE](https://github.com/SCDESPERTATE) **Created:** 6/26/2025 **Status:** ✅ Merged **Merged:** 7/7/2025 **Merged by:** [@alogfans](https://github.com/alogfans) **Base:** `main` ← **Head:** `add_async_transfer_support` --- ### 📝 Commits (2) - [`e1a5786`](https://github.com/kvcache-ai/Mooncake/commit/e1a57869f9223eb8f6a9c609c46d3ac6089bed71) add support for asynchronously a transferring batch (without waiting for the result to return) - [`a9f7438`](https://github.com/kvcache-ai/Mooncake/commit/a9f743813466a35bf49511855396ffadcd1ace44) fix the return type in read/write wrappers of `batchTransferAsync` ### 📊 Changes **4 files changed** (+231 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-integration/transfer_engine/transfer_engine_py.cpp` (+140 -0) 📝 `mooncake-integration/transfer_engine/transfer_engine_py.h` (+21 -0) 📝 `mooncake-transfer-engine/include/transport/transport.h` (+1 -0) 📝 `mooncake-wheel/tests/transfer_engine_initiator_test.py` (+69 -0) </details> ### 📄 Description ## Add Support for Asynchronous Batch Transfer Interface - Motivation As mentioned in #499 , the batch transfer operation can greatly accelerate the transfer process. With batch sizes exceeding thousands, Pybind interface overhead becomes substantial, reducing the overall throughput. One intuitive method in resolving this is dividing a large batch of transfer requests into micro-batches and posted them separately. But with current implementation of batch transfer, each micro-batch have to wait for all its tasks to complete, then return, making the division pointless. So this pull request aims to decouple task submission from result polling, allowing Pybind calling overhead to overlap with actual task transfer time. - Modification Introduce an asynchronous batch transfer Pybind interface for task submission without immediate result retrieval. A separate querying interface ensures batch completion. These modifications,based on #557, aim to enhance transfer efficiency and throughput. - Result Below are the batch transfer speed results based on a 200*2Gbps RDMA setup, ```bash ==================================================================================================== SUMMARY ==================================================================================================== Test Case Sync(s) Sync(GB/s) Async(GB/s) Async(GB/s) Speedup ---------------------------------------------------------------------------------------------------- 200MB/5000chunks 0.006 32.587 0.005 39.418 17.33% 200MB/10000chunks 0.007 26.612 0.007 28.794 7.575% 300MB/8000chunks 0.009 33.172 0.007 39.978 17.02% 400MB/10000chunks 0.011 34.076 0.010 41.097 17.08% 500MB/15000chunks 0.015 31.949 0.011 42.585 24.97% 600MB/12000chunks 0.016 35.754 0.014 42.936 16.72% 700MB/20000chunks 0.021 32.688 0.016 42.880 23.76% 700MB/10000chunks 0.018 38.163 0.016 43.102 11.46% Average Speedup: 16.99% Maximum Speedup: 24.97% Average Aync-Batch Throughput: 40.099 GB/s Average Sync-Batch Throughput: 33.125 GB/s ``` Related test has been added to `mooncake-wheel/tests/transfer_engine_initiator_test.py` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:29:15 +00:00