[PR #557] [MERGED] Optimize slice handling to accelerate the large batch transfer operation #1649

New issue

Closed

opened 2026-10-29 21:29:11 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:29:11 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/557
Author: @SCDESPERTATE
Created: 6/25/2025
Status: ✅ Merged
Merged: 7/1/2025
Merged by: @alogfans

Base: main ← Head: main

📝 Commits (5)

440518b kick off transfer first when there are too many slices to post in RdmaTransport::submitTransferTask
f1e38f4 allow the last slice of a TransferRequest to be larger to reduce wr&&slice related overhead
1a94176 Merge branch 'kvcache-ai:main' into main
dcc03a9 add configs && docs explanation
04bc3a3 remove a unnecessary reset operation

📊 Changes

6 files changed (+51 additions, -6 deletions)

View changed files

📝 doc/en/transfer-engine.md (+2 -1)
📝 doc/zh/transfer-engine.md (+1 -0)
📝 mooncake-transfer-engine/include/config.h (+1 -0)
📝 mooncake-transfer-engine/include/transport/transport.h (+10 -3)
📝 mooncake-transfer-engine/src/config.cpp (+12 -0)
📝 mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp (+25 -2)

📄 Description

Motivation
In scenarios such as cases mentioned in #499 , where transfers involve a large batch size (thousands or more), but each chunk within the batch is relatively small (tens to hundreds of KiB), a substantial number of slices and work requests must be generated for the transfer. The current implementation introduces non-negligible latency due to the following issue:
- In RdmaTransport::submitTransferTask, all slices must be allocated before posting to RdmaContext::submitPostSend. When there are many requests, the volume of slices can overwhelm the ThreadLocalSliceCache, causing page faults and delaying transfer initiation.
- The TransferRequest length may not be a multiple of GlobalConfig::slice_size, leading to smaller final slices. Each slice becomes a separate work request, and when many small slices are created, overhead increases, reducing throughput.
Modification
- Submit the slice_list when the number of slices allocated by the new operator reaches a predefined watermark, even if there are still pending requests to be processed in RdmaTransport::submitTransferTask.
- Merge the final slice with the previous slice if its size is below a specified threshold.
Result
Run the Python script provided by #499
Since the modification occurs with RdmaTransport::submitTransferTask, two results have been merged for comparison as follow

====================================================================================================
SUMMARY
====================================================================================================
Test Case            wo-opt(s)      wo-opt(GB/s)   w-opt(GB/s)    w-opt(GB/s)    Speedup   
----------------------------------------------------------------------------------------------------
200MB/5000chunks     0.008          25.369         0.006          32.637         22.27%    
200MB/10000chunks    0.011          17.395         0.007          26.642         34.70%    
300MB/8000chunks     0.012          24.628         0.009          33.058         25.50%    
400MB/10000chunks    0.015          25.459         0.012          33.151         23.20%    
500MB/15000chunks    0.022          22.699         0.015          31.840         28.71%    
600MB/12000chunks    0.021          27.744         0.016          35.838         22.58%    
700MB/20000chunks    0.030          22.782         0.021          32.551         30.00%    
700MB/10000chunks    0.027          25.788         0.018          38.102         32.31%    

Average Speedup: 27.41%
Maximum Speedup: 34.70%
Average w-opt Throughput: 32.977 GB/s
Average wo-opt Throughput: 23.983 GB/s

Result shows that the modification achieves 20%~30% boost in throughput.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/557 **Author:** [@SCDESPERTATE](https://github.com/SCDESPERTATE) **Created:** 6/25/2025 **Status:** ✅ Merged **Merged:** 7/1/2025 **Merged by:** [@alogfans](https://github.com/alogfans) **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (5) - [`440518b`](https://github.com/kvcache-ai/Mooncake/commit/440518bd66737b67343a7c2a35cbf586bff6961f) kick off transfer first when there are too many slices to post in `RdmaTransport::submitTransferTask` - [`f1e38f4`](https://github.com/kvcache-ai/Mooncake/commit/f1e38f4a09799c2d981adbf2f611a7bf0677b1ce) allow the last slice of a `TransferRequest` to be larger to reduce wr&&slice related overhead - [`1a94176`](https://github.com/kvcache-ai/Mooncake/commit/1a94176bcbe452efb73830c740d66012e4c959b6) Merge branch 'kvcache-ai:main' into main - [`dcc03a9`](https://github.com/kvcache-ai/Mooncake/commit/dcc03a92a3c43ea788ce9bda01b94a88b7956f40) add configs && docs explanation - [`04bc3a3`](https://github.com/kvcache-ai/Mooncake/commit/04bc3a31ba497f155b054e7b35bdbfaf6681dd9b) remove a unnecessary reset operation ### 📊 Changes **6 files changed** (+51 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `doc/en/transfer-engine.md` (+2 -1) 📝 `doc/zh/transfer-engine.md` (+1 -0) 📝 `mooncake-transfer-engine/include/config.h` (+1 -0) 📝 `mooncake-transfer-engine/include/transport/transport.h` (+10 -3) 📝 `mooncake-transfer-engine/src/config.cpp` (+12 -0) 📝 `mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp` (+25 -2) </details> ### 📄 Description - Motivation In scenarios such as cases mentioned in #499 , where transfers involve a large batch size (thousands or more), but each chunk within the batch is relatively small (tens to hundreds of KiB), a substantial number of slices and work requests must be generated for the transfer. The current implementation introduces non-negligible latency due to the following issue: - In `RdmaTransport::submitTransferTask`, all slices must be allocated before posting to `RdmaContext::submitPostSend`. When there are many requests, the volume of slices can overwhelm the `ThreadLocalSliceCache`, causing page faults and delaying transfer initiation. - The `TransferRequest` length may not be a multiple of `GlobalConfig::slice_size`, leading to smaller final slices. Each slice becomes a separate work request, and when many small slices are created, overhead increases, reducing throughput. - Modification - Submit the slice_list when the number of slices allocated by the `new` operator reaches a predefined watermark, even if there are still pending requests to be processed in `RdmaTransport::submitTransfer`Task. - Merge the final slice with the previous slice if its size is below a specified threshold. - Result Run the Python [script](https://gist.github.com/ssssnow/37f17c9815cd26c3337ceae2729397cb) provided by #499 Since the modification occurs with `RdmaTransport::submitTransferTask`, two results have been merged for comparison as follow ```bash ==================================================================================================== SUMMARY ==================================================================================================== Test Case wo-opt(s) wo-opt(GB/s) w-opt(GB/s) w-opt(GB/s) Speedup ---------------------------------------------------------------------------------------------------- 200MB/5000chunks 0.008 25.369 0.006 32.637 22.27% 200MB/10000chunks 0.011 17.395 0.007 26.642 34.70% 300MB/8000chunks 0.012 24.628 0.009 33.058 25.50% 400MB/10000chunks 0.015 25.459 0.012 33.151 23.20% 500MB/15000chunks 0.022 22.699 0.015 31.840 28.71% 600MB/12000chunks 0.021 27.744 0.016 35.838 22.58% 700MB/20000chunks 0.030 22.782 0.021 32.551 30.00% 700MB/10000chunks 0.027 25.788 0.018 38.102 32.31% Average Speedup: 27.41% Maximum Speedup: 34.70% Average w-opt Throughput: 32.977 GB/s Average wo-opt Throughput: 23.983 GB/s ``` Result shows that the modification achieves 20%~30% boost in throughput. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:29:11 +00:00