[PR #1204] [CLOSED] Optimizing Throughput for Large Batch Sizes #2071

New issue

Closed

opened 2026-10-29 21:32:34 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:32:34 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1204
Author: @hzt123123
Created: 12/12/2025
Status: ❌ Closed

Base: main ← Head: optimize-gettransferstatus

📝 Commits (4)

e310d62 optimize getTransferStatus
986fac1 optimize getTransferStatus
9667b6f optimize getTransferStatus
c1f84b0 optimize getTransferStatus

📊 Changes

1 file changed (+14 additions, -6 deletions)

View changed files

📝 mooncake-transfer-engine/include/transfer_engine.h (+14 -6)

📄 Description

Finding

As shown in the table, when running ./transfer_engine_bench on two H20 servers, the throughput decreases as the batch size increases.

Optimization

We optimize the implementation of getTransferStatus() in transfer_engine.h, reducing the time complexity from O(n²·m) to O(n·m), where n is the batch size.

Evaluation

Experiment Setup

2 nodes, with 8 × H20 GPUs per node.

./transfer_engine_bench --mode=initiator --metadata_server=etcd://etcd_server:port --segment_id=abcd:12345 --operation=read --device_name=mlx5_bond_1 --batch_size=xx --block_size=4096

./transfer_engine_bench --mode=target --metadata_server=etcd://etcd_server:port  --local_server_name=abcd:12345 --operation=read --device_name=mlx5_bond_1  --batch_size=xx --block_size=4096

Performance Evaluation

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1204 **Author:** [@hzt123123](https://github.com/hzt123123) **Created:** 12/12/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `optimize-gettransferstatus` --- ### 📝 Commits (4) - [`e310d62`](https://github.com/kvcache-ai/Mooncake/commit/e310d62ce7e62c8379709bccfcf7d2bf329fce6d) optimize getTransferStatus - [`986fac1`](https://github.com/kvcache-ai/Mooncake/commit/986fac136ace6f9bcdc783b0799c74a85582a6e2) optimize getTransferStatus - [`9667b6f`](https://github.com/kvcache-ai/Mooncake/commit/9667b6f721a9264a5a9a3f7f04a2d4c39c258f95) optimize getTransferStatus - [`c1f84b0`](https://github.com/kvcache-ai/Mooncake/commit/c1f84b016432a415a43c0257ade4eaa2d973ea67) optimize getTransferStatus ### 📊 Changes **1 file changed** (+14 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-transfer-engine/include/transfer_engine.h` (+14 -6) </details> ### 📄 Description ## Finding As shown in the table, when running ./transfer_engine_bench on two H20 servers, the throughput decreases as the batch size increases. <img width="308" height="164" alt="image" src="https://github.com/user-attachments/assets/544f9c5e-7281-460a-a778-c8f8928580f4" /> ## Optimization We optimize the implementation of getTransferStatus() in transfer_engine.h, reducing the time complexity from O(n²·m) to O(n·m), where n is the batch size. ## Evaluation ### Experiment Setup 2 nodes, with 8 × H20 GPUs per node. ``` ./transfer_engine_bench --mode=initiator --metadata_server=etcd://etcd_server:port --segment_id=abcd:12345 --operation=read --device_name=mlx5_bond_1 --batch_size=xx --block_size=4096 ./transfer_engine_bench --mode=target --metadata_server=etcd://etcd_server:port --local_server_name=abcd:12345 --operation=read --device_name=mlx5_bond_1 --batch_size=xx --block_size=4096 ``` ### Performance Evaluation <img width="453" height="165" alt="image" src="https://github.com/user-attachments/assets/05464129-a9ab-467f-bb54-d34a309ecfd9" /> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:32:34 +00:00