[GH-ISSUE #39] [Performance] transfer_engine_bench get poor throughput with RDMA protocol

bsakaguchi commented

2026-10-29 21:08:01 +00:00

Owner

Originally created by @woodyji on GitHub (Dec 16, 2024).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/39

First of all, thank you for open-sourcing TransferEngine. However, I encountered some performance issues during my testing:

Performance Discrepancies

Issue 1: Single NIC Performance

Test Setup: Performance test between two nodes using a single NIC.
Observed Throughput:
- transfer_engine_bench achieved only 6.56 GB/s.
Expected Throughput:
- ib_send_bw (RDMA perftest tool): 11.47 GB/s.

Commands Used:

target:

./transfer_engine_bench --mode=target --metadata_server=10.166.100.01:2379 --local_server_name=10.166.100.02:12346 --device_name=mlx5_0 --operation=read

initiator:

./transfer_engine_bench --metadata_server=10.166.100.01:2379 --segment_id=10.166.100.02:12346 --local_server_name=10.166.100.03:12340 --device_name=mlx5_0 --operation=read

ib_send_bw server:

ib_send_bw -d mlx5_0 -F --run_infinitely -p 12300 -x 7

ib_send_bw client:

ib_send_bw -d mlx5_0 -F --run_infinitely 10.166.100.02 -p 12300 -x 7

Issue 2: Multi NIC Performance

Test Setup: Performance test between two nodes using two NIC by two parallel processes.
Observed Throughput:
- transfer_engine_bench achieved only 1.57GB/s.
Expected Throughput:
- ib_send_bw (RDMA perftest tool): 11.47 GB/s.

Commands Used:

target:

nohup ./transfer_engine_bench --mode=target --metadata_server=10.166.100.01:2379 --local_server_name=10.166.100.02:12346 --device_name=mlx5_0 --operation=read &
nohup ./transfer_engine_bench --mode=target --metadata_server=10.166.100.01:2379 --local_server_name=10.166.100.02:12347 --device_name=mlx5_1 --operation=read &

initiator:

nohup ./transfer_engine_bench --metadata_server=10.166.100.01:2379 --segment_id=10.166.100.02:12346 --local_server_name=10.166.100.03:12340 --device_name=mlx5_0 --operation=read &
nohup ./transfer_engine_bench --metadata_server=10.166.100.01:2379 --segment_id=10.166.100.02:12347 --local_server_name=10.166.100.03:12341 --device_name=mlx5_1 --operation=read &

ib_send_bw server:

ib_send_bw -d mlx5_0 -F --run_infinitely -p 12300 -x 7
ib_send_bw -d mlx5_1 -F --run_infinitely -p 12301 -x 7

ib_send_bw client:

ib_send_bw -d mlx5_0 -F --run_infinitely 10.166.100.02 -p 12300 -x 7
ib_send_bw -d mlx5_1 -F --run_infinitely 10.166.100.02 -p 12301 -x 7

Questions

How can I deal with the poor performance?
Is it reasonable to use multi processes to test multi NIC?

Originally created by @woodyji on GitHub (Dec 16, 2024). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/39 First of all, thank you for open-sourcing TransferEngine. However, I encountered some performance issues during my testing: ## Performance Discrepancies ### Issue 1: Single NIC Performance - **Test Setup**: Performance test between two nodes using a single NIC. - **Observed Throughput**: - `transfer_engine_bench` achieved only **6.56 GB/s**. - **Expected Throughput**: - `ib_send_bw` (RDMA perftest tool): **11.47 GB/s**. #### Commands Used: - **target:** ```bash ./transfer_engine_bench --mode=target --metadata_server=10.166.100.01:2379 --local_server_name=10.166.100.02:12346 --device_name=mlx5_0 --operation=read - **initiator:** ```bash ./transfer_engine_bench --metadata_server=10.166.100.01:2379 --segment_id=10.166.100.02:12346 --local_server_name=10.166.100.03:12340 --device_name=mlx5_0 --operation=read - **ib_send_bw server:** ```bash ib_send_bw -d mlx5_0 -F --run_infinitely -p 12300 -x 7 - **ib_send_bw client:** ```bash ib_send_bw -d mlx5_0 -F --run_infinitely 10.166.100.02 -p 12300 -x 7 ### Issue 2: Multi NIC Performance - **Test Setup**: Performance test between two nodes using two NIC by two parallel processes. - **Observed Throughput**: - `transfer_engine_bench` achieved only **1.57GB/s**. - **Expected Throughput**: - `ib_send_bw` (RDMA perftest tool): **11.47 GB/s**. #### Commands Used: - **target:** ```bash nohup ./transfer_engine_bench --mode=target --metadata_server=10.166.100.01:2379 --local_server_name=10.166.100.02:12346 --device_name=mlx5_0 --operation=read & nohup ./transfer_engine_bench --mode=target --metadata_server=10.166.100.01:2379 --local_server_name=10.166.100.02:12347 --device_name=mlx5_1 --operation=read & - **initiator:** ```bash nohup ./transfer_engine_bench --metadata_server=10.166.100.01:2379 --segment_id=10.166.100.02:12346 --local_server_name=10.166.100.03:12340 --device_name=mlx5_0 --operation=read & nohup ./transfer_engine_bench --metadata_server=10.166.100.01:2379 --segment_id=10.166.100.02:12347 --local_server_name=10.166.100.03:12341 --device_name=mlx5_1 --operation=read & - **ib_send_bw server:** ```bash ib_send_bw -d mlx5_0 -F --run_infinitely -p 12300 -x 7 ib_send_bw -d mlx5_1 -F --run_infinitely -p 12301 -x 7 - **ib_send_bw client:** ```bash ib_send_bw -d mlx5_0 -F --run_infinitely 10.166.100.02 -p 12300 -x 7 ib_send_bw -d mlx5_1 -F --run_infinitely 10.166.100.02 -p 12301 -x 7 ## Questions - How can I deal with the poor performance? - Is it reasonable to use multi processes to test multi NIC?

bsakaguchi closed this issue

2026-10-29 21:08:01 +00:00

bsakaguchi commented

2026-10-29 21:08:02 +00:00

Author

Owner

@alogfans commented on GitHub (Dec 16, 2024):

How can I deal with the poor performance?

By default, transfer_engine_bench repeatedly performs reads/writes with --threads=4 threads. For each iteration, it submits --batch_size=128 work requests, each of them transfers --block_size=4096 bytes data. You can try to tune these parameters to see performance improvement (increase --block_size and --threads=4, reduce --batch_size=128, ...)

Is it reasonable to use multi processes to test multi NIC?

Your execution produces poor performance because both instances are binded with the same processor. You can set --device_name=mlx5_0,mlx5_1 (do not add space and comma) to test multi NIC using one target/initiator process.

@alogfans commented on GitHub (Dec 16, 2024): > How can I deal with the poor performance? By default, `transfer_engine_bench` repeatedly performs reads/writes with `--threads=4` threads. For each iteration, it submits `--batch_size=128` work requests, each of them transfers `--block_size=4096` bytes data. You can try to tune these parameters to see performance improvement (increase `--block_size` and `--threads=4`, reduce `--batch_size=128`, ...) > Is it reasonable to use multi processes to test multi NIC? Your execution produces poor performance because both instances are binded with the same processor. You can set `--device_name=mlx5_0,mlx5_1` (do not add space and comma) to test multi NIC using one target/initiator process.

bsakaguchi commented

2026-10-29 21:08:02 +00:00

Author

Owner

@woodyji commented on GitHub (Dec 16, 2024):

By default, transfer_engine_bench repeatedly performs reads/writes with --threads=4 threads. For each iteration, it submits --batch_size=128 work requests, each of them transfers --block_size=4096 bytes data. You can try to tune these parameters to see performance improvement (increase --block_size and --threads=4, reduce --batch_size=128, ...)

Yes, I have experimented parameters like these, but the performance remains at the same level. Do you have any best practices to share?

Your execution produces poor performance because both instances are binded with the same processor. You can set --device_name=mlx5_0,mlx5_1 (do not add space and comma) to test multi NIC using one target/initiator process.

The transfer_engine_bench achieved better throughput which changed from 1.57GB/s to 2.37GB/s, but it is still not satisfactory. Besides, your vllm integration use one target/initiator process per rank, might have the same issue?

@woodyji commented on GitHub (Dec 16, 2024): > By default, transfer_engine_bench repeatedly performs reads/writes with --threads=4 threads. For each iteration, it submits --batch_size=128 work requests, each of them transfers --block_size=4096 bytes data. You can try to tune these parameters to see performance improvement (increase --block_size and --threads=4, reduce --batch_size=128, ...) Yes, I have experimented parameters like these, but the performance remains at the same level. Do you have any best practices to share? > Your execution produces poor performance because both instances are binded with the same processor. You can set --device_name=mlx5_0,mlx5_1 (do not add space and comma) to test multi NIC using one target/initiator process. The transfer_engine_bench achieved better throughput which changed from 1.57GB/s to 2.37GB/s, but it is still not satisfactory. Besides, your vllm integration use one target/initiator process per rank, might have the same issue?

bsakaguchi commented

2026-10-29 21:08:02 +00:00

Author

Owner

@alogfans commented on GitHub (Dec 16, 2024):

We have tested the latest code in our environment.

For case 1, the transfer engine bench reports 10.85 GB/s~11.77 GB/s at default configuration, while ib_write_bw is 13.01 GB/s.
For case 2, the aggregated BW is greater than the theorical BW of a single RNIC.

Does your machine share with other users, especially they consume CPU resources?

@alogfans commented on GitHub (Dec 16, 2024): We have tested the latest code in our environment. - For case 1, the transfer engine bench reports 10.85 GB/s~11.77 GB/s at default configuration, while `ib_write_bw` is 13.01 GB/s. - For case 2, the aggregated BW is greater than the theorical BW of a single RNIC. Does your machine share with other users, especially they consume CPU resources?

bsakaguchi commented

2026-10-29 21:08:02 +00:00

Author

Owner

@woodyji commented on GitHub (Dec 18, 2024):

The performance issue can be resolved by adding IBV_ACCESS_RELAXED_ORDERING to the memory region access flags. However, I'm curious as to why this flag was not set initially.

@woodyji commented on GitHub (Dec 18, 2024): The performance issue can be resolved by adding `IBV_ACCESS_RELAXED_ORDERING` to the memory region access flags. However, I'm curious as to why this flag was not set initially.

bsakaguchi commented

2026-10-29 21:08:02 +00:00

Author

Owner

@alogfans commented on GitHub (Dec 19, 2024):

The performance issue can be resolved by adding IBV_ACCESS_RELAXED_ORDERING to the memory region access flags. However, I'm curious as to why this flag was not set initially.

Thanks for your report. I am preparing a PR to enable this flag if user requested.

@alogfans commented on GitHub (Dec 19, 2024): > The performance issue can be resolved by adding `IBV_ACCESS_RELAXED_ORDERING` to the memory region access flags. However, I'm curious as to why this flag was not set initially. Thanks for your report. I am preparing a PR to enable this flag if user requested.

bsakaguchi commented

2026-10-29 21:08:02 +00:00

Author

Owner

@woodyji commented on GitHub (Dec 20, 2024):

I notice that transfer_engine_bench use block_size as TransferRequest's length, dose it means PagedAttention's block_size?
In practice, is every block must be registered once? If that's the case, it could lead to a significant number of registered memory regions. Are there any potential performance implications associated with this? Additionally, when is a block unregistered? Is it after the request is processed or only once the inference server is destroyed?

@woodyji commented on GitHub (Dec 20, 2024): I notice that `transfer_engine_bench` use `block_size` as `TransferRequest`'s length, dose it means `PagedAttention`'s `block_size`? In practice, is every block must be registered once? If that's the case, it could lead to a significant number of registered memory regions. Are there any potential performance implications associated with this? Additionally, when is a block unregistered? Is it after the request is processed or only once the inference server is destroyed?

bsakaguchi commented

2026-10-29 21:08:03 +00:00

Author

Owner

@alogfans commented on GitHub (Dec 20, 2024):

block_size just represent how many bytes to be transferred per request in benchmark program transfer_engine_bench. It is not related to PagedAttention.
A block can be transferred for multiple times, if the previous one is completed.

@alogfans commented on GitHub (Dec 20, 2024): `block_size` just represent how many bytes to be transferred per request in benchmark program `transfer_engine_bench`. It is not related to `PagedAttention`. A block can be transferred for multiple times, if the previous one is completed.

bsakaguchi commented

2026-10-29 21:08:03 +00:00

Author

Owner

@woodyji commented on GitHub (Dec 20, 2024):

A block can be transferred for multiple times, if the previous one is completed.

This means every PagedAttention's block registered once? As I mentioned earlier, this could lead to a significant number of registered memory regions? I'm considering how the TransferEngine integrates with the PagedAttention's block.

@woodyji commented on GitHub (Dec 20, 2024): > A block can be transferred for multiple times, if the previous one is completed. This means every `PagedAttention`'s block registered once? As I mentioned earlier, this could lead to a significant number of registered memory regions? I'm considering how the TransferEngine integrates with the `PagedAttention`'s block.

bsakaguchi commented

2026-10-29 21:08:03 +00:00

Author

Owner

@alogfans commented on GitHub (Dec 23, 2024):

If these PageAttention's blocks are stored contiguously, they can be registered as one memory region. You may consider change the allocation of PageAttention's blocks.

@alogfans commented on GitHub (Dec 23, 2024): If these `PageAttention`'s blocks are stored contiguously, they can be registered as one memory region. You may consider change the allocation of `PageAttention`'s blocks.

bsakaguchi commented

2026-10-29 21:08:03 +00:00

Author

Owner

@woodyji commented on GitHub (Dec 27, 2024):

Thank you, the issue can now be closed.

@woodyji commented on GitHub (Dec 27, 2024): Thank you, the issue can now be closed.

Rows
Columns

[GH-ISSUE #39] [Performance] transfer_engine_bench get poor throughput with RDMA protocol #24

Performance Discrepancies

Issue 1: Single NIC Performance

Commands Used:

Issue 2: Multi NIC Performance

Commands Used:

Questions