[GH-ISSUE #49] The transfer_engine_bench cannot reach the RDMA NIC bandwidth limit #29

New issue

Closed

opened 2026-10-29 21:08:08 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:08:08 +00:00

Owner

Originally created by @CharlesJhonson on GitHub (Dec 25, 2024).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/49

I tested transfer_engine_bench on two machines, each with a 100Gb rdma network card. The bandwidth could not be increased by increasing the block_size/thread value, the bandwidth can only reach 6.9Gb, which is much less than 100Gb.
Is this normal? Or how can I configure or modify it to make the bandwidth close to the upper limit of the network card hardware?
ps: I added IBV_ACCESS_RELAXED_ORDERING in ibv_reg_mr flag #39

target:

./transfer_engine_bench --mode=target --metadata_server=192.168.1.24:2379 --local_server_name=192.168.1.24:12345 --device_name=mlx5_1
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1225 10:30:53.294373 25970 rdma_context.cpp:131] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:18

client test 1: with default configure

./transfer_engine_bench --metadata_server=192.168.1.24:2379 --local_server_name=1192.168.1.23:12345 --segment_id=192.168.1.24:12345 --device_name=mlx5_1
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1225 11:04:20.409399 84216 rdma_context.cpp:131] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:17
I1225 11:04:31.387705 84236 transfer_engine_bench.cpp:182] Worker 2 stopped!
I1225 11:04:31.387791 84235 transfer_engine_bench.cpp:182] Worker 1 stopped!
I1225 11:04:31.387799 84234 transfer_engine_bench.cpp:182] Worker 0 stopped!
I1225 11:04:31.387867 84237 transfer_engine_bench.cpp:182] Worker 3 stopped!
I1225 11:04:31.387917 84216 transfer_engine_bench.cpp:296] Test completed: duration 10.00, batch count 130202, throughput 6.83

client test 2: increase threads and block_size

./transfer_engine_bench --metadata_server=192.168.1.24:2379 --local_server_name=192.168.1.23:12345 --segment_id=192.168.1.24:12345 --device_name=mlx5_1 --threads=8 --block_size=1048576 --duration=60
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1225 11:05:07.084199 84992 rdma_context.cpp:131] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:17
I1225 11:06:08.012035 85008 transfer_engine_bench.cpp:182] Worker 7 stopped!
I1225 11:06:08.031401 85006 transfer_engine_bench.cpp:182] Worker 5 stopped!
I1225 11:06:08.050870 85001 transfer_engine_bench.cpp:182] Worker 0 stopped!
I1225 11:06:08.070183 85002 transfer_engine_bench.cpp:182] Worker 1 stopped!
I1225 11:06:08.089567 85004 transfer_engine_bench.cpp:182] Worker 3 stopped!
I1225 11:06:08.109010 85005 transfer_engine_bench.cpp:182] Worker 4 stopped!
I1225 11:06:08.128358 85003 transfer_engine_bench.cpp:182] Worker 2 stopped!
I1225 11:06:08.147769 85007 transfer_engine_bench.cpp:182] Worker 6 stopped!
I1225 11:06:08.147944 84992 transfer_engine_bench.cpp:296] Test completed: duration 60.15, batch count 3102, throughput 6.92

Originally created by @CharlesJhonson on GitHub (Dec 25, 2024). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/49 I tested `transfer_engine_bench` on two machines, each with a 100Gb rdma network card. The bandwidth could not be increased by increasing the block_size/thread value, the bandwidth can only reach 6.9Gb, which is much less than 100Gb. Is this normal? Or how can I configure or modify it to make the bandwidth close to the upper limit of the network card hardware? ps: I added `IBV_ACCESS_RELAXED_ORDERING` in ibv_reg_mr flag #39 target: ``` ./transfer_engine_bench --mode=target --metadata_server=192.168.1.24:2379 --local_server_name=192.168.1.24:12345 --device_name=mlx5_1 WARNING: Logging before InitGoogleLogging() is written to STDERR I1225 10:30:53.294373 25970 rdma_context.cpp:131] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:18 ``` client test 1: with default configure ``` ./transfer_engine_bench --metadata_server=192.168.1.24:2379 --local_server_name=1192.168.1.23:12345 --segment_id=192.168.1.24:12345 --device_name=mlx5_1 WARNING: Logging before InitGoogleLogging() is written to STDERR I1225 11:04:20.409399 84216 rdma_context.cpp:131] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:17 I1225 11:04:31.387705 84236 transfer_engine_bench.cpp:182] Worker 2 stopped! I1225 11:04:31.387791 84235 transfer_engine_bench.cpp:182] Worker 1 stopped! I1225 11:04:31.387799 84234 transfer_engine_bench.cpp:182] Worker 0 stopped! I1225 11:04:31.387867 84237 transfer_engine_bench.cpp:182] Worker 3 stopped! I1225 11:04:31.387917 84216 transfer_engine_bench.cpp:296] Test completed: duration 10.00, batch count 130202, throughput 6.83 ``` client test 2: increase `threads` and `block_size` ``` ./transfer_engine_bench --metadata_server=192.168.1.24:2379 --local_server_name=192.168.1.23:12345 --segment_id=192.168.1.24:12345 --device_name=mlx5_1 --threads=8 --block_size=1048576 --duration=60 WARNING: Logging before InitGoogleLogging() is written to STDERR I1225 11:05:07.084199 84992 rdma_context.cpp:131] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:17 I1225 11:06:08.012035 85008 transfer_engine_bench.cpp:182] Worker 7 stopped! I1225 11:06:08.031401 85006 transfer_engine_bench.cpp:182] Worker 5 stopped! I1225 11:06:08.050870 85001 transfer_engine_bench.cpp:182] Worker 0 stopped! I1225 11:06:08.070183 85002 transfer_engine_bench.cpp:182] Worker 1 stopped! I1225 11:06:08.089567 85004 transfer_engine_bench.cpp:182] Worker 3 stopped! I1225 11:06:08.109010 85005 transfer_engine_bench.cpp:182] Worker 4 stopped! I1225 11:06:08.128358 85003 transfer_engine_bench.cpp:182] Worker 2 stopped! I1225 11:06:08.147769 85007 transfer_engine_bench.cpp:182] Worker 6 stopped! I1225 11:06:08.147944 84992 transfer_engine_bench.cpp:296] Test completed: duration 60.15, batch count 3102, throughput 6.92 ```

bsakaguchi closed this issue

2026-10-29 21:08:08 +00:00

bsakaguchi commented

2026-10-29 21:08:09 +00:00

Author

Owner

@CharlesJhonson commented on GitHub (Dec 25, 2024):

Sorry, I misread it. The bandwidth unit output by the benchmark is GB (not Gb or GiB).

@CharlesJhonson commented on GitHub (Dec 25, 2024): Sorry, I misread it. The bandwidth unit output by the benchmark is GB (not Gb or GiB).

bsakaguchi referenced this issue

2026-10-29 21:26:06 +00:00

[PR #37] [MERGED] [TransferEngine][feature] add the http metadata server. #1294