kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #166] 【TransferEngine】 Failed to select remote NIC for address 0xb7869c1cb5cae9f3 on segment #1 #79

New issue

Closed

opened 2026-10-29 21:08:52 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-10-29 21:08:52 +00:00

Owner

Originally created by @deepindeed2022 on GitHub (Mar 24, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/166

首先感谢你们优秀的开源工作。

我尝试在一台节点上测试transfer_engine的性能情况。麻烦帮忙提供一些进一步排查的思路。

测试目标与当前状态

期望能达到～40G/s

当前单线程实际只到 17G/s，尝试多线程出现问题 Failed to select remote NIC for address 0xb7869c1cb5cae9f3 on segment #1

硬件基本情况

1.ibdev2netdev

mlx5_0 port 1 ==> ens39np0 (Up)
mlx5_1 port 1 ==> ens37np0 (Up)
mlx5_bond_0 port 1 ==> bond0 (Up)

ibstat

CA 'mlx5_0'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0x9c63c0030065ab90
        System image GUID: 0x9c63c0030065ab90
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x9e63c0fffe65ab90
                Link layer: Ethernet
CA 'mlx5_1'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0x9c63c0030065ab98
        System image GUID: 0x9c63c0030065ab98
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x9e63c0fffe65ab98
                Link layer: Ethernet

CA 'mlx5_bond_0'
        CA type: MT4125
        Number of ports: 1
        Firmware version: 22.39.1002
        Hardware version: 0
        Node GUID: 0x9c63c00300b6bb46
        System image GUID: 0x9c63c00300b6bb46
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 100
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x2c9d7ffffeb60143
                Link layer: Ethernet

进一步提升 threads 发现报错，报错信息如下

threads参数值为2，命令行如下:

./transfer_engine_bench --mode=target --metadata_server=http://127.0.0.1:8080/metadata --local_server_name=127.0.0.1:12345 --nic_priority_matrix top.json --threads 2

./transfer_engine_bench --metadata_server=http://127.0.0.1:8080/metadata --segment_id=127.0.0.1:12345 --local_server_name=127.0.0.1:12346 --batch_size 128 --block_size 8388608 --nic_priority_matrix top.json --gpu_id 1 --threads 2

报错信息：

WARNING: Logging before InitGoogleLogging() is written to STDERR
W0324 15:43:22.061453 315208 transfer_engine_bench.cpp:407] Invalid flag: buffer size is samller than require_size, adjust to 2147483648
I0324 15:43:22.061517 315208 transfer_engine_bench.cpp:411] require_size: 2147483648, FLAGS_buffer_size: 2147483648
I0324 15:43:22.061520 315208 transfer_engine_bench.cpp:417] NR_SOCKETS:2
I0324 15:43:22.086596 315208 rdma_context.cpp:435] Find best gid index: 0 on mlx5_0/
I0324 15:43:22.087548 315208 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9e:63:c0:ff:fe:65:ab:90
I0324 15:43:22.110468 315208 rdma_context.cpp:435] Find best gid index: 0 on mlx5_1/
I0324 15:43:22.111340 315208 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9e:63:c0:ff:fe:65:ab:98
I0324 15:43:22.111920 315208 transfer_engine_bench.cpp:294] VRAM is used
I0324 15:43:22.832818 315208 rdma_context.cpp:199] length:2147483648
I0324 15:43:22.841877 315208 rdma_context.cpp:199] length:2147483648
W0324 15:43:22.852104 315228 worker_pool.cpp:114] Reselect remote NIC for address 0xb7869c1cb5cae9f3 on segment #1
E0324 15:43:22.852526 315228 worker_pool.cpp:127] **Failed to select remote NIC for address 0xb7869c1cb5cae9f3 on segment #1**
W0324 15:43:22.852535 315228 worker_pool.cpp:114] Reselect remote NIC for address 0xb7869c1cb5cbe9f3 on segment #1
E0324 15:43:22.852742 315228 worker_pool.cpp:127] Failed to select remote NIC for address 0xb7869c1cb5cbe9f3 on segment #1
W0324 15:43:22.852748 315228 worker_pool.cpp:114] Reselect remote NIC for address 0xb7869c1cb5cce9f3 on segment #1
E0324 15:43:22.852908 315228 worker_pool.cpp:127] Failed to select remote NIC for address 0xb7869c1cb5cce9f3 on segment #1
W0324 15:43:22.852912 315228 worker_pool.cpp:114] Reselect remote NIC for address 0xb7869c1cb5cde9f3 on segment #1

Originally created by @deepindeed2022 on GitHub (Mar 24, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/166 首先感谢你们优秀的开源工作。我尝试在一台节点上测试transfer_engine的性能情况。麻烦帮忙提供一些进一步排查的思路。 ## 测试目标与当前状态期望能达到～40G/s 当前单线程实际只到 17G/s，尝试多线程出现问题 Failed to select remote NIC for address 0xb7869c1cb5cae9f3 on segment `#1` ## 相关复现步骤如下: 1. 启动meta server: `go run . --addr=:8080` 2. 一个terminal 中执行 `./transfer_engine_bench --mode=target --metadata_server=http://127.0.0.1:8080/metadata --local_server_name=127.0.0.1:12345 --nic_priority_matrix top.json --threads 1` 另外一个 terminal 中启动 `./transfer_engine_bench --metadata_server=http://127.0.0.1:8080/metadata --segment_id=127.0.0.1:12345 --local_server_name=127.0.0.1:12346 --batch_size 128 --block_size 8388608 --nic_priority_matrix top.json --gpu_id 1 --threads 1` 其中 nic_priority_matrix 配置如下: ```json // top.json { "cpu:0":[["mlx5_0"],["mlx5_1"]], "cpu:1":[["mlx5_0"],["mlx5_1"]], "cuda:0":[["mlx5_0"],["mlx5_1"]] } ``` 查看priority_matrix如下： `curl 127.0.0.1:8080/metadata?key=mooncake/ram/127.0.0.1:12345` ``` {"buffers":[{"addr":139839303319552,"length":1073741824,"lkey":[2109070,2109123],"name":"cuda:0","rkey":[2109070,2109123]}],"devices":[{"gid":"fe:80:00:00:00:00:00:00:9e:63:c0:ff:fe:65:ab:90","lid":0,"name":"mlx5_0"},{"gid":"fe:80:00:00:00:00:00:00:9e:63:c0:ff:fe:65:ab:98","lid":0,"name":"mlx5_1"}],"name":"127.0.0.1:12345","priority_matrix":{"cpu:0":[["mlx5_0"],["mlx5_1"]],"cpu:1":[["mlx5_0"],["mlx5_1"]],"cuda:0":[["mlx5_0"],["mlx5_1"]]},"protocol":"rdma"} ``` 当前性能： Test completed: duration 0.07, batch count 1, throughput 16.32 GB/s 期望的性能 40 GB/s 左右 ib_send_bw -d mlx5_0 ib_send_bw -d mlx5_0 127.0.0.1 ![Image](https://github.com/user-attachments/assets/ba2c537b-82c6-46c3-a725-0f6b55df0a1d) ## 硬件基本情况 1.ibdev2netdev ``` mlx5_0 port 1 ==> ens39np0 (Up) mlx5_1 port 1 ==> ens37np0 (Up) mlx5_bond_0 port 1 ==> bond0 (Up) ``` 2. ibstat ``` CA 'mlx5_0' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0x9c63c0030065ab90 System image GUID: 0x9c63c0030065ab90 Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x9e63c0fffe65ab90 Link layer: Ethernet CA 'mlx5_1' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0x9c63c0030065ab98 System image GUID: 0x9c63c0030065ab98 Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x9e63c0fffe65ab98 Link layer: Ethernet CA 'mlx5_bond_0' CA type: MT4125 Number of ports: 1 Firmware version: 22.39.1002 Hardware version: 0 Node GUID: 0x9c63c00300b6bb46 System image GUID: 0x9c63c00300b6bb46 Port 1: State: Active Physical state: LinkUp Rate: 100 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x2c9d7ffffeb60143 Link layer: Ethernet ``` ## 进一步提升 threads 发现报错，报错信息如下 threads参数值为2，命令行如下: `./transfer_engine_bench --mode=target --metadata_server=http://127.0.0.1:8080/metadata --local_server_name=127.0.0.1:12345 --nic_priority_matrix top.json --threads 2` `./transfer_engine_bench --metadata_server=http://127.0.0.1:8080/metadata --segment_id=127.0.0.1:12345 --local_server_name=127.0.0.1:12346 --batch_size 128 --block_size 8388608 --nic_priority_matrix top.json --gpu_id 1 --threads 2` 报错信息： ``` WARNING: Logging before InitGoogleLogging() is written to STDERR W0324 15:43:22.061453 315208 transfer_engine_bench.cpp:407] Invalid flag: buffer size is samller than require_size, adjust to 2147483648 I0324 15:43:22.061517 315208 transfer_engine_bench.cpp:411] require_size: 2147483648, FLAGS_buffer_size: 2147483648 I0324 15:43:22.061520 315208 transfer_engine_bench.cpp:417] NR_SOCKETS:2 I0324 15:43:22.086596 315208 rdma_context.cpp:435] Find best gid index: 0 on mlx5_0/ I0324 15:43:22.087548 315208 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9e:63:c0:ff:fe:65:ab:90 I0324 15:43:22.110468 315208 rdma_context.cpp:435] Find best gid index: 0 on mlx5_1/ I0324 15:43:22.111340 315208 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9e:63:c0:ff:fe:65:ab:98 I0324 15:43:22.111920 315208 transfer_engine_bench.cpp:294] VRAM is used I0324 15:43:22.832818 315208 rdma_context.cpp:199] length:2147483648 I0324 15:43:22.841877 315208 rdma_context.cpp:199] length:2147483648 W0324 15:43:22.852104 315228 worker_pool.cpp:114] Reselect remote NIC for address 0xb7869c1cb5cae9f3 on segment #1 E0324 15:43:22.852526 315228 worker_pool.cpp:127] **Failed to select remote NIC for address 0xb7869c1cb5cae9f3 on segment #1** W0324 15:43:22.852535 315228 worker_pool.cpp:114] Reselect remote NIC for address 0xb7869c1cb5cbe9f3 on segment #1 E0324 15:43:22.852742 315228 worker_pool.cpp:127] Failed to select remote NIC for address 0xb7869c1cb5cbe9f3 on segment #1 W0324 15:43:22.852748 315228 worker_pool.cpp:114] Reselect remote NIC for address 0xb7869c1cb5cce9f3 on segment #1 E0324 15:43:22.852908 315228 worker_pool.cpp:127] Failed to select remote NIC for address 0xb7869c1cb5cce9f3 on segment #1 W0324 15:43:22.852912 315228 worker_pool.cpp:114] Reselect remote NIC for address 0xb7869c1cb5cde9f3 on segment #1 ```

bsakaguchi

2026-10-29 21:08:52 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:08:53 +00:00

Author

Owner

@alogfans commented on GitHub (Mar 25, 2025):

建议排查思路：

你可以尝试用 --batch_size=1 --block_size=1048576 测量一下性能，在我的测试集群上可取得 90GB/s 性能（四张卡）
针对第二个问题，可先不开启 --gpu_id 选项，即使用 DRAM 模式进行测试。也可尝试用 --device_name=mlx5_0,mlx5_1 替代 --nic_priority_matrix。

@alogfans commented on GitHub (Mar 25, 2025): 建议排查思路： 1. 你可以尝试用 --batch_size=1 --block_size=1048576 测量一下性能，在我的测试集群上可取得 90GB/s 性能（四张卡） 1. 针对第二个问题，可先不开启 --gpu_id 选项，即使用 DRAM 模式进行测试。也可尝试用 --device_name=mlx5_0,mlx5_1 替代 --nic_priority_matrix。

bsakaguchi commented

2026-10-29 21:08:53 +00:00

Author

Owner

@deepindeed2022 commented on GitHub (Mar 26, 2025):

测试启动1

./transfer_engine_bench --mode=target --metadata_server=http://127.0.0.1:8080/metadata --local_server_name=127.0.0.1:12345 --device_name=mlx5_0
./transfer_engine_bench --metadata_server=http://127.0.0.1:8080/metadata --segment_id=127.0.0.1:12345 --local_server_name=127.0.0.1:12346 --batch_size 1 --block_size=1048576 --device_name=mlx5_0

输出信息

WARNING: Logging before InitGoogleLogging() is written to STDERR
I0326 11:28:40.291864 222467 transfer_engine_bench.cpp:411] require_size: 1048576, FLAGS_buffer_size: 1073741824
I0326 11:28:40.291920 222467 transfer_engine_bench.cpp:417] NR_SOCKETS:2
I0326 11:28:40.322546 222467 rdma_context.cpp:435] Find best gid index: 0 on mlx5_0/
I0326 11:28:40.323503 222467 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9e:63:c0:ff:fe:65:ab:90
I0326 11:28:40.323951 222467 transfer_engine_bench.cpp:294] VRAM is used
I0326 11:28:41.305171 222467 rdma_context.cpp:199] length:1073741824
I0326 11:28:41.316728 222550 transfer_engine_bench.cpp:221] Worker 0 stopped!
I0326 11:28:41.316787 222467 transfer_engine_bench.cpp:332] numa node num: 2
I0326 11:28:41.316805 222467 transfer_engine_bench.cpp:334] Test completed: duration 0.01, batch count 1, throughput 0.18 GB/s
I0326 11:28:41.326654 222467 transfer_metadata_plugin.cpp:390] SocketHandShakePlugin: closing listen socket

测试启动2

./transfer_engine_bench --mode=target --metadata_server=http://127.0.0.1:8080/metadata --local_server_name=127.0.0.1:12345 --device_name=mlx5_0,mlx5_1 

./transfer_engine_bench --metadata_server=http://127.0.0.1:8080/metadata --segment_id=127.0.0.1:12345 --local_server_name=127.0.0.1:12346 --batch_size 1 --block_size=1048576 --device_name=mlx5_0,mlx5_1

输出错误信息如下：

WARNING: Logging before InitGoogleLogging() is written to STDERR
I0326 11:23:55.373183 208557 transfer_engine_bench.cpp:411] require_size: 1048576, FLAGS_buffer_size: 1073741824
I0326 11:23:55.373272 208557 transfer_engine_bench.cpp:417] NR_SOCKETS:2
I0326 11:23:55.381505 208557 rdma_context.cpp:435] Find best gid index: 0 on mlx5_0/
I0326 11:23:55.382892 208557 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9e:63:c0:ff:fe:65:ab:90
I0326 11:23:55.387651 208557 rdma_context.cpp:435] Find best gid index: 0 on mlx5_1/
I0326 11:23:55.388777 208557 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9e:63:c0:ff:fe:65:ab:98
I0326 11:23:55.389629 208557 transfer_engine_bench.cpp:294] VRAM is used
I0326 11:23:58.125977 208557 rdma_context.cpp:199] length:1073741824
I0326 11:23:58.153359 208557 rdma_context.cpp:199] length:1073741824
E0326 11:24:01.695173 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560010000, length: 65536, dest_addr: 140132434903040, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 0): transport retry counter exceeded
E0326 11:24:01.695698 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560050000, length: 65536, dest_addr: 140132435165184, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 0): Work Request Flushed Error
E0326 11:24:01.695708 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560090000, length: 65536, dest_addr: 140132435427328, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 0): Work Request Flushed Error
E0326 11:24:01.695716 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe5600d0000, length: 65536, dest_addr: 140132435689472, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 0): Work Request Flushed Error
E0326 11:24:01.759552 208558 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560020000, length: 65536, dest_addr: 140132434968576, local_nic: mlx5_0, peer_nic: 127.0.0.1:12345@mlx5_1, dest_rkey: 2109141, retry_cnt: 0): transport retry counter exceeded
E0326 11:24:01.759956 208558 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560060000, length: 65536, dest_addr: 140132435230720, local_nic: mlx5_0, peer_nic: 127.0.0.1:12345@mlx5_1, dest_rkey: 2109141, retry_cnt: 0): Work Request Flushed Error
E0326 11:24:01.759967 208558 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe5600a0000, length: 65536, dest_addr: 140132435492864, local_nic: mlx5_0, peer_nic: 127.0.0.1:12345@mlx5_1, dest_rkey: 2109141, retry_cnt: 0): Work Request Flushed Error
E0326 11:24:01.759974 208558 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe5600e0000, length: 65536, dest_addr: 140132435755008, local_nic: mlx5_0, peer_nic: 127.0.0.1:12345@mlx5_1, dest_rkey: 2109141, retry_cnt: 0): Work Request Flushed Error
E0326 11:24:05.453018 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560010000, length: 65536, dest_addr: 140132434903040, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 1): transport retry counter exceeded
E0326 11:24:05.453560 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560050000, length: 65536, dest_addr: 140132435165184, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 1): Work Request Flushed Error
E0326 11:24:05.453569 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560090000, length: 65536, dest_addr: 140132435427328, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 1): Work Request Flushed Error
E0326 11:24:05.453577 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe5600d0000, length: 65536, dest_addr: 140132435689472, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 1): Work Request Flushed Error
I0326 11:24:05.453976 208672 transfer_engine_bench.cpp:221] Worker 0 stopped!
I0326 11:24:05.454190 208557 transfer_engine_bench.cpp:332] numa node num: 2
I0326 11:24:05.454221 208557 transfer_engine_bench.cpp:334] Test completed: duration 7.28, batch count 1, throughput 0.00 GB/s

@deepindeed2022 commented on GitHub (Mar 26, 2025): ## 测试启动1 ```bash ./transfer_engine_bench --mode=target --metadata_server=http://127.0.0.1:8080/metadata --local_server_name=127.0.0.1:12345 --device_name=mlx5_0 ./transfer_engine_bench --metadata_server=http://127.0.0.1:8080/metadata --segment_id=127.0.0.1:12345 --local_server_name=127.0.0.1:12346 --batch_size 1 --block_size=1048576 --device_name=mlx5_0 ``` 输出信息 ```bash WARNING: Logging before InitGoogleLogging() is written to STDERR I0326 11:28:40.291864 222467 transfer_engine_bench.cpp:411] require_size: 1048576, FLAGS_buffer_size: 1073741824 I0326 11:28:40.291920 222467 transfer_engine_bench.cpp:417] NR_SOCKETS:2 I0326 11:28:40.322546 222467 rdma_context.cpp:435] Find best gid index: 0 on mlx5_0/ I0326 11:28:40.323503 222467 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9e:63:c0:ff:fe:65:ab:90 I0326 11:28:40.323951 222467 transfer_engine_bench.cpp:294] VRAM is used I0326 11:28:41.305171 222467 rdma_context.cpp:199] length:1073741824 I0326 11:28:41.316728 222550 transfer_engine_bench.cpp:221] Worker 0 stopped! I0326 11:28:41.316787 222467 transfer_engine_bench.cpp:332] numa node num: 2 I0326 11:28:41.316805 222467 transfer_engine_bench.cpp:334] Test completed: duration 0.01, batch count 1, throughput 0.18 GB/s I0326 11:28:41.326654 222467 transfer_metadata_plugin.cpp:390] SocketHandShakePlugin: closing listen socket ``` ## 测试启动2 ```bash ./transfer_engine_bench --mode=target --metadata_server=http://127.0.0.1:8080/metadata --local_server_name=127.0.0.1:12345 --device_name=mlx5_0,mlx5_1 ./transfer_engine_bench --metadata_server=http://127.0.0.1:8080/metadata --segment_id=127.0.0.1:12345 --local_server_name=127.0.0.1:12346 --batch_size 1 --block_size=1048576 --device_name=mlx5_0,mlx5_1 ``` 输出错误信息如下： ``` WARNING: Logging before InitGoogleLogging() is written to STDERR I0326 11:23:55.373183 208557 transfer_engine_bench.cpp:411] require_size: 1048576, FLAGS_buffer_size: 1073741824 I0326 11:23:55.373272 208557 transfer_engine_bench.cpp:417] NR_SOCKETS:2 I0326 11:23:55.381505 208557 rdma_context.cpp:435] Find best gid index: 0 on mlx5_0/ I0326 11:23:55.382892 208557 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9e:63:c0:ff:fe:65:ab:90 I0326 11:23:55.387651 208557 rdma_context.cpp:435] Find best gid index: 0 on mlx5_1/ I0326 11:23:55.388777 208557 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9e:63:c0:ff:fe:65:ab:98 I0326 11:23:55.389629 208557 transfer_engine_bench.cpp:294] VRAM is used I0326 11:23:58.125977 208557 rdma_context.cpp:199] length:1073741824 I0326 11:23:58.153359 208557 rdma_context.cpp:199] length:1073741824 E0326 11:24:01.695173 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560010000, length: 65536, dest_addr: 140132434903040, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 0): transport retry counter exceeded E0326 11:24:01.695698 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560050000, length: 65536, dest_addr: 140132435165184, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 0): Work Request Flushed Error E0326 11:24:01.695708 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560090000, length: 65536, dest_addr: 140132435427328, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 0): Work Request Flushed Error E0326 11:24:01.695716 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe5600d0000, length: 65536, dest_addr: 140132435689472, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 0): Work Request Flushed Error E0326 11:24:01.759552 208558 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560020000, length: 65536, dest_addr: 140132434968576, local_nic: mlx5_0, peer_nic: 127.0.0.1:12345@mlx5_1, dest_rkey: 2109141, retry_cnt: 0): transport retry counter exceeded E0326 11:24:01.759956 208558 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560060000, length: 65536, dest_addr: 140132435230720, local_nic: mlx5_0, peer_nic: 127.0.0.1:12345@mlx5_1, dest_rkey: 2109141, retry_cnt: 0): Work Request Flushed Error E0326 11:24:01.759967 208558 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe5600a0000, length: 65536, dest_addr: 140132435492864, local_nic: mlx5_0, peer_nic: 127.0.0.1:12345@mlx5_1, dest_rkey: 2109141, retry_cnt: 0): Work Request Flushed Error E0326 11:24:01.759974 208558 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe5600e0000, length: 65536, dest_addr: 140132435755008, local_nic: mlx5_0, peer_nic: 127.0.0.1:12345@mlx5_1, dest_rkey: 2109141, retry_cnt: 0): Work Request Flushed Error E0326 11:24:05.453018 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560010000, length: 65536, dest_addr: 140132434903040, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 1): transport retry counter exceeded E0326 11:24:05.453560 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560050000, length: 65536, dest_addr: 140132435165184, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 1): Work Request Flushed Error E0326 11:24:05.453569 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe560090000, length: 65536, dest_addr: 140132435427328, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 1): Work Request Flushed Error E0326 11:24:05.453577 208561 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7fe5600d0000, length: 65536, dest_addr: 140132435689472, local_nic: mlx5_1, peer_nic: 127.0.0.1:12345@mlx5_0, dest_rkey: 2109095, retry_cnt: 1): Work Request Flushed Error I0326 11:24:05.453976 208672 transfer_engine_bench.cpp:221] Worker 0 stopped! I0326 11:24:05.454190 208557 transfer_engine_bench.cpp:332] numa node num: 2 I0326 11:24:05.454221 208557 transfer_engine_bench.cpp:334] Test completed: duration 7.28, batch count 1, throughput 0.00 GB/s ```

bsakaguchi commented

2026-10-29 21:08:53 +00:00

Author

Owner

@alogfans commented on GitHub (Mar 26, 2025):

在测试案例1中，只有 mlx5_0 设备被使用，本地和远端的 mlx5_0 是通的；
在测试案例2中，启用了 mlx5_0 和 mlx5_1。目前的 Transfer Engine 假设本地和远端任意选择一个网卡之间是通的，而您的网络可能不是如此。（比方说，本地 mlx5_0 和远端 mlx5_1 不通，可使用 ib_send_bw 等工具验证一下。）

@alogfans commented on GitHub (Mar 26, 2025): 在测试案例1中，只有 mlx5_0 设备被使用，本地和远端的 mlx5_0 是通的；在测试案例2中，启用了 mlx5_0 和 mlx5_1。目前的 Transfer Engine 假设本地和远端任意选择一个网卡之间是通的，而您的网络可能不是如此。（比方说，本地 mlx5_0 和远端 mlx5_1 不通，可使用 ib_send_bw 等工具验证一下。）

bsakaguchi commented

2026-10-29 21:08:53 +00:00

Author

Owner

@donghua100 commented on GitHub (Mar 31, 2025):

在测试案例1中，只有 mlx5_0 设备被使用，本地和远端的 mlx5_0 是通的；在测试案例2中，启用了 mlx5_0 和 mlx5_1。目前的 Transfer Engine 假设本地和远端任意选择一个网卡之间是通的，而您的网络可能不是如此。（比方说，本地 mlx5_0 和远端 mlx5_1 不通，可使用 ib_send_bw 等工具验证一下。）

测试都在一台机器上完成，使用ib_send_bw 进行测试，服务端使用ib_send_bw -d mlx5_1 -x 1 -D 10 -F，客户端使用ib_send_bw -d mlx5_0 -x 1 -D 10 -F 192.168.1.2，发现无法正常通信：

ib_send_bw -d mlx5_0 -x 1 -D 10 -F 192.168.1.2
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 2048[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
Failed to modify QP 1399 to RTR
 Unable to Connect the HCA's through the link

这对Transfer Engine 的性能测试有影响吗？使用ib_send_bw只通过mlx5_0进行测试，约40GB/s。

 ib_send_bw -d mlx5_0 -x 1 -D 10 -F 192.168.1.2
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 2048[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0578 PSN 0x3dd7a1
 GID: 254:128:00:00:00:00:00:00:158:99:192:255:254:101:171:144
 remote address: LID 0000 QPN 0x0579 PSN 0xeaf055
 GID: 254:128:00:00:00:00:00:00:158:99:192:255:254:101:171:144
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      4185995          0.00               43602.97                   0.697647
---------------------------------------------------------------------------------------

使用transfer engine 进行测试，有22GB/s，相差几乎一半，这是为什么呢

服务端：

 ./transfer_engine_bench --mode=target \
                        --metadata_server=http://10.249.32.181:8080/metadata \
                        --local_server_name=10.249.32.181:12345 \
                        --device_name=mlx5_0 --batch_size 16 --block_size 1073741824
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0331 10:39:33.634567 270214 transfer_engine_bench.cpp:410] Invalid flag: buffer size is smaller than require_size, adjust to 68719476736
I0331 10:39:33.662564 270214 rdma_context.cpp:426] Find best gid index: 3 on mlx5_0/
I0331 10:39:33.663642 270214 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:01
I0331 10:40:44.284797 270214 transfer_engine_bench.cpp:395] numa node num: 2

客户端：

 ./transfer_engine_bench --metadata_server=http://10.249.32.181:8080/metadata    --segment_id=10.249.32.181:12345 \
                        --local_server_name=10.249.32.181:12346 --device_name mlx5_0 --threads 32 --batch_size 1 --block_size 1073741824

WARNING: Logging before InitGoogleLogging() is written to STDERR
W0331 10:48:21.452488 292440 transfer_engine_bench.cpp:410] Invalid flag: buffer size is smaller than require_size, adjust to 34359738368
I0331 10:48:21.479751 292440 rdma_context.cpp:426] Find best gid index: 3 on mlx5_0/
I0331 10:48:21.481020 292440 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:01
I0331 10:48:47.487704 293475 transfer_engine_bench.cpp:226] Worker 24 stopped!
I0331 10:48:47.536581 293468 transfer_engine_bench.cpp:226] Worker 17 stopped!
I0331 10:48:47.585222 293464 transfer_engine_bench.cpp:226] Worker 13 stopped!
I0331 10:48:47.631556 293463 transfer_engine_bench.cpp:226] Worker 12 stopped!
I0331 10:48:47.680270 293480 transfer_engine_bench.cpp:226] Worker 29 stopped!
I0331 10:48:47.729001 293476 transfer_engine_bench.cpp:226] Worker 25 stopped!
I0331 10:48:47.777766 293470 transfer_engine_bench.cpp:226] Worker 19 stopped!
I0331 10:48:47.824056 293477 transfer_engine_bench.cpp:226] Worker 26 stopped!
I0331 10:48:47.870352 293471 transfer_engine_bench.cpp:226] Worker 20 stopped!
I0331 10:48:47.916616 293467 transfer_engine_bench.cpp:226] Worker 16 stopped!
I0331 10:48:47.962829 293465 transfer_engine_bench.cpp:226] Worker 14 stopped!
I0331 10:48:48.009110 293479 transfer_engine_bench.cpp:226] Worker 28 stopped!
I0331 10:48:48.055405 293481 transfer_engine_bench.cpp:226] Worker 30 stopped!
I0331 10:48:48.104084 293472 transfer_engine_bench.cpp:226] Worker 21 stopped!
I0331 10:48:48.152894 293478 transfer_engine_bench.cpp:226] Worker 27 stopped!
I0331 10:48:48.201711 293474 transfer_engine_bench.cpp:226] Worker 23 stopped!
I0331 10:48:48.250352 293482 transfer_engine_bench.cpp:226] Worker 31 stopped!
I0331 10:48:48.299057 293452 transfer_engine_bench.cpp:226] Worker 1 stopped!
I0331 10:48:48.345305 293457 transfer_engine_bench.cpp:226] Worker 6 stopped!
I0331 10:48:48.394061 293456 transfer_engine_bench.cpp:226] Worker 5 stopped!
I0331 10:48:48.440336 293455 transfer_engine_bench.cpp:226] Worker 4 stopped!
I0331 10:48:48.489151 293454 transfer_engine_bench.cpp:226] Worker 3 stopped!
I0331 10:48:48.535493 293451 transfer_engine_bench.cpp:226] Worker 0 stopped!
I0331 10:48:48.581693 293453 transfer_engine_bench.cpp:226] Worker 2 stopped!
I0331 10:48:48.630438 293462 transfer_engine_bench.cpp:226] Worker 11 stopped!
I0331 10:48:48.679209 293466 transfer_engine_bench.cpp:226] Worker 15 stopped!
I0331 10:48:48.725497 293473 transfer_engine_bench.cpp:226] Worker 22 stopped!
I0331 10:48:48.771833 293461 transfer_engine_bench.cpp:226] Worker 10 stopped!
I0331 10:48:48.818122 293459 transfer_engine_bench.cpp:226] Worker 8 stopped!
I0331 10:48:48.866873 293460 transfer_engine_bench.cpp:226] Worker 9 stopped!
I0331 10:48:48.915711 293458 transfer_engine_bench.cpp:226] Worker 7 stopped!
I0331 10:48:48.962065 293469 transfer_engine_bench.cpp:226] Worker 18 stopped!
I0331 10:48:48.962255 292440 transfer_engine_bench.cpp:337] numa node num: 2
I0331 10:48:48.962272 292440 transfer_engine_bench.cpp:339] Test completed: duration 11.49, batch count 239, throughput 22.33 GB/s
I0331 10:48:52.826879 292440 transfer_metadata_plugin.cpp:390] SocketHandShakePlugin: closing listen socket

服务端，客户端使用device_name mlx5_0,mlx5_1就不能正常通信了：
服务端：

./transfer_engine_bench --mode=target \
                        --metadata_server=http://10.249.32.181:8080/metadata \
                        --local_server_name=10.249.32.181:12345 \
                        --device_name=mlx5_0,mlx5_1 --batch_size 16 --block_size 1073741824
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0331 10:52:07.700984 297905 transfer_engine_bench.cpp:410] Invalid flag: buffer size is smaller than require_size, adjust to 68719476736
I0331 10:52:07.731801 297905 rdma_context.cpp:426] Find best gid index: 3 on mlx5_0/
I0331 10:52:07.733109 297905 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:01
I0331 10:52:07.758379 297905 rdma_context.cpp:426] Find best gid index: 3 on mlx5_1/
I0331 10:52:07.759268 297905 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:02
I0331 10:53:21.523730 297905 transfer_engine_bench.cpp:395] numa node num: 2
E0331 10:54:20.638639 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110]
E0331 10:54:21.662643 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110]
E0331 10:54:22.686630 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110]
E0331 10:54:23.710633 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110]
E0331 10:54:24.734627 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110]
E0331 10:54:25.822625 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110]
E0331 10:54:26.846632 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110]
E0331 10:54:27.870617 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110]

客户端：

./transfer_engine_bench --metadata_server=http://10.249.32.181:8080/metadata    --segment_id=10.249.32.181:12345 \
                        --local_server_name=10.249.32.181:12346 --device_name mlx5_0,mlx5_1 --threads 32 --batch_size 1 --block_size 1073741824

WARNING: Logging before InitGoogleLogging() is written to STDERR
W0331 10:54:01.738992 300744 transfer_engine_bench.cpp:410] Invalid flag: buffer size is smaller than require_size, adjust to 34359738368
I0331 10:54:01.771806 300744 rdma_context.cpp:426] Find best gid index: 3 on mlx5_0/
I0331 10:54:01.773362 300744 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:01
I0331 10:54:01.799746 300744 rdma_context.cpp:426] Find best gid index: 3 on mlx5_1/
I0331 10:54:01.801182 300744 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:02
E0331 10:54:20.638882 300750 rdma_endpoint.cpp:129] Invalid argument: received packet mismatch
E0331 10:54:20.638911 300750 worker_pool.cpp:253] Worker: Cannot make connection for endpoint: 10.249.32.181:12345@mlx5_0
E0331 10:54:21.662834 300746 rdma_endpoint.cpp:129] Invalid argument: received packet mismatch
E0331 10:54:21.662866 300746 worker_pool.cpp:253] Worker: Cannot make connection for endpoint: 10.249.32.181:12345@mlx5_1
E0331 10:54:22.686839 300750 rdma_endpoint.cpp:129] Invalid argument: received packet mismatch
E0331 10:54:22.686877 300750 worker_pool.cpp:253] Worker: Cannot make connection for endpoint: 10.249.32.181:12345@mlx5_0
E0331 10:54:23.710799 300746 rdma_endpoint.cpp:129] Invalid argument: received packet mismatch
E0331 10:54:23.710834 300746 worker_pool.cpp:253] Worker: Cannot make connection for endpoint: 10.249.32.181:12345@mlx5_1
E0331 10:54:24.734818 300750 rdma_endpoint.cpp:129] Invalid argument: received packet mismatch
E0331 10:54:24.734853 300750 worker_pool.cpp:253] Worker: Cannot make connection for endpoint: 10.249.32.181:12345@mlx5_0
E0331 10:54:25.822809 300750 rdma_endpoint.cpp:129] Invalid argument: received packet mismatch
E0331 10:54:25.822842 300750 worker_pool.cpp:253] Worker: Cannot make connection for endpoint: 10.249.32.181:12345@mlx5_0

如果两张网卡无法正常通信，只使用一张卡，transfer engine 能到ib_send_bw最大带宽的多少？

@donghua100 commented on GitHub (Mar 31, 2025): > 在测试案例1中，只有 mlx5_0 设备被使用，本地和远端的 mlx5_0 是通的；在测试案例2中，启用了 mlx5_0 和 mlx5_1。目前的 Transfer Engine 假设本地和远端任意选择一个网卡之间是通的，而您的网络可能不是如此。（比方说，本地 mlx5_0 和远端 mlx5_1 不通，可使用 ib_send_bw 等工具验证一下。）测试都在一台机器上完成，使用```ib_send_bw``` 进行测试，服务端使用```ib_send_bw -d mlx5_1 -x 1 -D 10 -F```，客户端使用```ib_send_bw -d mlx5_0 -x 1 -D 10 -F 192.168.1.2```，发现无法正常通信： ```bash ib_send_bw -d mlx5_0 -x 1 -D 10 -F 192.168.1.2 --------------------------------------------------------------------------------------- Send BW Test Dual-port : OFF Device : mlx5_0 Number of qps : 1 Transport type : IB Connection type : RC Using SRQ : OFF PCIe relax order: ON ibv_wr* API : ON TX depth : 128 CQ Moderation : 1 Mtu : 2048[B] Link type : Ethernet GID index : 1 Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- Failed to modify QP 1399 to RTR Unable to Connect the HCA's through the link ``` 这对Transfer Engine 的性能测试有影响吗？使用```ib_send_bw```只通过mlx5_0进行测试，约40GB/s。 ```bash ib_send_bw -d mlx5_0 -x 1 -D 10 -F 192.168.1.2 --------------------------------------------------------------------------------------- Send BW Test Dual-port : OFF Device : mlx5_0 Number of qps : 1 Transport type : IB Connection type : RC Using SRQ : OFF PCIe relax order: ON ibv_wr* API : ON TX depth : 128 CQ Moderation : 1 Mtu : 2048[B] Link type : Ethernet GID index : 1 Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- local address: LID 0000 QPN 0x0578 PSN 0x3dd7a1 GID: 254:128:00:00:00:00:00:00:158:99:192:255:254:101:171:144 remote address: LID 0000 QPN 0x0579 PSN 0xeaf055 GID: 254:128:00:00:00:00:00:00:158:99:192:255:254:101:171:144 --------------------------------------------------------------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] 65536 4185995 0.00 43602.97 0.697647 --------------------------------------------------------------------------------------- ``` 使用transfer engine 进行测试，有22GB/s，相差几乎一半，这是为什么呢服务端： ```bash ./transfer_engine_bench --mode=target \ --metadata_server=http://10.249.32.181:8080/metadata \ --local_server_name=10.249.32.181:12345 \ --device_name=mlx5_0 --batch_size 16 --block_size 1073741824 WARNING: Logging before InitGoogleLogging() is written to STDERR W0331 10:39:33.634567 270214 transfer_engine_bench.cpp:410] Invalid flag: buffer size is smaller than require_size, adjust to 68719476736 I0331 10:39:33.662564 270214 rdma_context.cpp:426] Find best gid index: 3 on mlx5_0/ I0331 10:39:33.663642 270214 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:01 I0331 10:40:44.284797 270214 transfer_engine_bench.cpp:395] numa node num: 2 ``` 客户端： ```bash ./transfer_engine_bench --metadata_server=http://10.249.32.181:8080/metadata --segment_id=10.249.32.181:12345 \ --local_server_name=10.249.32.181:12346 --device_name mlx5_0 --threads 32 --batch_size 1 --block_size 1073741824 WARNING: Logging before InitGoogleLogging() is written to STDERR W0331 10:48:21.452488 292440 transfer_engine_bench.cpp:410] Invalid flag: buffer size is smaller than require_size, adjust to 34359738368 I0331 10:48:21.479751 292440 rdma_context.cpp:426] Find best gid index: 3 on mlx5_0/ I0331 10:48:21.481020 292440 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:01 I0331 10:48:47.487704 293475 transfer_engine_bench.cpp:226] Worker 24 stopped! I0331 10:48:47.536581 293468 transfer_engine_bench.cpp:226] Worker 17 stopped! I0331 10:48:47.585222 293464 transfer_engine_bench.cpp:226] Worker 13 stopped! I0331 10:48:47.631556 293463 transfer_engine_bench.cpp:226] Worker 12 stopped! I0331 10:48:47.680270 293480 transfer_engine_bench.cpp:226] Worker 29 stopped! I0331 10:48:47.729001 293476 transfer_engine_bench.cpp:226] Worker 25 stopped! I0331 10:48:47.777766 293470 transfer_engine_bench.cpp:226] Worker 19 stopped! I0331 10:48:47.824056 293477 transfer_engine_bench.cpp:226] Worker 26 stopped! I0331 10:48:47.870352 293471 transfer_engine_bench.cpp:226] Worker 20 stopped! I0331 10:48:47.916616 293467 transfer_engine_bench.cpp:226] Worker 16 stopped! I0331 10:48:47.962829 293465 transfer_engine_bench.cpp:226] Worker 14 stopped! I0331 10:48:48.009110 293479 transfer_engine_bench.cpp:226] Worker 28 stopped! I0331 10:48:48.055405 293481 transfer_engine_bench.cpp:226] Worker 30 stopped! I0331 10:48:48.104084 293472 transfer_engine_bench.cpp:226] Worker 21 stopped! I0331 10:48:48.152894 293478 transfer_engine_bench.cpp:226] Worker 27 stopped! I0331 10:48:48.201711 293474 transfer_engine_bench.cpp:226] Worker 23 stopped! I0331 10:48:48.250352 293482 transfer_engine_bench.cpp:226] Worker 31 stopped! I0331 10:48:48.299057 293452 transfer_engine_bench.cpp:226] Worker 1 stopped! I0331 10:48:48.345305 293457 transfer_engine_bench.cpp:226] Worker 6 stopped! I0331 10:48:48.394061 293456 transfer_engine_bench.cpp:226] Worker 5 stopped! I0331 10:48:48.440336 293455 transfer_engine_bench.cpp:226] Worker 4 stopped! I0331 10:48:48.489151 293454 transfer_engine_bench.cpp:226] Worker 3 stopped! I0331 10:48:48.535493 293451 transfer_engine_bench.cpp:226] Worker 0 stopped! I0331 10:48:48.581693 293453 transfer_engine_bench.cpp:226] Worker 2 stopped! I0331 10:48:48.630438 293462 transfer_engine_bench.cpp:226] Worker 11 stopped! I0331 10:48:48.679209 293466 transfer_engine_bench.cpp:226] Worker 15 stopped! I0331 10:48:48.725497 293473 transfer_engine_bench.cpp:226] Worker 22 stopped! I0331 10:48:48.771833 293461 transfer_engine_bench.cpp:226] Worker 10 stopped! I0331 10:48:48.818122 293459 transfer_engine_bench.cpp:226] Worker 8 stopped! I0331 10:48:48.866873 293460 transfer_engine_bench.cpp:226] Worker 9 stopped! I0331 10:48:48.915711 293458 transfer_engine_bench.cpp:226] Worker 7 stopped! I0331 10:48:48.962065 293469 transfer_engine_bench.cpp:226] Worker 18 stopped! I0331 10:48:48.962255 292440 transfer_engine_bench.cpp:337] numa node num: 2 I0331 10:48:48.962272 292440 transfer_engine_bench.cpp:339] Test completed: duration 11.49, batch count 239, throughput 22.33 GB/s I0331 10:48:52.826879 292440 transfer_metadata_plugin.cpp:390] SocketHandShakePlugin: closing listen socket ``` 服务端，客户端使用```device_name mlx5_0,mlx5_1```就不能正常通信了：服务端： ```bash ./transfer_engine_bench --mode=target \ --metadata_server=http://10.249.32.181:8080/metadata \ --local_server_name=10.249.32.181:12345 \ --device_name=mlx5_0,mlx5_1 --batch_size 16 --block_size 1073741824 WARNING: Logging before InitGoogleLogging() is written to STDERR W0331 10:52:07.700984 297905 transfer_engine_bench.cpp:410] Invalid flag: buffer size is smaller than require_size, adjust to 68719476736 I0331 10:52:07.731801 297905 rdma_context.cpp:426] Find best gid index: 3 on mlx5_0/ I0331 10:52:07.733109 297905 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:01 I0331 10:52:07.758379 297905 rdma_context.cpp:426] Find best gid index: 3 on mlx5_1/ I0331 10:52:07.759268 297905 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:02 I0331 10:53:21.523730 297905 transfer_engine_bench.cpp:395] numa node num: 2 E0331 10:54:20.638639 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110] E0331 10:54:21.662643 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110] E0331 10:54:22.686630 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110] E0331 10:54:23.710633 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110] E0331 10:54:24.734627 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110] E0331 10:54:25.822625 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110] E0331 10:54:26.846632 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110] E0331 10:54:27.870617 297912 rdma_endpoint.cpp:385] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Connection timed out [110] ``` 客户端： ```bash ./transfer_engine_bench --metadata_server=http://10.249.32.181:8080/metadata --segment_id=10.249.32.181:12345 \ --local_server_name=10.249.32.181:12346 --device_name mlx5_0,mlx5_1 --threads 32 --batch_size 1 --block_size 1073741824 WARNING: Logging before InitGoogleLogging() is written to STDERR W0331 10:54:01.738992 300744 transfer_engine_bench.cpp:410] Invalid flag: buffer size is smaller than require_size, adjust to 34359738368 I0331 10:54:01.771806 300744 rdma_context.cpp:426] Find best gid index: 3 on mlx5_0/ I0331 10:54:01.773362 300744 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:01 I0331 10:54:01.799746 300744 rdma_context.cpp:426] Find best gid index: 3 on mlx5_1/ I0331 10:54:01.801182 300744 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:02 E0331 10:54:20.638882 300750 rdma_endpoint.cpp:129] Invalid argument: received packet mismatch E0331 10:54:20.638911 300750 worker_pool.cpp:253] Worker: Cannot make connection for endpoint: 10.249.32.181:12345@mlx5_0 E0331 10:54:21.662834 300746 rdma_endpoint.cpp:129] Invalid argument: received packet mismatch E0331 10:54:21.662866 300746 worker_pool.cpp:253] Worker: Cannot make connection for endpoint: 10.249.32.181:12345@mlx5_1 E0331 10:54:22.686839 300750 rdma_endpoint.cpp:129] Invalid argument: received packet mismatch E0331 10:54:22.686877 300750 worker_pool.cpp:253] Worker: Cannot make connection for endpoint: 10.249.32.181:12345@mlx5_0 E0331 10:54:23.710799 300746 rdma_endpoint.cpp:129] Invalid argument: received packet mismatch E0331 10:54:23.710834 300746 worker_pool.cpp:253] Worker: Cannot make connection for endpoint: 10.249.32.181:12345@mlx5_1 E0331 10:54:24.734818 300750 rdma_endpoint.cpp:129] Invalid argument: received packet mismatch E0331 10:54:24.734853 300750 worker_pool.cpp:253] Worker: Cannot make connection for endpoint: 10.249.32.181:12345@mlx5_0 E0331 10:54:25.822809 300750 rdma_endpoint.cpp:129] Invalid argument: received packet mismatch E0331 10:54:25.822842 300750 worker_pool.cpp:253] Worker: Cannot make connection for endpoint: 10.249.32.181:12345@mlx5_0 ``` 如果两张网卡无法正常通信，只使用一张卡，transfer engine 能到ib_send_bw最大带宽的多少？

bsakaguchi commented

2026-10-29 21:08:53 +00:00

Author

Owner

@alogfans commented on GitHub (Apr 10, 2025):

@donghua100

测试都在一台机器上完成，使用ib_send_bw 进行测试，服务端使用ib_send_bw -d mlx5_1 -x 1 -D 10 -F，客户端使用ib_send_bw -d mlx5_0 -x 1 -D 10 -F 192.168.1.2，发现无法正常通信：
这可能是驱动配置的原因，同时我们在实验中发现某些特殊软硬件环境下非 root 权限执行 RDMA 操作也可能存在问题。

使用transfer engine 进行测试，有 22GB/s，相差几乎一半，这是为什么呢
我们在早期版本的时候验证了单卡带宽可达到 40GB/s 以上，但是也注意到引擎性能、读写模式等会对实际测得的端到端带宽产生影响。我们下一阶段重点在优化 RDMA 模块实现。目前建议您考虑从以下方面调优：

获取最新的代码，该版本修复了 memory leak 问题并保证内存分配回收效率
客户端线程数量不宜过多，深度不宜过深
参考 doc/zh/transfer-engine.md 的说明适当调节 MC_SLICE_SIZE、MC_WORKERS_PER_CTX 等参数

@alogfans commented on GitHub (Apr 10, 2025): @donghua100 > 测试都在一台机器上完成，使用ib_send_bw 进行测试，服务端使用ib_send_bw -d mlx5_1 -x 1 -D 10 -F，客户端使用ib_send_bw -d mlx5_0 -x 1 -D 10 -F 192.168.1.2，发现无法正常通信：这可能是驱动配置的原因，同时我们在实验中发现某些特殊软硬件环境下非 root 权限执行 RDMA 操作也可能存在问题。 > 使用transfer engine 进行测试，有 22GB/s，相差几乎一半，这是为什么呢我们在早期版本的时候验证了单卡带宽可达到 40GB/s 以上，但是也注意到引擎性能、读写模式等会对实际测得的端到端带宽产生影响。我们下一阶段重点在优化 RDMA 模块实现。目前建议您考虑从以下方面调优： - 获取最新的代码，该版本修复了 memory leak 问题并保证内存分配回收效率 - 客户端线程数量不宜过多，深度不宜过深 - 参考 doc/zh/transfer-engine.md 的说明适当调节 MC_SLICE_SIZE、MC_WORKERS_PER_CTX 等参数

bsakaguchi commented

2026-10-29 21:08:53 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 21, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 21, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:08:53 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 28, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 28, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:26:17 +00:00

[PR #79] [MERGED] [TransferEngine] Correct the count of slice_quque_ in rdma/worker_pool, remove the repeated check for overlap #1313