kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #284] Runing transfer_engine_bench on H800 and H20，but got local protection error. #112

New issue
Closed
opened 2026-10-29 21:09:31 +00:00 by bsakaguchi · 8 comments
bsakaguchi commented 2026-10-29 21:09:31 +00:00
Owner
Copy link

Originally created by @Time-Limit on GitHub (Apr 22, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/284

When I run transfer_engine_bench on H800 and H20 to test bandwidth，I got an error 'local protection error' from initiator node, and got some warning about 'Re-establish connection' from target node.

Image

Image

initiator script on H800

./transfer_engine_bench \
    --metadata_server=redis://172.30.12.3:6379 \
    --segment_id=10.0.0.78:12333 \
    --protocol=rdma \
    --operation=write \
    --local_server_name=10.0.0.4:12346 \
    --auto_discovery=true

target script on H20

./transfer_engine_bench \
    --metadata_server=redis://172.30.12.3:6379 \
    --local_server_name=10.0.0.78:12333 \
    --auto_discovery=true \
    --protocol=rdma \
    --mode=target
Originally created by @Time-Limit on GitHub (Apr 22, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/284 When I run transfer_engine_bench on H800 and H20 to test bandwidth，I got an error 'local protection error' from initiator node, and got some warning about 'Re-establish connection' from target node. ![Image](https://github.com/user-attachments/assets/5016fb5e-27f6-4e98-affb-827d99480636) ![Image](https://github.com/user-attachments/assets/a4b29a60-cd9a-46dc-8b49-324d1e3c8360) initiator script on H800 ``` ./transfer_engine_bench \ --metadata_server=redis://172.30.12.3:6379 \ --segment_id=10.0.0.78:12333 \ --protocol=rdma \ --operation=write \ --local_server_name=10.0.0.4:12346 \ --auto_discovery=true ``` target script on H20 ``` ./transfer_engine_bench \ --metadata_server=redis://172.30.12.3:6379 \ --local_server_name=10.0.0.78:12333 \ --auto_discovery=true \ --protocol=rdma \ --mode=target ```
bsakaguchi 2026-10-29 21:09:31 +00:00
bsakaguchi commented 2026-10-29 21:09:32 +00:00
Author
Owner
Copy link

@Time-Limit commented on GitHub (Apr 22, 2025):

H20(initiator) write data to H800(target), ok
H20(initiator) read data from H800(target), ok

H800(initiator) write data to H20(target), failed
H800(initiator) read data from H20(target), ok

<!-- gh-comment-id:2820638107 --> @Time-Limit commented on GitHub (Apr 22, 2025): H20(initiator) write data to H800(target), ok H20(initiator) read data from H800(target), ok H800(initiator) write data to H20(target), failed H800(initiator) read data from H20(target), ok
bsakaguchi commented 2026-10-29 21:09:32 +00:00
Author
Owner
Copy link

@stmatengss commented on GitHub (Apr 22, 2025):

"local protection error" means memory cannot be registered or accessed by RDMA. This may be due to low permissions. You can run the benchmark as a root user or using sudo.

<!-- gh-comment-id:2820663515 --> @stmatengss commented on GitHub (Apr 22, 2025): "local protection error" means memory cannot be registered or accessed by RDMA. This may be due to low permissions. You can run the benchmark as a root user or using sudo.
bsakaguchi commented 2026-10-29 21:09:32 +00:00
Author
Owner
Copy link

@a4zhangfei commented on GitHub (Apr 28, 2025):

"local protection error" means memory cannot be registered or accessed by RDMA. This may be due to low permissions. You can run the benchmark as a root user or using sudo.

@stmatengss I'm having the same problem. I found that h800- > h20 rdma transmission can only be successfully using mlx5_bond_0 device, and others will encounter local protection error problem. transfer_engine_bench is executed under root privileges.

The mlx5_bond_0 of h800 can be sent successfully to any device of h20, and other devices of h800 cannot send successfully to any device of h20.

I use the following rdma example to achieve successful transmission of any rdma device from h800 to h20, so I feel like there is a problem with the code?
https://gitee.com/brucewanght/RDMA-Tutorial/blob/master/RDMA_RC_example.c

The rdma device for h800 and h20 are as follows:
h800:

[RDMA-Tutorial]$ show_gids
DEV	PORT	INDEX	GID					IPv4  		VER	DEV
---	----	-----	---					------------  	---	---
mlx5_bond_0	1	0	fe80:0000:0000:0000:a288:c2ff:fe01:9f5c			v1	bond0
mlx5_bond_0	1	1	fe80:0000:0000:0000:a288:c2ff:fe01:9f5c			v2	bond0
mlx5_bond_0	1	2	0000:0000:0000:0000:0000:ffff:1e8b:0dfe	30.139.13.254  	v1	bond0
mlx5_bond_0	1	3	0000:0000:0000:0000:0000:ffff:1e8b:0dfe	30.139.13.254  	v2	bond0
mlx5_bond_1	1	0	fe80:0000:0000:0000:a288:c2ff:fe01:7a1c			v1	bond1
mlx5_bond_1	1	1	fe80:0000:0000:0000:a288:c2ff:fe01:7a1c			v2	bond1
mlx5_bond_1	1	2	0000:0000:0000:0000:0000:ffff:1e8b:3066	30.139.48.102  	v1	bond1
mlx5_bond_1	1	3	0000:0000:0000:0000:0000:ffff:1e8b:3066	30.139.48.102  	v2	bond1
mlx5_bond_2	1	0	fe80:0000:0000:0000:a288:c2ff:fe01:780c			v1	bond2
mlx5_bond_2	1	1	fe80:0000:0000:0000:a288:c2ff:fe01:780c			v2	bond2
mlx5_bond_2	1	2	0000:0000:0000:0000:0000:ffff:1e8b:303e	30.139.48.62  	v1	bond2
mlx5_bond_2	1	3	0000:0000:0000:0000:0000:ffff:1e8b:303e	30.139.48.62  	v2	bond2
mlx5_bond_3	1	0	fe80:0000:0000:0000:a288:c2ff:fe01:79fc			v1	bond3
mlx5_bond_3	1	1	fe80:0000:0000:0000:a288:c2ff:fe01:79fc			v2	bond3
mlx5_bond_3	1	2	0000:0000:0000:0000:0000:ffff:1e8b:3022	30.139.48.34  	v1	bond3
mlx5_bond_3	1	3	0000:0000:0000:0000:0000:ffff:1e8b:3022	30.139.48.34  	v2	bond3
mlx5_bond_4	1	0	fe80:0000:0000:0000:a288:c2ff:fe01:782c			v1	bond4
mlx5_bond_4	1	1	fe80:0000:0000:0000:a288:c2ff:fe01:782c			v2	bond4
mlx5_bond_4	1	2	0000:0000:0000:0000:0000:ffff:1e8b:300a	30.139.48.10  	v1	bond4
mlx5_bond_4	1	3	0000:0000:0000:0000:0000:ffff:1e8b:300a	30.139.48.10  	v2	bond4
mlx5_bond_5	1	0	fe80:0000:0000:0000:a288:c2ff:fe01:791c			v1	bond5
mlx5_bond_5	1	1	fe80:0000:0000:0000:a288:c2ff:fe01:791c			v2	bond5
mlx5_bond_5	1	2	0000:0000:0000:0000:0000:ffff:1e8b:3052	30.139.48.82  	v1	bond5
mlx5_bond_5	1	3	0000:0000:0000:0000:0000:ffff:1e8b:3052	30.139.48.82  	v2	bond5
mlx5_bond_6	1	0	fe80:0000:0000:0000:a288:c2ff:fe01:778c			v1	bond6
mlx5_bond_6	1	1	fe80:0000:0000:0000:a288:c2ff:fe01:778c			v2	bond6
mlx5_bond_6	1	2	0000:0000:0000:0000:0000:ffff:1e8b:3042	30.139.48.66  	v1	bond6
mlx5_bond_6	1	3	0000:0000:0000:0000:0000:ffff:1e8b:3042	30.139.48.66  	v2	bond6
mlx5_bond_7	1	0	fe80:0000:0000:0000:a288:c2ff:fe01:77ac			v1	bond7
mlx5_bond_7	1	1	fe80:0000:0000:0000:a288:c2ff:fe01:77ac			v2	bond7
mlx5_bond_7	1	2	0000:0000:0000:0000:0000:ffff:1e8b:304e	30.139.48.78  	v1	bond7
mlx5_bond_7	1	3	0000:0000:0000:0000:0000:ffff:1e8b:304e	30.139.48.78  	v2	bond7
n_gids_found=32

h20:

DEV	PORT	INDEX	GID					IPv4  		VER	DEV
---	----	-----	---					------------  	---	---
mlx5_bond_0	1	0	fe80:0000:0000:0000:e29d:73ff:fe49:e5b8			v1	bond0
mlx5_bond_0	1	1	fe80:0000:0000:0000:e29d:73ff:fe49:e5b8			v2	bond0
mlx5_bond_0	1	2	0000:0000:0000:0000:0000:ffff:1e8b:0012	30.139.0.18  	v1	bond0
mlx5_bond_0	1	3	0000:0000:0000:0000:0000:ffff:1e8b:0012	30.139.0.18  	v2	bond0
mlx5_bond_1	1	0	fe80:0000:0000:0000:e29d:73ff:fe49:e608			v1	bond1
mlx5_bond_1	1	1	fe80:0000:0000:0000:e29d:73ff:fe49:e608			v2	bond1
mlx5_bond_1	1	2	0000:0000:0000:0000:0000:ffff:1e8b:0036	30.139.0.54  	v1	bond1
mlx5_bond_1	1	3	0000:0000:0000:0000:0000:ffff:1e8b:0036	30.139.0.54  	v2	bond1
mlx5_bond_2	1	0	fe80:0000:0000:0000:e29d:73ff:fe4b:450c			v1	bond2
mlx5_bond_2	1	1	fe80:0000:0000:0000:e29d:73ff:fe4b:450c			v2	bond2
mlx5_bond_2	1	2	0000:0000:0000:0000:0000:ffff:1e8b:0026	30.139.0.38  	v1	bond2
mlx5_bond_2	1	3	0000:0000:0000:0000:0000:ffff:1e8b:0026	30.139.0.38  	v2	bond2
mlx5_bond_3	1	0	fe80:0000:0000:0000:e29d:73ff:fe4b:3c0c			v1	bond3
mlx5_bond_3	1	1	fe80:0000:0000:0000:e29d:73ff:fe4b:3c0c			v2	bond3
mlx5_bond_3	1	2	0000:0000:0000:0000:0000:ffff:1e8b:001e	30.139.0.30  	v1	bond3
mlx5_bond_3	1	3	0000:0000:0000:0000:0000:ffff:1e8b:001e	30.139.0.30  	v2	bond3
mlx5_bond_4	1	0	fe80:0000:0000:0000:e29d:73ff:fe49:d8a8			v1	bond4
mlx5_bond_4	1	1	fe80:0000:0000:0000:e29d:73ff:fe49:d8a8			v2	bond4
mlx5_bond_4	1	2	0000:0000:0000:0000:0000:ffff:1e8b:005a	30.139.0.90  	v1	bond4
mlx5_bond_4	1	3	0000:0000:0000:0000:0000:ffff:1e8b:005a	30.139.0.90  	v2	bond4
mlx5_bond_5	1	0	fe80:0000:0000:0000:e29d:73ff:fe49:d888			v1	bond5
mlx5_bond_5	1	1	fe80:0000:0000:0000:e29d:73ff:fe49:d888			v2	bond5
mlx5_bond_5	1	2	0000:0000:0000:0000:0000:ffff:1e8b:004e	30.139.0.78  	v1	bond5
mlx5_bond_5	1	3	0000:0000:0000:0000:0000:ffff:1e8b:004e	30.139.0.78  	v2	bond5
mlx5_bond_6	1	0	fe80:0000:0000:0000:e29d:73ff:fe49:dce8			v1	bond6
mlx5_bond_6	1	1	fe80:0000:0000:0000:e29d:73ff:fe49:dce8			v2	bond6
mlx5_bond_6	1	2	0000:0000:0000:0000:0000:ffff:1e8b:000a	30.139.0.10  	v1	bond6
mlx5_bond_6	1	3	0000:0000:0000:0000:0000:ffff:1e8b:000a	30.139.0.10  	v2	bond6
mlx5_bond_7	1	0	fe80:0000:0000:0000:e29d:73ff:fe49:d0d8			v1	bond7
mlx5_bond_7	1	1	fe80:0000:0000:0000:e29d:73ff:fe49:d0d8			v2	bond7
mlx5_bond_7	1	2	0000:0000:0000:0000:0000:ffff:1e8b:0042	30.139.0.66  	v1	bond7
mlx5_bond_7	1	3	0000:0000:0000:0000:0000:ffff:1e8b:0042	30.139.0.66  	v2	bond7
n_gids_found=32

I'd appreciate any help.

<!-- gh-comment-id:2833927854 --> @a4zhangfei commented on GitHub (Apr 28, 2025): > "local protection error" means memory cannot be registered or accessed by RDMA. This may be due to low permissions. You can run the benchmark as a root user or using sudo. @stmatengss I'm having the same problem. I found that h800- > h20 rdma transmission can only be successfully using mlx5_bond_0 device, and others will encounter `local protection error` problem. `transfer_engine_bench` is executed under root privileges. **The mlx5_bond_0 of h800 can be sent successfully to any device of h20, and other devices of h800 cannot send successfully to any device of h20.** I use the following rdma example to achieve successful transmission of any rdma device from h800 to h20, so I feel like there is a problem with the code? https://gitee.com/brucewanght/RDMA-Tutorial/blob/master/RDMA_RC_example.c The rdma device for h800 and h20 are as follows: h800: ``` [RDMA-Tutorial]$ show_gids DEV PORT INDEX GID IPv4 VER DEV --- ---- ----- --- ------------ --- --- mlx5_bond_0 1 0 fe80:0000:0000:0000:a288:c2ff:fe01:9f5c v1 bond0 mlx5_bond_0 1 1 fe80:0000:0000:0000:a288:c2ff:fe01:9f5c v2 bond0 mlx5_bond_0 1 2 0000:0000:0000:0000:0000:ffff:1e8b:0dfe 30.139.13.254 v1 bond0 mlx5_bond_0 1 3 0000:0000:0000:0000:0000:ffff:1e8b:0dfe 30.139.13.254 v2 bond0 mlx5_bond_1 1 0 fe80:0000:0000:0000:a288:c2ff:fe01:7a1c v1 bond1 mlx5_bond_1 1 1 fe80:0000:0000:0000:a288:c2ff:fe01:7a1c v2 bond1 mlx5_bond_1 1 2 0000:0000:0000:0000:0000:ffff:1e8b:3066 30.139.48.102 v1 bond1 mlx5_bond_1 1 3 0000:0000:0000:0000:0000:ffff:1e8b:3066 30.139.48.102 v2 bond1 mlx5_bond_2 1 0 fe80:0000:0000:0000:a288:c2ff:fe01:780c v1 bond2 mlx5_bond_2 1 1 fe80:0000:0000:0000:a288:c2ff:fe01:780c v2 bond2 mlx5_bond_2 1 2 0000:0000:0000:0000:0000:ffff:1e8b:303e 30.139.48.62 v1 bond2 mlx5_bond_2 1 3 0000:0000:0000:0000:0000:ffff:1e8b:303e 30.139.48.62 v2 bond2 mlx5_bond_3 1 0 fe80:0000:0000:0000:a288:c2ff:fe01:79fc v1 bond3 mlx5_bond_3 1 1 fe80:0000:0000:0000:a288:c2ff:fe01:79fc v2 bond3 mlx5_bond_3 1 2 0000:0000:0000:0000:0000:ffff:1e8b:3022 30.139.48.34 v1 bond3 mlx5_bond_3 1 3 0000:0000:0000:0000:0000:ffff:1e8b:3022 30.139.48.34 v2 bond3 mlx5_bond_4 1 0 fe80:0000:0000:0000:a288:c2ff:fe01:782c v1 bond4 mlx5_bond_4 1 1 fe80:0000:0000:0000:a288:c2ff:fe01:782c v2 bond4 mlx5_bond_4 1 2 0000:0000:0000:0000:0000:ffff:1e8b:300a 30.139.48.10 v1 bond4 mlx5_bond_4 1 3 0000:0000:0000:0000:0000:ffff:1e8b:300a 30.139.48.10 v2 bond4 mlx5_bond_5 1 0 fe80:0000:0000:0000:a288:c2ff:fe01:791c v1 bond5 mlx5_bond_5 1 1 fe80:0000:0000:0000:a288:c2ff:fe01:791c v2 bond5 mlx5_bond_5 1 2 0000:0000:0000:0000:0000:ffff:1e8b:3052 30.139.48.82 v1 bond5 mlx5_bond_5 1 3 0000:0000:0000:0000:0000:ffff:1e8b:3052 30.139.48.82 v2 bond5 mlx5_bond_6 1 0 fe80:0000:0000:0000:a288:c2ff:fe01:778c v1 bond6 mlx5_bond_6 1 1 fe80:0000:0000:0000:a288:c2ff:fe01:778c v2 bond6 mlx5_bond_6 1 2 0000:0000:0000:0000:0000:ffff:1e8b:3042 30.139.48.66 v1 bond6 mlx5_bond_6 1 3 0000:0000:0000:0000:0000:ffff:1e8b:3042 30.139.48.66 v2 bond6 mlx5_bond_7 1 0 fe80:0000:0000:0000:a288:c2ff:fe01:77ac v1 bond7 mlx5_bond_7 1 1 fe80:0000:0000:0000:a288:c2ff:fe01:77ac v2 bond7 mlx5_bond_7 1 2 0000:0000:0000:0000:0000:ffff:1e8b:304e 30.139.48.78 v1 bond7 mlx5_bond_7 1 3 0000:0000:0000:0000:0000:ffff:1e8b:304e 30.139.48.78 v2 bond7 n_gids_found=32 ``` h20: ``` DEV PORT INDEX GID IPv4 VER DEV --- ---- ----- --- ------------ --- --- mlx5_bond_0 1 0 fe80:0000:0000:0000:e29d:73ff:fe49:e5b8 v1 bond0 mlx5_bond_0 1 1 fe80:0000:0000:0000:e29d:73ff:fe49:e5b8 v2 bond0 mlx5_bond_0 1 2 0000:0000:0000:0000:0000:ffff:1e8b:0012 30.139.0.18 v1 bond0 mlx5_bond_0 1 3 0000:0000:0000:0000:0000:ffff:1e8b:0012 30.139.0.18 v2 bond0 mlx5_bond_1 1 0 fe80:0000:0000:0000:e29d:73ff:fe49:e608 v1 bond1 mlx5_bond_1 1 1 fe80:0000:0000:0000:e29d:73ff:fe49:e608 v2 bond1 mlx5_bond_1 1 2 0000:0000:0000:0000:0000:ffff:1e8b:0036 30.139.0.54 v1 bond1 mlx5_bond_1 1 3 0000:0000:0000:0000:0000:ffff:1e8b:0036 30.139.0.54 v2 bond1 mlx5_bond_2 1 0 fe80:0000:0000:0000:e29d:73ff:fe4b:450c v1 bond2 mlx5_bond_2 1 1 fe80:0000:0000:0000:e29d:73ff:fe4b:450c v2 bond2 mlx5_bond_2 1 2 0000:0000:0000:0000:0000:ffff:1e8b:0026 30.139.0.38 v1 bond2 mlx5_bond_2 1 3 0000:0000:0000:0000:0000:ffff:1e8b:0026 30.139.0.38 v2 bond2 mlx5_bond_3 1 0 fe80:0000:0000:0000:e29d:73ff:fe4b:3c0c v1 bond3 mlx5_bond_3 1 1 fe80:0000:0000:0000:e29d:73ff:fe4b:3c0c v2 bond3 mlx5_bond_3 1 2 0000:0000:0000:0000:0000:ffff:1e8b:001e 30.139.0.30 v1 bond3 mlx5_bond_3 1 3 0000:0000:0000:0000:0000:ffff:1e8b:001e 30.139.0.30 v2 bond3 mlx5_bond_4 1 0 fe80:0000:0000:0000:e29d:73ff:fe49:d8a8 v1 bond4 mlx5_bond_4 1 1 fe80:0000:0000:0000:e29d:73ff:fe49:d8a8 v2 bond4 mlx5_bond_4 1 2 0000:0000:0000:0000:0000:ffff:1e8b:005a 30.139.0.90 v1 bond4 mlx5_bond_4 1 3 0000:0000:0000:0000:0000:ffff:1e8b:005a 30.139.0.90 v2 bond4 mlx5_bond_5 1 0 fe80:0000:0000:0000:e29d:73ff:fe49:d888 v1 bond5 mlx5_bond_5 1 1 fe80:0000:0000:0000:e29d:73ff:fe49:d888 v2 bond5 mlx5_bond_5 1 2 0000:0000:0000:0000:0000:ffff:1e8b:004e 30.139.0.78 v1 bond5 mlx5_bond_5 1 3 0000:0000:0000:0000:0000:ffff:1e8b:004e 30.139.0.78 v2 bond5 mlx5_bond_6 1 0 fe80:0000:0000:0000:e29d:73ff:fe49:dce8 v1 bond6 mlx5_bond_6 1 1 fe80:0000:0000:0000:e29d:73ff:fe49:dce8 v2 bond6 mlx5_bond_6 1 2 0000:0000:0000:0000:0000:ffff:1e8b:000a 30.139.0.10 v1 bond6 mlx5_bond_6 1 3 0000:0000:0000:0000:0000:ffff:1e8b:000a 30.139.0.10 v2 bond6 mlx5_bond_7 1 0 fe80:0000:0000:0000:e29d:73ff:fe49:d0d8 v1 bond7 mlx5_bond_7 1 1 fe80:0000:0000:0000:e29d:73ff:fe49:d0d8 v2 bond7 mlx5_bond_7 1 2 0000:0000:0000:0000:0000:ffff:1e8b:0042 30.139.0.66 v1 bond7 mlx5_bond_7 1 3 0000:0000:0000:0000:0000:ffff:1e8b:0042 30.139.0.66 v2 bond7 n_gids_found=32 ``` I'd appreciate any help.
bsakaguchi commented 2026-10-29 21:09:33 +00:00
Author
Owner
Copy link

@staryxchen commented on GitHub (Jul 24, 2025):

Hi @a4zhangfei @Time-Limit

The reason for this issue may be a system limitation that does not allow the NIC to access the memory of a GPU that is not in the same PCIe switch(BTW, we already has some workaround to ensure that this limitation is avoided, see #661 and #581 ). As an example, mlx5_bond_0 cannot access the memory of GPU1. In transfer_engine_bench, the gpu_id is always 0 by default, so you will see that only mlx5_bond_0 works. There are two ways to confirm the root cause of this problem:

  1. Run perftest and try different combinations with the -d parameter and the --use_cuda parameter, which will verify that the GDR function is correct;
  2. Run transfer_engine_bench and try different combinations of --device_name and --gpu_id to verify that mooncake is functioning correctly;
<!-- gh-comment-id:3112323473 --> @staryxchen commented on GitHub (Jul 24, 2025): Hi @a4zhangfei @Time-Limit The reason for this issue may be a system limitation that does not allow the NIC to access the memory of a GPU that is not in the same PCIe switch(BTW, we already has some workaround to ensure that this limitation is avoided, see #661 and #581 ). As an example, mlx5_bond_0 cannot access the memory of GPU1. In transfer_engine_bench, the gpu_id is always 0 by default, so you will see that only mlx5_bond_0 works. There are two ways to confirm the root cause of this problem: 1. Run [perftest](https://github.com/linux-rdma/perftest) and try different combinations with the -d parameter and the --use_cuda parameter, which will verify that the GDR function is correct; 2. Run transfer_engine_bench and try different combinations of --device_name and --gpu_id to verify that mooncake is functioning correctly;
bsakaguchi commented 2026-10-29 21:09:33 +00:00
Author
Owner
Copy link

@duzeyan commented on GitHub (Jul 25, 2025):

Hi @a4zhangfei @Time-Limit

The reason for this issue may be a system limitation that does not allow the NIC to access the memory of a GPU that is not in the same PCIe switch(BTW, we already has some workaround to ensure that this limitation is avoided, see #661 and #581 ). As an example, mlx5_bond_0 cannot access the memory of GPU1. In transfer_engine_bench, the gpu_id is always 0 by default, so you will see that only mlx5_bond_0 works. There are two ways to confirm the root cause of this problem:

  1. Run perftest and try different combinations with the -d parameter and the --use_cuda parameter, which will verify that the GDR function is correct;
  2. Run transfer_engine_bench and try different combinations of --device_name and --gpu_id to verify that mooncake is functioning correctly;

We have the same problem when using sglang. The "remote operation error" error occurs when the target node's --device_name and --gpu_id parameters in transfer_engine_bench don't match. Do you have any configuration to avoid this problem?

<!-- gh-comment-id:3116852763 --> @duzeyan commented on GitHub (Jul 25, 2025): > Hi [@a4zhangfei](https://github.com/a4zhangfei) [@Time-Limit](https://github.com/Time-Limit) > > The reason for this issue may be a system limitation that does not allow the NIC to access the memory of a GPU that is not in the same PCIe switch(BTW, we already has some workaround to ensure that this limitation is avoided, see [#661](https://github.com/kvcache-ai/Mooncake/pull/661) and [#581](https://github.com/kvcache-ai/Mooncake/pull/581) ). As an example, mlx5_bond_0 cannot access the memory of GPU1. In transfer_engine_bench, the gpu_id is always 0 by default, so you will see that only mlx5_bond_0 works. There are two ways to confirm the root cause of this problem: > > 1. Run [perftest](https://github.com/linux-rdma/perftest) and try different combinations with the -d parameter and the --use_cuda parameter, which will verify that the GDR function is correct; > 2. Run transfer_engine_bench and try different combinations of --device_name and --gpu_id to verify that mooncake is functioning correctly; We have the same problem when using sglang. The "remote operation error" error occurs when the target node's --device_name and --gpu_id parameters in transfer_engine_bench don't match. Do you have any configuration to avoid this problem?
bsakaguchi commented 2026-10-29 21:09:33 +00:00
Author
Owner
Copy link

@staryxchen commented on GitHub (Jul 25, 2025):

Hi @a4zhangfei @Time-Limit
The reason for this issue may be a system limitation that does not allow the NIC to access the memory of a GPU that is not in the same PCIe switch(BTW, we already has some workaround to ensure that this limitation is avoided, see #661 and #581 ). As an example, mlx5_bond_0 cannot access the memory of GPU1. In transfer_engine_bench, the gpu_id is always 0 by default, so you will see that only mlx5_bond_0 works. There are two ways to confirm the root cause of this problem:

  1. Run perftest and try different combinations with the -d parameter and the --use_cuda parameter, which will verify that the GDR function is correct;
  2. Run transfer_engine_bench and try different combinations of --device_name and --gpu_id to verify that mooncake is functioning correctly;

We have the same problem when using sglang. The "remote operation error" error occurs when the target node's --device_name and --gpu_id parameters in transfer_engine_bench don't match. Do you have any configuration to avoid this problem?

Unfortunately, mooncake (or any user-mode program) cannot remove this restriction. We can only try to avoid situations where RNIC and GPU are not match (btw, for optimal performance, we should also avoid such situations), as mentioned in the PR I mentioned earlier.

<!-- gh-comment-id:3117596025 --> @staryxchen commented on GitHub (Jul 25, 2025): > > Hi [@a4zhangfei](https://github.com/a4zhangfei) [@Time-Limit](https://github.com/Time-Limit) > > The reason for this issue may be a system limitation that does not allow the NIC to access the memory of a GPU that is not in the same PCIe switch(BTW, we already has some workaround to ensure that this limitation is avoided, see [#661](https://github.com/kvcache-ai/Mooncake/pull/661) and [#581](https://github.com/kvcache-ai/Mooncake/pull/581) ). As an example, mlx5_bond_0 cannot access the memory of GPU1. In transfer_engine_bench, the gpu_id is always 0 by default, so you will see that only mlx5_bond_0 works. There are two ways to confirm the root cause of this problem: > > > > 1. Run [perftest](https://github.com/linux-rdma/perftest) and try different combinations with the -d parameter and the --use_cuda parameter, which will verify that the GDR function is correct; > > 2. Run transfer_engine_bench and try different combinations of --device_name and --gpu_id to verify that mooncake is functioning correctly; > > We have the same problem when using sglang. The "remote operation error" error occurs when the target node's --device_name and --gpu_id parameters in transfer_engine_bench don't match. Do you have any configuration to avoid this problem? Unfortunately, mooncake (or any user-mode program) cannot remove this restriction. We can only try to avoid situations where RNIC and GPU are not match (btw, for optimal performance, we should also avoid such situations), as mentioned in the PR I mentioned earlier.
bsakaguchi commented 2026-10-29 21:09:33 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jul 14, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4965315640 --> @github-actions[bot] commented on GitHub (Jul 14, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
bsakaguchi commented 2026-10-29 21:09:33 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jul 21, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:5030321815 --> @github-actions[bot] commented on GitHub (Jul 21, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#112
No description provided.