kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #97] target node error #53

New issue

Closed

opened 2026-10-29 21:08:27 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-29 21:08:27 +00:00

Owner

Originally created by @zx-ai on GitHub (Feb 11, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/97

./transfer_engine_bench --mode=target
--metadata_server=10.0.0.1:2379
--local_server_name=10.0.0.2:12345
--device_name=erdma_0
error as follow:
unsuppored mode: must be 'inititator' or 'target'
how to fix it?

Originally created by @zx-ai on GitHub (Feb 11, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/97 ./transfer_engine_bench --mode=target \ --metadata_server=10.0.0.1:2379 \ --local_server_name=10.0.0.2:12345 \ --device_name=erdma_0 error as follow: unsuppored mode: must be 'inititator' or 'target' how to fix it?

bsakaguchi

2026-10-29 21:08:27 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:08:29 +00:00

Author

Owner

@alogfans commented on GitHub (Feb 12, 2025):

You can try to rebuild Mooncake using the main branch, and check your command typo. If you need to enter commands on separate lines, you need to mark the end of each line with a backslash (see run-examples.md)

@alogfans commented on GitHub (Feb 12, 2025): You can try to rebuild Mooncake using the main branch, and check your command typo. If you need to enter commands on separate lines, you need to mark the end of each line with a backslash (see run-examples.md)

bsakaguchi commented

2026-10-29 21:08:29 +00:00

Author

Owner

@zx-ai commented on GitHub (Feb 12, 2025):

thank you so much! I update the gflags，then I build transfer_engine sucessfully.
After compiling (without enabling USE_CUDA), I successfully ran a demo on the NPU device (transferring data from A machine's DRAM to B machine's DRAM).
In order to verify if data can be transferred from the NPU memory of machine A to the NPU memory of machine B using the transfer_engine on NPU devices.So, I attempted to modify the allocateMemoryPool() function on machine A, using the aclrtMalloc function to allocate memory on the NPU device.
static void *allocateMemoryPool(size_t size, int socket_id,
bool from_vram = false) {
#ifdef USE_ATLAS
if (from_vram) {
void *d_buf;
aclrtMalloc(&d_buf, size, ACL _MEM_MALLOC_HUGE_FIRST);
return d_buf;
}
#endif
return numa_alloc_onnode(size, socket_id);
}
Upon running the modified code, I encountered an error:
E0212 17:34:25.448876 212704 rdma_context.cpp:227] Fail to register memory 0x14ef5930: Cannot allocate memory [12]
*** Check failure stack trace: ***F0212 17:34:25.448904 212704 transfer_engine_bench.cpp:286] Assert failed: !rc

I am curious as to why GPU memory can be registered using the ibv_reg_mr() function, whereas NPU memory cannot be successfully registered. And Is there any way to solve this problem? Currently, can the main version or the V2.0 tag solve this problem? Will this project support deployment on NPU devices in the future?
I am very much looking forward to your response. Thank you again.

@zx-ai commented on GitHub (Feb 12, 2025): thank you so much! I update the gflags，then I build transfer_engine sucessfully. After compiling (without enabling USE_CUDA), I successfully ran a demo on the NPU device (transferring data from A machine's DRAM to B machine's DRAM). In order to verify if data can be transferred from the NPU memory of machine A to the NPU memory of machine B using the transfer_engine on NPU devices.So, I attempted to modify the allocateMemoryPool() function on machine A, using the aclrtMalloc function to allocate memory on the NPU device. static void *allocateMemoryPool(size_t size, int socket_id, bool from_vram = false) { #ifdef USE_ATLAS if (from_vram) { void *d_buf; aclrtMalloc(&d_buf, size, ACL _MEM_MALLOC_HUGE_FIRST); return d_buf; } #endif return numa_alloc_onnode(size, socket_id); } Upon running the modified code, I encountered an error: E0212 17:34:25.448876 212704 rdma_context.cpp:227] **Fail to register memory 0x14ef5930: Cannot allocate memory [12]** *** Check failure stack trace: ***F0212 17:34:25.448904 212704 transfer_engine_bench.cpp:286] Assert failed: !rc I am curious as to why GPU memory can be registered using the ibv_reg_mr() function, whereas NPU memory cannot be successfully registered. And Is there any way to solve this problem? Currently, can the main version or the V2.0 tag solve this problem? Will this project support deployment on NPU devices in the future? I am very much looking forward to your response. Thank you again.

bsakaguchi commented

2026-10-29 21:08:29 +00:00

Author

Owner

@alogfans commented on GitHub (Feb 13, 2025):

The ibv_reg_mr() is used to register memory allocated from DRAM, such asmalloc() and new keywords. With the CUDA support, it can be also used to register CUDA memory.

You can use the ib_write_bw utility to check if there are error(s) in data transfer. If have, it indicates that the installed RDMA driver does not support register NPU memory. You can contact with the NPU vendor for more information (e.g., special APIs, or upgrade the firmware or driver).

@alogfans commented on GitHub (Feb 13, 2025): The `ibv_reg_mr()` is used to register memory allocated from DRAM, such as`malloc()` and `new` keywords. With the CUDA support, it can be also used to register CUDA memory. You can use the `ib_write_bw` utility to check if there are error(s) in data transfer. If have, it indicates that the installed RDMA driver does not support register NPU memory. You can contact with the NPU vendor for more information (e.g., special APIs, or upgrade the firmware or driver).

bsakaguchi commented

2026-10-29 21:08:30 +00:00

Author

Owner

@Huixxi commented on GitHub (Feb 14, 2025):

./transfer_engine_bench --mode=target --metadata_server=10.0.0.1:2379 --local_server_name=10.0.0.2:12345 --device_name=erdma_0 error as follow: unsuppored mode: must be 'inititator' or 'target' how to fix it?

Shouldn't 10.0.0.1 need to replace with your own machine's intra-net ip?

@Huixxi commented on GitHub (Feb 14, 2025): > ./transfer_engine_bench --mode=target --metadata_server=10.0.0.1:2379 --local_server_name=10.0.0.2:12345 --device_name=erdma_0 error as follow: unsuppored mode: must be 'inititator' or 'target' how to fix it? Shouldn't 10.0.0.1 need to replace with your own machine's intra-net ip?

bsakaguchi commented

2026-10-29 21:08:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 22, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 22, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:08:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 29, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 29, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.