[GH-ISSUE #2121] [Bug]: Now KV Cache stores in SSD failed in 910C PD separated deployment. #581

New issue

Closed

opened 2026-10-29 21:16:46 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:16:46 +00:00

Owner

Originally created by @xuyucheng220 on GitHub (May 17, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2121

Originally assigned to: @ascend-direct-dev on GitHub.

Bug Report

Now KV Cache stores in SSD failed in 910C PD separated deployment, using main of vllm-ascend and mooncake code version.

When the Env started and curl a request, the mooncake dummy client sever throw ERROR in logs:
I0517 00:12:50.115051 439100 real_client.cp:2227] Free ipc mem suc.
I0517 00:12:50.115468 439100 real_client.cpp:227] Free ipc mem suc.
I0517 00:12:50.115875 439100 real_client.cpp:227] Free ipc mem suc.
I0517 00:12:50.116258 439100 real_client.cpp:2227] Free ipc mem suc.
E0517 00:13:37.132926 439080 transfer_executor_base.cpp:263] Failed to connect to target: 172.16.0.49:20032, status: 503900, errmsg: [PID: 438288] 2026-05-17-00:13:37.121.879 Communication_Error_Bin
d_IP_Port(EI0020): Failed to enable listening for the NPU network adapter socket. Reason: The IP address 192.10.2.199 and port 1666 have already been bound.
Solution: Check whether the single-card multi-process scenario is used. If yes, configure the port number using the environment variable HCCL_NPU_sOCKET_PORT_RANGE.
TraceBack (most recent call last):
Call 1lm:HcclAdapter:GetInstance().HcclCommInitClusterInfoMemConfig( channelinfo_rank_table.c_str(), channel_info_.local_rank_id, &channel_info_.comm_config, &channel_info_comm) fail, r
et: Ox7[FUNC:Initialize][FILE:channel.cc][LINE:51]
Call chanel->Initialize(enable_use_fabric_mem) fail. Failed to init channel[FUNC:CreateChannel][FILE:channel_manager.cc][LINE:415]
Call channel_manager_->CreateChannel(channelinfo, channel, enable_use_fabric_mem) fail. [FUNC:CreateChannel][FILE:channel_msghandler.cc][LINE:337]
Call CreateChannel(channel_info, is_client, peer_channel_info) fail. Failed to create channel[FUNC:ConnectInfoProcess][FILE:channel_msg_handler.cc][LINE:411]
Call status.error_code fail. Failed to check peer process ret status, error code[5039], err msg[], local engine:172.16.0.202:20051, remote engine:172.16.0.49:20032, timeout:1000 ms.[FUNC:
DoConnect][FILE:channel_msg_handler.cc][LINE:568]
Call DoConnect(remoteengine, timeout_in_millis) fail. Failed to connect, local engine:172.16.0.202:20051, remote engine:172.16.0.49:20032, timeout:100 ms.[FUNC:Connect][FILE:channel_msgh
andler.cc][LINE:525]
Call msg_handler_.Connect(remote_engine.GetString(), timeout_in_millis) fail. Failed to connect, remote engine:172.16.0.49:20032, timeout:10000 ms[FUNC:Connect][FILE:adxl_inner_engine.cc][LI
NE : 445]

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @xuyucheng220 on GitHub (May 17, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2121 Originally assigned to: @ascend-direct-dev on GitHub. ### Bug Report Now KV Cache stores in SSD failed in 910C PD separated deployment, using main of vllm-ascend and mooncake code version. When the Env started and curl a request, the mooncake dummy client sever throw ERROR in logs: I0517 00:12:50.115051 439100 real_client.cp:2227] Free ipc mem suc. I0517 00:12:50.115468 439100 real_client.cpp:227] Free ipc mem suc. I0517 00:12:50.115875 439100 real_client.cpp:227] Free ipc mem suc. I0517 00:12:50.116258 439100 real_client.cpp:2227] Free ipc mem suc. E0517 00:13:37.132926 439080 transfer_executor_base.cpp:263] Failed to connect to target: 172.16.0.49:20032, status: 503900, errmsg: [PID: 438288] 2026-05-17-00:13:37.121.879 Communication_Error_Bin d_IP_Port(EI0020): Failed to enable listening for the NPU network adapter socket. Reason: The IP address 192.10.2.199 and port 1666 have already been bound. Solution: Check whether the single-card multi-process scenario is used. If yes, configure the port number using the environment variable HCCL_NPU_sOCKET_PORT_RANGE. TraceBack (most recent call last): Call 1lm:HcclAdapter:GetInstance().HcclCommInitClusterInfoMemConfig( channelinfo_rank_table.c_str(), channel_info_.local_rank_id, &channel_info_.comm_config, &channel_info_comm) fail, r et: Ox7[FUNC:Initialize][FILE:channel.cc][LINE:51] Call chanel->Initialize(enable_use_fabric_mem) fail. Failed to init channel[FUNC:CreateChannel][FILE:channel_manager.cc][LINE:415] Call channel_manager_->CreateChannel(channelinfo, channel, enable_use_fabric_mem) fail. [FUNC:CreateChannel][FILE:channel_msghandler.cc][LINE:337] Call CreateChannel(channel_info, is_client, peer_channel_info) fail. Failed to create channel[FUNC:ConnectInfoProcess][FILE:channel_msg_handler.cc][LINE:411] Call status.error_code fail. Failed to check peer process ret status, error code[5039], err msg[], local engine:172.16.0.202:20051, remote engine:172.16.0.49:20032, timeout:1000 ms.[FUNC: DoConnect][FILE:channel_msg_handler.cc][LINE:568] Call DoConnect(remoteengine, timeout_in_millis) fail. Failed to connect, local engine:172.16.0.202:20051, remote engine:172.16.0.49:20032, timeout:100 ms.[FUNC:Connect][FILE:channel_msgh andler.cc][LINE:525] Call msg_handler_.Connect(remote_engine.GetString(), timeout_in_millis) fail. Failed to connect, remote engine:172.16.0.49:20032, timeout:10000 ms[FUNC:Connect][FILE:adxl_inner_engine.cc][LI NE : 445] ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:16:46 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:16:47 +00:00

Author

Owner

@ascend-direct-dev commented on GitHub (May 18, 2026):

Thanks for the detailed report and logs.

This failure aligns with a single NIC / single-card multi-process limitation on the Ascend side: hixl does not currently support multiple processes sharing the same network adapter in this setup. The stack is trying to bind the same IP/port on the NPU network interface, which leads to the “already been bound” error you’re seeing. Proper support for this scenario is still pending; we’re waiting for a future release or upstream change that officially supports it.

Workaround: run Mooncake Store in FabricMem mode so the data path does not depend on the same NIC binding pattern that breaks under multi-process use. That’s the practical alternative until multi-process + single NIC is supported end-to-end.

@ascend-direct-dev commented on GitHub (May 18, 2026): Thanks for the detailed report and logs. This failure aligns with a single NIC / single-card multi-process limitation on the Ascend side: hixl does not currently support multiple processes sharing the same network adapter in this setup. The stack is trying to bind the same IP/port on the NPU network interface, which leads to the “already been bound” error you’re seeing. Proper support for this scenario is still pending; we’re waiting for a future release or upstream change that officially supports it. Workaround: run Mooncake Store in FabricMem mode so the data path does not depend on the same NIC binding pattern that breaks under multi-process use. That’s the practical alternative until multi-process + single NIC is supported end-to-end.

bsakaguchi referenced this issue

2026-10-29 21:29:20 +00:00

[PR #581] [MERGED] feat(topology): improve HCA selection by considering PCIe distance #1667