mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #1050] [Bug]: vllm ascend mooncake store initialize failed #345
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#345
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @liyu119 on GitHub (Nov 12, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1050
Bug Report
[version info]:
VLLM v0.11.0
Vllm ascend commitid:8a7154001e1b5f67e1edeef0ead5bb4bf49c4b5b
mooncake commitid:
723474f74c[mooncake.json]
{
"local_hostname": "localhost",
"metadata_server": "P2PHANDSHAKE",
"protocol": "ascend",
"use_ascend_direct": true,
"device_name": "",
"master_server_address": "masterip:50088",
"global_segment_size": 60000000000
}
[mooncake master]
mooncake_master --eviction_high_watermark_ratio 0.8 --eviction_ratio 0.05 --port 50088 --rpc_thread_num 16 --rpc_conn_timeout_seconds 60000
[detail description]
I tried to launch a vllm service on 8 ascend910B3 machines with 64 npus. I encountered the error like this. With the same env, code and scripts, I can launch a service on 4 machines successfully.
2025-11-12 11:16:11.129717 ERROR [58058] [coro_rpc_client.hpp:954] read rpc head failed, error msg:Operation aborted.. close the socket.value=125
E20251112 11:16:11.129794 58058 master_client.cpp:146] RPC call failed: Connection timed out
E20251112 11:16:11.129853 56431 client.cpp:1431] mount_segment_to_master_failed base=0x12c2000000 00 size=60000000000, error=RPC_FAIL
E20251112 11:16:11.129913 56431 pybind_client.cpp:233] Failed to mount segment: RPC_FAIL
ERROR 11-12 11:16:11 [mooncake_store.py:65] Initialize mooncake failed.
ERROR 11-12 11:16:11 [multiproc_executor.py:597] WorkerProc failed to start.
ERROR 11-12 11:16:11 [multiproc_executor.py:597] Traceback (most recent call last):
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm/v1/executor/multiproc_executor.py", line 571, in worker_main
ERROR 11-12 11:16:11 [multiproc_executor.py:597] worker = WorkerProc(*args, **kwargs)
ERROR 11-12 11:16:11 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm/v1/executor/multiproc_executor.py", line 430, in init
ERROR 11-12 11:16:11 [multiproc_executor.py:597] self.worker.init_device()
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm/worker/worker_base.py", line 259, in init_device
ERROR 11-12 11:16:11 [multiproc_executor.py:597] self.worker.init_device() # type: ignore
ERROR 11-12 11:16:11 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm_ascend/worker/worker_v1.py", line 209, in init_device
ERROR 11-12 11:16:11 [multiproc_executor.py:597] device = self._init_device()
ERROR 11-12 11:16:11 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm_ascend/worker/worker_v1.py", line 203, in _init_device
ERROR 11-12 11:16:11 [multiproc_executor.py:597] self._init_worker_distributed_environment()
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm_ascend/worker/worker_v1.py", line 388, in init_worker_distributed_environment
ERROR 11-12 11:16:11 [multiproc_executor.py:597] ensure_kv_transfer_initialized(self.vllm_con fig)
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 63, in ensure_kv_transfer initialized
ERROR 11-12 11:16:11 [multiproc_executor.py:597] _KV_CONNECTOR_AGENT = KVConnectorFactory.cre ate_connector(
ERROR 11-12 11:16:11 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 61, in create_connector
ERROR 11-12 11:16:11 [multiproc_executor.py:597] return connector_cls(config, role)
ERROR 11-12 11:16:11 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py", line 99, in ini t
ERROR 11-12 11:16:11 [multiproc_executor.py:597] KVConnectorFactory.create_connector(temp_con fig, role))
ERROR 11-12 11:16:11 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 61, in create_connector
ERROR 11-12 11:16:11 [multiproc_executor.py:597] return connector_cls(config, role)
ERROR 11-12 11:16:11 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm_ascend/distributed/mooncake/mooncake_store_connector_v1.py", line 42, in ini t
ERROR 11-12 11:16:11 [multiproc_executor.py:597] self.connector_worker = MooncakeEngine(
ERROR 11-12 11:16:11 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm_ascend/distributed/mooncake/mooncake_engine.py", line 72, in init
ERROR 11-12 11:16:11 [multiproc_executor.py:597] self.m_store = Mooncakestore(parallel_config )
ERROR 11-12 11:16:11 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^
ERROR 11-12 11:16:11 [multiproc_executor.py:597] File "/usr/local/python3.11.10/lib/python3.11/ site-packages/vllm_ascend/distributed/mooncake/mooncake_store.py", line 66, in init
ERROR 11-12 11:16:11 [multiproc_executor.py:597] raise RuntimeError(msg)
ERROR 11-12 11:16:11 [multiproc_executor.py:597] RuntimeError: Initialize mooncake failed.
Pls help me to find out the problem. If any more info needed, pls let me know. Thannks.
Before submitting...
@ykwd commented on GitHub (Nov 12, 2025):
According to the log, the communication between the client and the master failed during initialization. Possible causes include:
@liyu119 commented on GitHub (Nov 12, 2025):
Thank you so much for you kindly reply.
I have checked the master address in the config file and I am sure it is correct.
I can successfully launch a vllm service in any four of these machines. So I think there is no network problem.
By different versions, what exactly do you mean? Is the version of mooncake?
@ykwd commented on GitHub (Nov 12, 2025):
Yes. You can check whether the Mooncake master and the Mooncake used by vLLM are of the same version.
If the versions match, the issue might be caused by something else. The error message looks quite similar to the one discussed here: https://github.com/kvcache-ai/Mooncake/issues/1048
@liyu119 commented on GitHub (Nov 18, 2025):
I use the same docker image, so it's the same version. And the problem occasionally disappear. But for most time, it fails. I use protocol ascend, so i think it's different with issue #1048 . Is there any way to get more logs or to find what's wrong？
@ykwd commented on GitHub (Nov 19, 2025):
According to the error logs, the RPC communication between the mooncake client and the master failed during the startup phase. To make debugging easier, you can temporarily set vLLM aside and focus only on Mooncake. The approach is to first start the Mooncake master and meta services, and then start a standalone Mooncake client to see whether it can launch normally.
You can start the standalone client with:
For more details, see the “Launch Mooncake store service (Optional)” section in: this doc
@liyu119 commented on GitHub (Nov 19, 2025):
I tried to start the standalone client on every machine. And it runs correctly.
@ykwd commented on GitHub (Nov 20, 2025):
That is indeed strange. According to the logs, the RPC failed when the client tried to mount the segment on the master, and the failure reason was a timeout cancellation. In theory, the client should have already connected to the master before reaching the “mount segment” step, which means the client can reach the master over the network.
I don’t have any ideas at the moment.
@ascend-direct-dev Could you help take a look?
@ykwd commented on GitHub (Nov 21, 2025):
I’ve heard that increasing the number of master threads can reduce the likelihood of this issue. Since the error is a timeout, it looks as if the master is fully occupied and unable to respond to RPC requests. However, this is still just the setup phase, before any actual workload begins, so in theory this shouldn’t be happening.
I have a hypothesis: maybe the mount segment request is taking too long and blocking the master. Perhaps you could try changing
"global_segment_size": 60000000000(60 GB) to600000000(600 MB) and see if it helps.If this really is the cause, it suggests that the current mount-segment operation may have a performance bug.
@liyu119 commented on GitHub (Nov 21, 2025):
Yes, it works. But decreasing the size of cache is not what we want. Is there any other workaround?
@ykwd commented on GitHub (Nov 21, 2025):
Then it’s quite possible that mount-segment RPC is the problem. But in theory, the mount-segment RPC should be very fast, so maybe something in the implementation is suboptimal. I’ll test it when I have time. If you have time as well, you’re welcome to work together on solving this.
@liyu119 commented on GitHub (Dec 8, 2025):
hi, i'd like to know is there any update on this issue? @ykwd
@github-actions[bot] commented on GitHub (Jun 19, 2026):
This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
@github-actions[bot] commented on GitHub (Jun 26, 2026):
Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.