mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #1088] [Usage]: Failed to complete transfers after 60 seconds for batch #355
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#355
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @zgzicon on GitHub (Nov 20, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1088
Describe your usage question
When running the SGLang service with HiCache and Mooncake, the following issue occurs:
"I1120 11:09:23.594488 203018 pybind_client.cpp:218] Mounting segment: 13421772800 bytes, 13421772800 of 13421772800
I1120 11:09:23.594596 203019 pybind_client.cpp:218] Mounting segment: 13421772800 bytes, 13421772800 of 13421772800
[ TP2] Connect to Mooncake store successfully.
[ TP3] Connect to Mooncake store successfully.
[ TP1] Connect to Mooncake store successfully.
[ TP0] Connect to Mooncake store successfully.
[ TP0] Mooncake store warmup successfully.
E1120 11:10:27.133715 203020 transfer_task.cpp:305] Failed to complete transfers after 60 seconds for batch 2586731328
[ TP2] An error occurred while loading the configuration:
[ TP2] Scheduler hit an exception: Traceback (most recent call last):
File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2712, in run_scheduler_process
scheduler = Scheduler(
^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 423, in init
self.init_memory_pool_and_cache()
File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 736, in init_memory_pool_and_cache
self.tree_cache = HiRadixCache(
^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 100, in init
self.cache_controller = HiCacheController(
^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 290, in init
self.storage_backend = StorageBackendFactory.create_backend(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/backend_factory.py", line 94, in create_backend
return cls._create_builtin_backend(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/backend_factory.py", line 166, in _create_builtin_backend
backend = backend_class(storage_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_store.py", line 206, in init
self.warmup()
File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_store.py", line 259, in warmup
assert self.store.put(warmup_key, warmup_value) == 0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError"
No matter how the Mooncake parameters are adjusted, the above error log always appears. What is the issue and how can it be resolved? Thank you for your response.
Before submitting a new issue...
@ykwd commented on GitHub (Nov 21, 2025):
According to the error log, Mooncake failed during the warmup phase of its initialization. During warmup, Mooncake attempts to put an object and then get it to verify that the operation succeeds.
The warmup failure was caused by a data transfer timeout, which means that the connection between the Mooncake client, master, and meta service is functioning correctly—the client successfully obtained the metadata for the data to be transferred. However, the end-to-end data transfer itself failed.
Could you share how your deployment is set up?
If you want to debug the issue, you can start by checking whether network communication between all nodes is working properly.
If Mooncake is configured to use RDMA, then you should focus on verifying whether RDMA communication is functioning as expected.
@zgzicon commented on GitHub (Nov 21, 2025):
Thank you for your prompt reply. I have verified that the network connectivity and RDMA are functioning properly.
My deployment environment consists of 4 machines, each equipped with 8 H20 GPUs, and the model is 32B. The first machine serves as the master node and has a sglang hicache instance deployed (successfully started and operational). The other three machines only have sglang deployed, with hicache mooncake enabled on each, but they all failed to start. The failure reason is the same across all three machines, as shown in the logs above, indicating that a certain TP (Tensor Parallelism) process encountered a "Failed to complete transfers after 60 seconds for batch" error.
The mooncake environment variables are configured as follows:
export MOONCAKE_TE_META_DATA_SERVER="http://xxxx:8080/metadata"
export MOONCAKE_MASTER="xxxx:50051"
export MOONCAKE_PROTOCOL="rdma"
export MOONCAKE_DEVICE="mlx5_8"
export MOONCAKE_GLOBAL_SEGMENT_SIZE=53687091200
Thank you again for your response.
@ykwd commented on GitHub (Nov 21, 2025):
The information you provided is very detailed. I think we can verify the issue from the following two aspects:
Try using the TCP protocol.
If the service can start normally with TCP, it means the TCP network is fine and other configurations are correct — only the RDMA network is not working.
Test whether RDMA communication between the servers is working properly. For example:
On Server A:
On Server B:
@zgzicon commented on GitHub (Nov 21, 2025):
I switched to TCP but encountered a similar startup failure, though with different error logs as follows:
“I1121 03:36:30.375252 242273 transfer_engine.cpp:146] Transfer Engine RPC using new RPC mapping, listening on xxxx:15989 I1121 03:36:30.376376 242273 client.cpp:270] Transfer engine auto discovery is disabled for protocol: tcp I1121 03:36:30.376662 242273 tcp_transport.cpp:299] TcpTransport: listen on port 16100 I1121 03:36:30.378968 242273 pybind_client.cpp:218] Mounting segment: 13421772800 bytes, 13421772800 of 13421772800 [2025-11-21 03:36:30 TP2] Connect to Mooncake store successfully. E1121 03:36:30.461072 242273 transfer_task.cpp:528] Failed to submit all transfers, error code is NotSupportedTransport E1121 03:36:30.461088 242273 client.cpp:1601] Failed to submit transfer operation [2025-11-21 03:36:30 TP2] An error occurred while loading the configuration: [2025-11-21 03:36:30 TP2] Scheduler hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2712, in run_scheduler_process scheduler = Scheduler(”Subsequently, I used ib_write_bw for testing and confirmed that RDMA between the servers starts up normally.
Finally, I restarted sglang on all four machines (Mooncake still using RDMA), this time starting the second machine first. I found that the second machine's service started successfully, while the other three failed (it seems that only one machine can succeed at a same time). The error logs from the other three machines still showed: "transfer_task.cpp:305] Failed to complete transfers after 60 seconds for batch 2556715136". This seems quite strange. May I ask for your advice on how to resolve this? Thank you for your response.
@alogfans commented on GitHub (Nov 21, 2025):
@zgzicon
ib_write_bw -d mlx5_8according to your config file, and@ykwd commented on GitHub (Nov 21, 2025):
The cause of this error may be that the remote instance’s protocol is not set to TCP. You should check whether the configurations of all instances are consistent.
The first instance starts successfully while the others fail. This happens because when an instance starts, it contributes a chunk of local memory—of size MOONCAKE_GLOBAL_SEGMENT_SIZE—to the distributed memory pool. When the first instance starts, it is the only contributor to the pool, so the warmup operation uses a local-to-local data transfer. However, since the warmup process does not prioritize putting data locally, starting from the second instance, the operation may involve remote data transfer, which leads to failure.
This indicates that local data transfer works correctly, and the issue is likely related to RDMA communication between the servers.
@zgzicon commented on GitHub (Nov 21, 2025):
I restarted mooncake_master and relaunched sglang instances on all four machines using TCP, but the result remains that only one machine can start successfully while the other three fail to start. The logs are as shown in the figure below:
Additionally, I verified the RDMA connectivity between services using "ib_write_bw -d mlx5_8" as shown in the figure below:
Could you please advise what other issues might be causing this problem? Thank you for your help.
@ykwd commented on GitHub (Nov 21, 2025):
I have another hypothesis: could it be that the MOONCAKE_LOCAL_HOSTNAME parameter is set to localhost instead of the actual IP address? This address is used for data transfers between clients, and if it’s set incorrectly, the client-to-client data transfer will time out. For the hostname, i.e., localhost, Mooncake internally appends a random port number, so the first client can communicate with itself, but the other clients won’t be able to connect.
I tested this locally, and if this parameter is set incorrectly, the error is also a timeout.
@weibingo commented on GitHub (Jan 23, 2026):
@ykwd I encountered an issue where sglang integrates with mooncake. The service can start up, and it runs for a while with data transmission working. But after some time, the storage num on metadata stops updating, and it prints the log mentioned above, resulting in a 60s timeout transmission failure.
@weibingo commented on GitHub (Jan 23, 2026):
@ykwd can i set client put/get operation timeout param ?
@ykwd commented on GitHub (Jan 23, 2026):
@weibingo The 60-second timeout is more likely caused by deployment or environmental issues, rather than problems with the execution of the put/get operations.
@weibingo commented on GitHub (Jan 23, 2026):
@ykwd log like:
Failed to complete transfers after 60 seconds for batch xxx
Transfer failed for key xxxxx_k: TRANSFER_FAIL(Transfer 0 failed)
Put operation failed for key xxxxx_k, context: TRANSFER_FAIL: Transfer 0 failed;
@chenkaiyue commented on GitHub (Mar 5, 2026):
I encountered a similar issue: after batch_set_v1(), the BatchPutStart request jumped from 0 to 8, but PutEnd remained at 0—indicating that RDMA transmission started but couldn’t complete. I’m using a single machine where sglang and Mooncake Store are co-located.
@riZZZhik commented on GitHub (Apr 28, 2026):
@zgzicon @chenkaiyue Hello!
Have you had any luck resolving this issue?
I’m facing the same problem with the latest SGLang and Mooncake versions.
@github-actions[bot] commented on GitHub (Jul 28, 2026):
This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
@github-actions[bot] commented on GitHub (Aug 4, 2026):
Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.