kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #1055] [Bug]: Worker: Process failed for slice #346

New issue
Closed
opened 2026-10-29 21:13:07 +00:00 by bsakaguchi · 8 comments
bsakaguchi commented 2026-10-29 21:13:07 +00:00
Owner
Copy link

Originally created by @BilyZ98 on GitHub (Nov 13, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1055

Bug Report

I started 8 vll instances and run vllm benchmark. and got this transport retry counter exceeded error in P/D separation testing. 4 prefill node and 4 decode node. Why is it happen?
The source code is in worker_pool.cpp.

Command to start the benchmark script
MODEL=./Qwen2-7B-Instruct bash ./benchmarks.sh

mooncake.json

{
"local_hostname": "172.16.37.132",
"metadata_server": "http://172.16.37.132:8081/metadata",
"protocol": "tcp",
"device_name": "",
"master_server_address": "172.16.37.132:10001"
}

prefill started log

NFO 11-13 17:04:34 [mooncake_store_connector.py:52] Initializing KVStoreConnector under kv_transfer_config KVTransferConfig(kv_connector='MooncakeStoreConnector', engine_id='3a79ef8b-6c4c-48ab-b52a-f827f4dd14b2', kv_buffer_device='cuda', kv_buffer_size=1000000000.0, kv_role='kv_producer', kv_rank=None, kv_parallel_size=1, kv_ip='127.0.0.1', kv_port=14579, kv_connector_extra_config={}, kv_connector_module_path=None)
INFO 11-13 17:04:35 [mooncake_store.py:84] Mooncake Configuration loaded successfully.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251113 17:04:35.119301 22885017610048 client_metric.cpp:76] Client metrics enabled (default enabled)
I20251113 17:04:35.120329 22885017610048 transfer_engine.cpp:486] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I20251113 17:04:35.120934 22885017610048 client.cpp:46] client_id=6648689678256049533-9781957128945823390
I20251113 17:04:35.120953 22885017610048 client.cpp:54] Client metrics enabled but reporting disabled (interval=0)
I20251113 17:04:35.121084 22885017610048 master_client.cpp:214] Connecting to master at 172.16.37.132:10001 from mooncake::ErrorCode mooncake::MasterClient::Connect(const string&)
E20251113 17:04:35.123646 22885017610048 client.cpp:318] Failed to get fsdir from master
I20251113 17:04:35.123687 22885017610048 transfer_engine.cpp:91] Transfer Engine parseHostNameWithPort. server_name: 172.16.37.132 port: 13201
I20251113 17:04:35.125404 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface eth0 with IP 172.16.37.132
I20251113 17:04:35.125415 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib0 with IP 99.72.99.1
I20251113 17:04:35.125420 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib1 with IP 99.72.101.1
I20251113 17:04:35.125423 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib2 with IP 99.72.102.1
I20251113 17:04:35.125427 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib3 with IP 99.72.103.1
I20251113 17:04:35.125431 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib4 with IP 99.72.14.1
I20251113 17:04:35.125434 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib5 with IP 99.72.104.1
I20251113 17:04:35.125438 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib6 with IP 99.72.105.1
I20251113 17:04:35.125442 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib7 with IP 99.72.106.1
I20251113 17:04:35.125446 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib8 with IP 99.72.107.1
I20251113 17:04:35.125470 22885017610048 transfer_engine.cpp:146] Transfer Engine RPC using new RPC mapping, listening on 172.16.37.132:16532
I20251113 17:04:35.127012 22885017610048 transfer_engine.cpp:185] Auto-discovering topology...
I20251113 17:04:35.128735 22885017610048 transfer_engine.cpp:200] Topology discovery complete. Found 9 HCAs.
I20251113 17:04:35.135474 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_5/
I20251113 17:04:35.137094 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_5, LID: 228, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9c:63:c0:03:00:1a:9c:70
I20251113 17:04:35.144284 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_6/
I20251113 17:04:35.145681 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_6, LID: 230, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9c:63:c0:03:00:0b:42:92
I20251113 17:04:35.152539 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_7/
I20251113 17:04:35.153860 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_7, LID: 227, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9c:63:c0:03:00:1a:9b:b0
I20251113 17:04:35.160384 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_8/
I20251113 17:04:35.161777 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_8, LID: 229, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9c:63:c0:03:00:1a:9d:3c
I20251113 17:04:35.184781 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_0/
I20251113 17:04:35.237633 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_0, LID: 221, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9c:63:c0:03:00:0b:3f:1a
I20251113 17:04:35.245813 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_1/
I20251113 17:04:35.247700 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_1, LID: 222, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9c:63:c0:03:00:0b:42:2a
I20251113 17:04:35.253928 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_2/
I20251113 17:04:35.255509 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_2, LID: 370, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:5c:25:73:03:00:4b:fa:3a
I20251113 17:04:35.278148 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_3/
I20251113 17:04:35.278872 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_3, LID: 351, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:5c:25:73:03:00:4b:fa:3b
I20251113 17:04:35.284942 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_4/
I20251113 17:04:35.286494 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_4, LID: 117, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:58:a2:e1:03:00:c9:4a:cc
I20251113 17:04:35.287515 22885017610048 client.cpp:233] Transfer engine auto discovery is enabled for protocol: tcp
I20251113 17:04:35.765493 22885017610048 pybind_client.cpp:216] Mounting segment: 3355443200 bytes, 3355443200 of 3355443200
INFO 11-13 17:04:37 [model_runner.py:1171] Starting to load model

prefill error log

E20251113 17:07:39.324105 22859357349440 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x14c97801f040, length: 65536, dest_addr: 0x1474ac020000, local_nic: mlx5_8, peer_nic: 172.16.37.132:14127@mlx5_4, dest_rkey: 2359727, retry_cnt: 0): transport retry counter exceeded
INFO 11-13 17:07:39 [engine.py:317] Added request cmpl-e53723bee34b44c08fa87f39871563b9-0.
E20251113 17:07:43.082274 22859357349440 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x14c978fff030, length: 65536, dest_addr: 0x1474ad000000, local_nic: mlx5_8, peer_nic: 172.16.37.132:14127@mlx5_4, dest_rkey: 2359727, retry_cnt: 0): transport retry counter exceeded
E20251113 17:07:43.149099 22860183537216 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x14c977fff040, length: 65536, dest_addr: 0x146e85000000, local_nic: mlx5_5, peer_nic: 172.16.37.132:12704@mlx5_4, dest_rkey: 2358702, retry_cnt: 0): transport retry counter exceeded
E20251113 17:07:43.149622 22860183537216 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x14c97c00f000, length: 65536, dest_addr: 0x1474ae010000, local_nic: mlx5_5, peer_nic: 172.16.37.132:14127@mlx5_4, dest_rkey: 2359727, retry_cnt: 0): transport retry counter exceeded
E20251113 17:07:46.691009 22860170929728 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x14c97801f040, length: 65536, dest_addr: 0x1474af020000, local_nic: mlx5_7, peer_nic: 172.16.37.132:14127@mlx5_4, dest_rkey: 2359727, retry_cnt: 0): transport retry counter exceeded
@@@

Before submitting...

  • Ensure you searched for relevant issues and read the [documentation]
Originally created by @BilyZ98 on GitHub (Nov 13, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1055 ### Bug Report I started 8 vll instances and run vllm benchmark. and got this transport retry counter exceeded error in P/D separation testing. 4 prefill node and 4 decode node. Why is it happen? The source code is in worker_pool.cpp. Command to start the benchmark script MODEL=./Qwen2-7B-Instruct bash ./benchmarks.sh mooncake.json ``` { "local_hostname": "172.16.37.132", "metadata_server": "http://172.16.37.132:8081/metadata", "protocol": "tcp", "device_name": "", "master_server_address": "172.16.37.132:10001" } ``` prefill started log ``` NFO 11-13 17:04:34 [mooncake_store_connector.py:52] Initializing KVStoreConnector under kv_transfer_config KVTransferConfig(kv_connector='MooncakeStoreConnector', engine_id='3a79ef8b-6c4c-48ab-b52a-f827f4dd14b2', kv_buffer_device='cuda', kv_buffer_size=1000000000.0, kv_role='kv_producer', kv_rank=None, kv_parallel_size=1, kv_ip='127.0.0.1', kv_port=14579, kv_connector_extra_config={}, kv_connector_module_path=None) INFO 11-13 17:04:35 [mooncake_store.py:84] Mooncake Configuration loaded successfully. WARNING: Logging before InitGoogleLogging() is written to STDERR I20251113 17:04:35.119301 22885017610048 client_metric.cpp:76] Client metrics enabled (default enabled) I20251113 17:04:35.120329 22885017610048 transfer_engine.cpp:486] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I20251113 17:04:35.120934 22885017610048 client.cpp:46] client_id=6648689678256049533-9781957128945823390 I20251113 17:04:35.120953 22885017610048 client.cpp:54] Client metrics enabled but reporting disabled (interval=0) I20251113 17:04:35.121084 22885017610048 master_client.cpp:214] Connecting to master at 172.16.37.132:10001 from mooncake::ErrorCode mooncake::MasterClient::Connect(const string&) E20251113 17:04:35.123646 22885017610048 client.cpp:318] Failed to get fsdir from master I20251113 17:04:35.123687 22885017610048 transfer_engine.cpp:91] Transfer Engine parseHostNameWithPort. server_name: 172.16.37.132 port: 13201 I20251113 17:04:35.125404 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface eth0 with IP 172.16.37.132 I20251113 17:04:35.125415 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib0 with IP 99.72.99.1 I20251113 17:04:35.125420 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib1 with IP 99.72.101.1 I20251113 17:04:35.125423 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib2 with IP 99.72.102.1 I20251113 17:04:35.125427 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib3 with IP 99.72.103.1 I20251113 17:04:35.125431 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib4 with IP 99.72.14.1 I20251113 17:04:35.125434 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib5 with IP 99.72.104.1 I20251113 17:04:35.125438 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib6 with IP 99.72.105.1 I20251113 17:04:35.125442 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib7 with IP 99.72.106.1 I20251113 17:04:35.125446 22885017610048 transfer_metadata_plugin.cpp:1100] Found active interface ib8 with IP 99.72.107.1 I20251113 17:04:35.125470 22885017610048 transfer_engine.cpp:146] Transfer Engine RPC using new RPC mapping, listening on 172.16.37.132:16532 I20251113 17:04:35.127012 22885017610048 transfer_engine.cpp:185] Auto-discovering topology... I20251113 17:04:35.128735 22885017610048 transfer_engine.cpp:200] Topology discovery complete. Found 9 HCAs. I20251113 17:04:35.135474 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_5/ I20251113 17:04:35.137094 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_5, LID: 228, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9c:63:c0:03:00:1a:9c:70 I20251113 17:04:35.144284 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_6/ I20251113 17:04:35.145681 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_6, LID: 230, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9c:63:c0:03:00:0b:42:92 I20251113 17:04:35.152539 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_7/ I20251113 17:04:35.153860 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_7, LID: 227, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9c:63:c0:03:00:1a:9b:b0 I20251113 17:04:35.160384 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_8/ I20251113 17:04:35.161777 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_8, LID: 229, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9c:63:c0:03:00:1a:9d:3c I20251113 17:04:35.184781 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_0/ I20251113 17:04:35.237633 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_0, LID: 221, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9c:63:c0:03:00:0b:3f:1a I20251113 17:04:35.245813 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_1/ I20251113 17:04:35.247700 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_1, LID: 222, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:9c:63:c0:03:00:0b:42:2a I20251113 17:04:35.253928 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_2/ I20251113 17:04:35.255509 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_2, LID: 370, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:5c:25:73:03:00:4b:fa:3a I20251113 17:04:35.278148 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_3/ I20251113 17:04:35.278872 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_3, LID: 351, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:5c:25:73:03:00:4b:fa:3b I20251113 17:04:35.284942 22885017610048 rdma_context.cpp:491] Find best gid index: 0 on mlx5_4/ I20251113 17:04:35.286494 22885017610048 rdma_context.cpp:129] RDMA device: mlx5_4, LID: 117, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:58:a2:e1:03:00:c9:4a:cc I20251113 17:04:35.287515 22885017610048 client.cpp:233] Transfer engine auto discovery is enabled for protocol: tcp I20251113 17:04:35.765493 22885017610048 pybind_client.cpp:216] Mounting segment: 3355443200 bytes, 3355443200 of 3355443200 INFO 11-13 17:04:37 [model_runner.py:1171] Starting to load model ``` prefill error log ``` E20251113 17:07:39.324105 22859357349440 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x14c97801f040, length: 65536, dest_addr: 0x1474ac020000, local_nic: mlx5_8, peer_nic: 172.16.37.132:14127@mlx5_4, dest_rkey: 2359727, retry_cnt: 0): transport retry counter exceeded INFO 11-13 17:07:39 [engine.py:317] Added request cmpl-e53723bee34b44c08fa87f39871563b9-0. E20251113 17:07:43.082274 22859357349440 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x14c978fff030, length: 65536, dest_addr: 0x1474ad000000, local_nic: mlx5_8, peer_nic: 172.16.37.132:14127@mlx5_4, dest_rkey: 2359727, retry_cnt: 0): transport retry counter exceeded E20251113 17:07:43.149099 22860183537216 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x14c977fff040, length: 65536, dest_addr: 0x146e85000000, local_nic: mlx5_5, peer_nic: 172.16.37.132:12704@mlx5_4, dest_rkey: 2358702, retry_cnt: 0): transport retry counter exceeded E20251113 17:07:43.149622 22860183537216 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x14c97c00f000, length: 65536, dest_addr: 0x1474ae010000, local_nic: mlx5_5, peer_nic: 172.16.37.132:14127@mlx5_4, dest_rkey: 2359727, retry_cnt: 0): transport retry counter exceeded E20251113 17:07:46.691009 22860170929728 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x14c97801f040, length: 65536, dest_addr: 0x1474af020000, local_nic: mlx5_7, peer_nic: 172.16.37.132:14127@mlx5_4, dest_rkey: 2359727, retry_cnt: 0): transport retry counter exceeded @@@ ``` ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]
bsakaguchi 2026-10-29 21:13:07 +00:00
bsakaguchi commented 2026-10-29 21:13:08 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (Nov 13, 2025):

Try export MC_MS_AUTO_DISC=0？

<!-- gh-comment-id:3527478115 --> @ShangmingCai commented on GitHub (Nov 13, 2025): Try `export MC_MS_AUTO_DISC=0`？
bsakaguchi commented 2026-10-29 21:13:08 +00:00
Author
Owner
Copy link

@BilyZ98 commented on GitHub (Nov 13, 2025):

Thanks for reply. This helps solve the ib transport retyr counter exceeded error.

Do you know how to fix issues above if I set to use infiniband NIC ?
What changes should I make if I set mooncake.json like this ?
The error above occurs as well if I set to use rdma protocol and ib NIC.

{
"local_hostname": "172.16.37.132",
"metadata_server": "http://172.16.37.132:8081/metadata",
"protocol": "rdma",
"device_name": "mlx5_0",
"master_server_address": "172.16.37.132:10001"
}
<!-- gh-comment-id:3527568748 --> @BilyZ98 commented on GitHub (Nov 13, 2025): Thanks for reply. This helps solve the ib transport retyr counter exceeded error. Do you know how to fix issues above if I set to use infiniband NIC ? What changes should I make if I set mooncake.json like this ? The error above occurs as well if I set to use rdma protocol and ib NIC. ``` { "local_hostname": "172.16.37.132", "metadata_server": "http://172.16.37.132:8081/metadata", "protocol": "rdma", "device_name": "mlx5_0", "master_server_address": "172.16.37.132:10001" } ```
bsakaguchi commented 2026-10-29 21:13:08 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (Nov 13, 2025):

Use show_gids and nvidia-smi topo -m to get ib and topo info and paste them to LLM to let it help you find the right device_name config.

<!-- gh-comment-id:3527714107 --> @ShangmingCai commented on GitHub (Nov 13, 2025): Use `show_gids` and `nvidia-smi topo -m` to get ib and topo info and paste them to LLM to let it help you find the right device_name config.
bsakaguchi commented 2026-10-29 21:13:08 +00:00
Author
Owner
Copy link

@XiaotaoChen commented on GitHub (Nov 17, 2025):

@BilyZ98 Hi, have you solve the error? I try sglang remote-instance with transfer-engine backend， raise the same error. detail info as: https://github.com/sgl-project/sglang/issues/13380 shows. And export export MC_MS_AUTO_DISC=0 didnt't work. Do you have any advices?

<!-- gh-comment-id:3540142739 --> @XiaotaoChen commented on GitHub (Nov 17, 2025): @BilyZ98 Hi, have you solve the error? I try sglang remote-instance with `transfer-engine` backend， raise the same error. detail info as: https://github.com/sgl-project/sglang/issues/13380 shows. And export `export MC_MS_AUTO_DISC=0` didnt't work. Do you have any advices?
bsakaguchi commented 2026-10-29 21:13:08 +00:00
Author
Owner
Copy link

@BilyZ98 commented on GitHub (Nov 21, 2025):

Hi @XiaotaoChen ,
I checked the link you gave and saw that you saw this issue by passing mlx5_0 into initialize function. Can you tell me which file that function is located at ?
I did not find the file that includes that function remote_instance_transfer_engine.initialize

<!-- gh-comment-id:3562003980 --> @BilyZ98 commented on GitHub (Nov 21, 2025): Hi @XiaotaoChen , I checked the link you gave and saw that you saw this issue by passing `mlx5_0` into initialize function. Can you tell me which file that function is located at ? I did not find the file that includes that function `remote_instance_transfer_engine.initialize`
bsakaguchi commented 2026-10-29 21:13:09 +00:00
Author
Owner
Copy link

@BilyZ98 commented on GitHub (Nov 21, 2025):

I tried this config with 4 gpus and it works fine without reporting worker failed for slice error.

{
"local_hostname": "localhost",
"metadata_server": "http://localhost:8081/metadata",
"protocol": "rdma",
"device_name": "mlx5_0,mlx5_1",
"master_server_address": "localhost:10001"
}
<!-- gh-comment-id:3562325919 --> @BilyZ98 commented on GitHub (Nov 21, 2025): I tried this config with 4 gpus and it works fine without reporting worker failed for slice error. ``` { "local_hostname": "localhost", "metadata_server": "http://localhost:8081/metadata", "protocol": "rdma", "device_name": "mlx5_0,mlx5_1", "master_server_address": "localhost:10001" } ```
bsakaguchi commented 2026-10-29 21:13:09 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jun 19, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4748522453 --> @github-actions[bot] commented on GitHub (Jun 19, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
bsakaguchi commented 2026-10-29 21:13:09 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jun 26, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:4806400157 --> @github-actions[bot] commented on GitHub (Jun 26, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#346
No description provided.