kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #4536] [Bug]: metadata not found #1158

New issue
Open
opened 2026-10-29 21:23:25 +00:00 by bsakaguchi · 7 comments
bsakaguchi commented 2026-10-29 21:23:25 +00:00
Owner
Copy link

Originally created by @artemus717 on GitHub (Oct 9, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4536

Bug Report

vLLM had successfully get mooncake store address and port. IP & port info in vLLM logs match store process info on server.

But segment info in metrics does not match store process info on server. And vLLM logs give metadata not found / Failed to open segment.

0.3.13.post1 vLLM 0.27.1

(EngineCore pid=104) INFO 10-09 11:32:26 [factory.py:62] Creating v1 connector with name: MooncakeStoreConnector and engine_id: 9547d0e5-dd0a-46ce-96ef-82ca588feebe
(EngineCore pid=104) WARNING 10-09 11:32:26 [base.py:202] Initializing KVConnectorBase_V1. This API is experimental and subject to change in the future as we iterate the design.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1009 11:32:26.473309 104 client_metric.cpp:102] Client metrics enabled (default enabled)
I1009 11:32:26.473346 104 client_metric.cpp:103] Client bandwidth summary enabled via MC_STORE_CLIENT_METRIC_BANDWIDTH
I1009 11:32:26.477298 104 client_service.cpp:423] client_id=8019590807347727255-11476044403673636779
I1009 11:32:26.477316 104 client_service.cpp:425] client_id=8019590807347727255-11476044403673636779, host_id=10.62.XXX.48
I1009 11:32:26.477322 104 client_service.cpp:437] Client metrics enabled but reporting disabled (interval=0)
I1009 11:32:26.480544 104 client_service.cpp:1077] Storage root directory is not set. persisting data is disabled.
I1009 11:32:26.480674 104 transfer_engine_impl.cpp:1046] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I1009 11:32:26.480695 104 transfer_engine_impl.cpp:116] Transfer Engine parseHostNameWithPort. server_name: 10.62.XXX.48 port: 13713
I1009 11:32:26.482698 104 transfer_metadata_plugin.cpp:1298] Found active interface bond0 with IP 10.62.XXX.48
I1009 11:32:26.482713 104 transfer_metadata_plugin.cpp:1298] Found active interface flannel.1 with IP 172.12.10.0
I1009 11:32:26.482719 104 transfer_metadata_plugin.cpp:1298] Found active interface cni0 with IP 172.12.10.1
I1009 11:32:26.482751 104 transfer_engine_impl.cpp:183] Transfer Engine RPC using new RPC mapping, listening on 10.62.XXX.48:15159
I1009 11:32:26.485147 104 client_service.cpp:872] Transfer engine auto discovery is disabled for protocol: rdma
I1009 11:32:26.485203 104 client_service.cpp:884] Using specified RDMA devices: bnxt_re_bond0
I1009 11:32:26.492988 104 topology.cpp:127] Device bnxt_re_bond0 port 1 is available
I1009 11:32:26.493392 104 client_service.cpp:894] Topology discovery complete with specified devices. Found 1 HCAs
I1009 11:32:26.493430 104 rdma_transport.cpp:150] [RDMA] Relaxed ordering disabled via MC_IB_PCI_RELAXED_ORDERING=0. Falling back to strict ordering.
I1009 11:32:26.493469 104 rdma_context.cpp:207] Using SIEVE endpoint store
I1009 11:32:26.494362 104 rdma_context.cpp:1363] Find best gid index: 1 on bnxt_re_bond0/1 (network state: with network device)
I1009 11:32:26.494988 104 rdma_context.cpp:290] RDMA device: bnxt_re_bond0, LID: 0, GID: (GID_Index 1) 00:00:00:00:00:00:00:00:00:00:ff:ff:0a:3e:d9:30
I1009 11:32:26.496236 104 real_client.cpp:851] Successfully created client on port 13713 after 1 attempt(s)
I1009 11:32:26.497290 104 real_client.cpp:884] Registering local memory: 536870912 bytes
I1009 11:32:26.552043 104 real_client.cpp:1088] Global segment size is 0, skip mounting segment
(EngineCore pid=104) INFO 10-09 11:32:26 [worker.py:1219] Mooncake mode=standalone-store (global_segment_size=0, local_buffer_size=536870912, preferred_segment=, enable_offload=False)
(EngineCore pid=104) WARNING 10-09 11:32:26 [worker.py:1244] standalone-store mode without enable_offload: large prefills may exceed the owner DirectIO budget.
(EngineCore pid=104) INFO 10-09 11:32:26 [worker.py:1484] Registered KV caches: num_groups=2, num_segments=12, num_blocks=9510
(EngineCore pid=104) INFO 10-09 11:32:26 [worker.py:1534] Started 1 Mooncake KV-load receive thread(s)
(EngineCore pid=104) INFO 10-09 11:32:28 [gpu_worker.py:789] Free memory on device (21.87/22.06 GiB) on startup. Desired GPU memory utilization is (0.8, 17.65 GiB). Actual usage is 14.1 GiB for consumed memory (weights + non-torch), 0.07 GiB for peak activation, and 0.0 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with --kv-cache-memory=3582534349 (3.34 GiB) to fit into requested memory, or --kv-cache-memory=8116983296 (7.56 GiB) to fully utilize gpu memory. Current kv cache memory in use is 3.48 GiB.
(EngineCore pid=104) INFO 10-09 11:32:30 [jit_monitor.py:79] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=104) INFO 10-09 11:32:30 [core.py:355] init engine (profile, create kv cache, warmup model) took 6.26 s
(EngineCore pid=104) INFO 10-09 11:32:32 [factory.py:62] Creating v1 connector with name: MooncakeStoreConnector and engine_id: 9547d0e5-dd0a-46ce-96ef-82ca588feebe
(EngineCore pid=104) WARNING 10-09 11:32:32 [base.py:202] Initializing KVConnectorBase_V1. This API is experimental and subject to change in the future as we iterate the design.
(EngineCore pid=104) WARNING 10-09 11:32:32 [vllm.py:1194] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(EngineCore pid=104) WARNING 10-09 11:32:32 [vllm.py:1247] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(EngineCore pid=104) INFO 10-09 11:32:32 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(EngineCore pid=104) INFO 10-09 11:32:32 [vllm.py:1426] Cudagraph is disabled under eager mode
(EngineCore pid=104) INFO 10-09 11:32:32 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
(APIServer pid=1) INFO 10-09 11:32:32 [api_server.py:678] Supported tasks: ['generate']
(APIServer pid=1) INFO 10-09 11:32:35 [hf.py:540] Detected the chat template content format to be 'string'. You can set --chat-template-content-format to override this.
(APIServer pid=1) WARNING 10-09 11:32:35 [serving.py:218] For gpt-oss, we ignore --enable-auto-tool-choice and always enable tool use.
(APIServer pid=1) INFO 10-09 11:32:35 [api_server.py:682] Starting vLLM server on http://10.62.XXX.48:40000
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:37] Available routes are:
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:99] API server: waiting for HTTP server to start
(APIServer pid=1) INFO: Started server process [1]
(APIServer pid=1) INFO: Waiting for application startup.
(APIServer pid=1) INFO: Application startup complete.
(APIServer pid=1) INFO 10-09 11:32:36 [launcher.py:105] API server: HTTP server started
(APIServer pid=1) INFO: 10.62.XXX.48:57186 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 10.62.XXX.48:57188 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 10.62.XXX.48:57200 - "GET /version HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 10.62.XXX.48:37450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=104) WARNING 10-09 11:36:24 [jit_monitor.py:135] Triton kernel JIT compilation during inference: kernel_unified_attention. This causes a latency spike; consider extending warmup to cover this shape/config.
E1009 11:36:24.669632 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.55%3A16414 http=404 body: metadata not found
W1009 11:36:24.669696 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.55:16414
E1009 11:36:24.669704 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.55:16414'
E1009 11:36:24.669881 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.56%3A15639 http=404 body: metadata not found
W1009 11:36:24.669893 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.56:15639
E1009 11:36:24.669898 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.56:15639'
E1009 11:36:24.670006 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.56%3A15639 http=404 body: metadata not found
W1009 11:36:24.670024 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.56:15639
E1009 11:36:24.670032 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.56:15639'
E1009 11:36:24.670125 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.55%3A16414 http=404 body: metadata not found
W1009 11:36:24.670136 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.55:16414
E1009 11:36:24.670145 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.55:16414'
E1009 11:36:24.670243 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.55%3A16414 http=404 body: metadata not found
W1009 11:36:24.670253 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.55:16414
E1009 11:36:24.670259 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.55:16414'
E1009 11:36:24.670364 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.48%3A15860 http=404 body: metadata not found
W1009 11:36:24.670377 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.48:15860
E1009 11:36:24.670383 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.48:15860'
E1009 11:36:24.670508 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.48%3A15860 http=404 body: metadata not found
W1009 11:36:24.670526 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.48:15860
E1009 11:36:24.670534 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.48:15860'
E1009 11:36:24.670639 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.56%3A15639 http=404 body: metadata not found
W1009 11:36:24.670650 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.56:15639
E1009 11:36:24.670656 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.56:15639'
E1009 11:36:24.671003 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:0@861009739929014aedc33398a1b6cfcbfee21647cb9c3acdd9b1df87bd683980 failed: TRANSFER_FAIL (Failed to submit transfer for replica 0)
E1009 11:36:24.671048 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:0@e2e1de4e40a90f93f2cff4138735e2c3e54d66172adace87752503c046093b54 failed: TRANSFER_FAIL (Failed to submit transfer for replica 0)
E1009 11:36:24.671056 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:0@c21ab5f0a8df590796e53383364004c4467ff65c2a23ed7db1fa289376840140 failed: TRANSFER_FAIL (Failed to submit transfer for replica 0)
E1009 11:36:24.671062 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:0@7d565ded266299bf852d567113ed97eaad9cc0900bb714d17adbea1941eab7bf failed: TRANSFER_FAIL (Failed to submit transfer for replica 0)
E1009 11:36:24.671068 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:1@861009739929014aedc33398a1b6cfcbfee21647cb9c3acdd9b1df87bd683980 failed: TRANSFER_FAIL (Failed to submit transfer for replica 0)
E1009 11:36:24.671074 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:1@e2e1de4e40a90f93f2cff4138735e2c3e54d66172adace87752503c046093b54 failed: TRANSFER_FAIL (Failed to submit transfer for replica 0)
E1009 11:36:24.671082 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:1@c21ab5f0a8df590796e53383364004c4467ff65c2a23ed7db1fa289376840140 failed: TRANSFER_FAIL (Failed to submit transfer for replica 0)
E1009 11:36:24.671089 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:1@7d565ded266299bf852d567113ed97eaad9cc0900bb714d17adbea1941eab7bf failed: TRANSFER_FAIL (Failed to submit transfer for replica 0)
(EngineCore pid=104) WARNING 10-09 11:36:24 [worker.py:881] batch_put failed: 8/8 keys failed (codes={-800}, batch_bytes=3145728, num_keys=8), first_key=gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:0@861009739929014aedc33398a1b6cfcbfee21647cb9c3acdd9b1df87bd683980
(EngineCore pid=104) WARNING 10-09 11:36:25 [jit_monitor.py:135] Triton kernel JIT compilation during inference: reduce_segments. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=1) INFO 10-09 11:36:26 [loggers.py:310] Engine 000: Avg prompt throughput: 6.8 tokens/s, Avg generation throughput: 0.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 10-09 11:36:26 [metrics.py:103] KV Transfer metrics: lookup_exists_count=1, lookup_exists_avg_ms=1.488, lookup_exists_p90_ms=1.488, lookup_exists_total_keys=8, lookup_exists_total_bytes=0, lookup_exists_failed_keys=0, lookup_exists_error_count=0, save_exists_count=1, save_exists_avg_ms=0.551, save_exists_p90_ms=0.551, save_exists_total_keys=8, save_exists_total_bytes=0, save_exists_failed_keys=0, save_exists_error_count=0, save_put_count=1, save_put_avg_ms=10.845, save_put_p90_ms=10.845, save_put_total_keys=8, save_put_total_bytes=3145728, save_put_failed_keys=8, save_put_error_count=0
(APIServer pid=1) INFO 10-09 11:36:36 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 10-09 11:36:46 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 0.0%

mooncake store info

netstat -anp |grep 3221876
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 3221876/python3
tcp 0 0 0.0.0.0:15860 0.0.0.0:* LISTEN 3221876/python3
tcp 0 0 10.62.XXX.48:34822 192.168.1.28:50051 ESTABLISHED 3221876/python3
tcp 0 0 10.62.XXX.48:34810 192.168.1.28:50051 ESTABLISHED 3221876/python3

netstat -anp |grep 2771186
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 2771186/python3
tcp 0 0 0.0.0.0:15639 0.0.0.0:* LISTEN 2771186/python3
tcp 0 0 10.62.XXX.56:48110 192.168.1.28:50051 ESTABLISHED 2771186/python3
tcp 0 0 10.62.XXX.56:48094 192.168.1.28:50051 ESTABLISHED 2771186/python3

netstat -anp |grep 16973
tcp 0 0 0.0.0.0:16414 0.0.0.0:* LISTEN 16973/python3
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 16973/python3
tcp 0 0 10.62.XXX.55:50668 192.168.1.28:50051 ESTABLISHED 16973/python3
tcp 0 0 10.62.XXX.55:50676 192.168.1.28:50051 ESTABLISHED 16973/python3

curl -s http://mooncake-master.mooncake:9003/metrics |grep XXX
segment_allocated_bytes{segment="10.62.XXX.48:13190"} 0
segment_allocated_bytes{segment="10.62.XXX.55:13106"} 0
segment_allocated_bytes{segment="10.62.XXX.56:13867"} 0
segment_total_capacity_bytes{segment="10.62.XXX.48:13190"} 3355443200
segment_total_capacity_bytes{segment="10.62.XXX.55:13106"} 3355443200
segment_total_capacity_bytes{segment="10.62.XXX.56:13867"} 3355443200

curl -s http://mooncake-master.mooncake:9003/get_all_segments
10.62.XXX.55:13106
10.62.XXX.56:13867
10.62.XXX.48:13190

Before submitting...

  • Ensure you searched for relevant issues and read the [documentation]
Originally created by @artemus717 on GitHub (Oct 9, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4536 ### Bug Report vLLM had successfully get mooncake store address and port. IP & port info in vLLM logs match store process info on server. But segment info in metrics does not match store process info on server. And vLLM logs give metadata not found / Failed to open segment. 0.3.13.post1 vLLM 0.27.1 (EngineCore pid=104) INFO 10-09 11:32:26 [factory.py:62] Creating v1 connector with name: MooncakeStoreConnector and engine_id: 9547d0e5-dd0a-46ce-96ef-82ca588feebe (EngineCore pid=104) WARNING 10-09 11:32:26 [base.py:202] Initializing KVConnectorBase_V1. This API is experimental and subject to change in the future as we iterate the design. WARNING: Logging before InitGoogleLogging() is written to STDERR I1009 11:32:26.473309 104 client_metric.cpp:102] Client metrics enabled (default enabled) I1009 11:32:26.473346 104 client_metric.cpp:103] Client bandwidth summary enabled via MC_STORE_CLIENT_METRIC_BANDWIDTH I1009 11:32:26.477298 104 client_service.cpp:423] client_id=8019590807347727255-11476044403673636779 I1009 11:32:26.477316 104 client_service.cpp:425] client_id=8019590807347727255-11476044403673636779, host_id=10.62.XXX.48 I1009 11:32:26.477322 104 client_service.cpp:437] Client metrics enabled but reporting disabled (interval=0) I1009 11:32:26.480544 104 client_service.cpp:1077] Storage root directory is not set. persisting data is disabled. I1009 11:32:26.480674 104 transfer_engine_impl.cpp:1046] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I1009 11:32:26.480695 104 transfer_engine_impl.cpp:116] Transfer Engine parseHostNameWithPort. server_name: 10.62.XXX.48 port: 13713 I1009 11:32:26.482698 104 transfer_metadata_plugin.cpp:1298] Found active interface bond0 with IP 10.62.XXX.48 I1009 11:32:26.482713 104 transfer_metadata_plugin.cpp:1298] Found active interface flannel.1 with IP 172.12.10.0 I1009 11:32:26.482719 104 transfer_metadata_plugin.cpp:1298] Found active interface cni0 with IP 172.12.10.1 I1009 11:32:26.482751 104 transfer_engine_impl.cpp:183] Transfer Engine RPC using new RPC mapping, listening on 10.62.XXX.48:15159 I1009 11:32:26.485147 104 client_service.cpp:872] Transfer engine auto discovery is disabled for protocol: rdma I1009 11:32:26.485203 104 client_service.cpp:884] Using specified RDMA devices: bnxt_re_bond0 I1009 11:32:26.492988 104 topology.cpp:127] Device bnxt_re_bond0 port 1 is available I1009 11:32:26.493392 104 client_service.cpp:894] Topology discovery complete with specified devices. Found 1 HCAs I1009 11:32:26.493430 104 rdma_transport.cpp:150] [RDMA] Relaxed ordering disabled via MC_IB_PCI_RELAXED_ORDERING=0. Falling back to strict ordering. I1009 11:32:26.493469 104 rdma_context.cpp:207] Using SIEVE endpoint store I1009 11:32:26.494362 104 rdma_context.cpp:1363] Find best gid index: 1 on bnxt_re_bond0/1 (network state: with network device) I1009 11:32:26.494988 104 rdma_context.cpp:290] RDMA device: bnxt_re_bond0, LID: 0, GID: (GID_Index 1) 00:00:00:00:00:00:00:00:00:00:ff:ff:0a:3e:d9:30 I1009 11:32:26.496236 104 real_client.cpp:851] Successfully created client on port 13713 after 1 attempt(s) I1009 11:32:26.497290 104 real_client.cpp:884] Registering local memory: 536870912 bytes I1009 11:32:26.552043 104 real_client.cpp:1088] Global segment size is 0, skip mounting segment (EngineCore pid=104) INFO 10-09 11:32:26 [worker.py:1219] Mooncake mode=standalone-store (global_segment_size=0, local_buffer_size=536870912, preferred_segment=<none>, enable_offload=False) (EngineCore pid=104) WARNING 10-09 11:32:26 [worker.py:1244] standalone-store mode without enable_offload: large prefills may exceed the owner DirectIO budget. (EngineCore pid=104) INFO 10-09 11:32:26 [worker.py:1484] Registered KV caches: num_groups=2, num_segments=12, num_blocks=9510 (EngineCore pid=104) INFO 10-09 11:32:26 [worker.py:1534] Started 1 Mooncake KV-load receive thread(s) (EngineCore pid=104) INFO 10-09 11:32:28 [gpu_worker.py:789] Free memory on device (21.87/22.06 GiB) on startup. Desired GPU memory utilization is (0.8, 17.65 GiB). Actual usage is 14.1 GiB for consumed memory (weights + non-torch), 0.07 GiB for peak activation, and 0.0 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=3582534349` (3.34 GiB) to fit into requested memory, or `--kv-cache-memory=8116983296` (7.56 GiB) to fully utilize gpu memory. Current kv cache memory in use is 3.48 GiB. (EngineCore pid=104) INFO 10-09 11:32:30 [jit_monitor.py:79] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. (EngineCore pid=104) INFO 10-09 11:32:30 [core.py:355] init engine (profile, create kv cache, warmup model) took 6.26 s (EngineCore pid=104) INFO 10-09 11:32:32 [factory.py:62] Creating v1 connector with name: MooncakeStoreConnector and engine_id: 9547d0e5-dd0a-46ce-96ef-82ca588feebe (EngineCore pid=104) WARNING 10-09 11:32:32 [base.py:202] Initializing KVConnectorBase_V1. This API is experimental and subject to change in the future as we iterate the design. (EngineCore pid=104) WARNING 10-09 11:32:32 [vllm.py:1194] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none (EngineCore pid=104) WARNING 10-09 11:32:32 [vllm.py:1247] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (EngineCore pid=104) INFO 10-09 11:32:32 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) (EngineCore pid=104) INFO 10-09 11:32:32 [vllm.py:1426] Cudagraph is disabled under eager mode (EngineCore pid=104) INFO 10-09 11:32:32 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant (APIServer pid=1) INFO 10-09 11:32:32 [api_server.py:678] Supported tasks: ['generate'] (APIServer pid=1) INFO 10-09 11:32:35 [hf.py:540] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. (APIServer pid=1) WARNING 10-09 11:32:35 [serving.py:218] For gpt-oss, we ignore --enable-auto-tool-choice and always enable tool use. (APIServer pid=1) INFO 10-09 11:32:35 [api_server.py:682] Starting vLLM server on http://10.62.XXX.48:40000 (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:37] Available routes are: (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /docs, Methods: GET, HEAD (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /redoc, Methods: GET, HEAD (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /load, Methods: GET (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /version, Methods: GET (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /health, Methods: GET (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /metrics, Methods: GET (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /tokenize, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /detokenize, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/models, Methods: GET (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /ping, Methods: GET (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /ping, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /invocations, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/chat/completions, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/responses, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/completions, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/messages, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /generative_scoring, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/completions/render, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /v1/completions/derender, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:46] Route: /inference/v1/generate, Methods: POST (APIServer pid=1) INFO 10-09 11:32:35 [launcher.py:99] API server: waiting for HTTP server to start (APIServer pid=1) INFO: Started server process [1] (APIServer pid=1) INFO: Waiting for application startup. (APIServer pid=1) INFO: Application startup complete. (APIServer pid=1) INFO 10-09 11:32:36 [launcher.py:105] API server: HTTP server started (APIServer pid=1) INFO: 10.62.XXX.48:57186 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 10.62.XXX.48:57188 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 10.62.XXX.48:57200 - "GET /version HTTP/1.1" 200 OK (APIServer pid=1) INFO: 10.62.XXX.48:37450 - "POST /v1/chat/completions HTTP/1.1" 200 OK (EngineCore pid=104) WARNING 10-09 11:36:24 [jit_monitor.py:135] Triton kernel JIT compilation during inference: kernel_unified_attention. This causes a latency spike; consider extending warmup to cover this shape/config. E1009 11:36:24.669632 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.55%3A16414 http=404 body: metadata not found W1009 11:36:24.669696 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.55:16414 E1009 11:36:24.669704 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.55:16414' E1009 11:36:24.669881 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.56%3A15639 http=404 body: metadata not found W1009 11:36:24.669893 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.56:15639 E1009 11:36:24.669898 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.56:15639' E1009 11:36:24.670006 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.56%3A15639 http=404 body: metadata not found W1009 11:36:24.670024 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.56:15639 E1009 11:36:24.670032 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.56:15639' E1009 11:36:24.670125 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.55%3A16414 http=404 body: metadata not found W1009 11:36:24.670136 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.55:16414 E1009 11:36:24.670145 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.55:16414' E1009 11:36:24.670243 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.55%3A16414 http=404 body: metadata not found W1009 11:36:24.670253 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.55:16414 E1009 11:36:24.670259 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.55:16414' E1009 11:36:24.670364 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.48%3A15860 http=404 body: metadata not found W1009 11:36:24.670377 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.48:15860 E1009 11:36:24.670383 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.48:15860' E1009 11:36:24.670508 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.48%3A15860 http=404 body: metadata not found W1009 11:36:24.670526 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.48:15860 E1009 11:36:24.670534 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.48:15860' E1009 11:36:24.670639 233 transfer_metadata_plugin.cpp:292] GET http://mooncake-master.mooncake:8080/metadata?key=mooncake%2Fram%2F10.62.XXX.56%3A15639 http=404 body: metadata not found W1009 11:36:24.670650 233 transfer_metadata.cpp:1115] Failed to retrieve segment descriptor, name 10.62.XXX.56:15639 E1009 11:36:24.670656 233 transfer_task.cpp:1285] Failed to open segment for endpoint='10.62.XXX.56:15639' E1009 11:36:24.671003 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:0@861009739929014aedc33398a1b6cfcbfee21647cb9c3acdd9b1df87bd683980 failed: TRANSFER_FAIL (Failed to submit transfer for replica 0) E1009 11:36:24.671048 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:0@e2e1de4e40a90f93f2cff4138735e2c3e54d66172adace87752503c046093b54 failed: TRANSFER_FAIL (Failed to submit transfer for replica 0) E1009 11:36:24.671056 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:0@c21ab5f0a8df590796e53383364004c4467ff65c2a23ed7db1fa289376840140 failed: TRANSFER_FAIL (Failed to submit transfer for replica 0) E1009 11:36:24.671062 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:0@7d565ded266299bf852d567113ed97eaad9cc0900bb714d17adbea1941eab7bf failed: TRANSFER_FAIL (Failed to submit transfer for replica 0) E1009 11:36:24.671068 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:1@861009739929014aedc33398a1b6cfcbfee21647cb9c3acdd9b1df87bd683980 failed: TRANSFER_FAIL (Failed to submit transfer for replica 0) E1009 11:36:24.671074 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:1@e2e1de4e40a90f93f2cff4138735e2c3e54d66172adace87752503c046093b54 failed: TRANSFER_FAIL (Failed to submit transfer for replica 0) E1009 11:36:24.671082 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:1@c21ab5f0a8df590796e53383364004c4467ff65c2a23ed7db1fa289376840140 failed: TRANSFER_FAIL (Failed to submit transfer for replica 0) E1009 11:36:24.671089 233 client_service.cpp:3085] Operation for key gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:1@7d565ded266299bf852d567113ed97eaad9cc0900bb714d17adbea1941eab7bf failed: TRANSFER_FAIL (Failed to submit transfer for replica 0) (EngineCore pid=104) WARNING 10-09 11:36:24 [worker.py:881] batch_put failed: 8/8 keys failed (codes={-800}, batch_bytes=3145728, num_keys=8), first_key=gpt-oss-20b@tp_rank:0@pcp0@dcp0@pp_rank:0@group:0@861009739929014aedc33398a1b6cfcbfee21647cb9c3acdd9b1df87bd683980 (EngineCore pid=104) WARNING 10-09 11:36:25 [jit_monitor.py:135] Triton kernel JIT compilation during inference: reduce_segments. This causes a latency spike; consider extending warmup to cover this shape/config. (APIServer pid=1) INFO 10-09 11:36:26 [loggers.py:310] Engine 000: Avg prompt throughput: 6.8 tokens/s, Avg generation throughput: 0.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 0.0% (APIServer pid=1) INFO 10-09 11:36:26 [metrics.py:103] KV Transfer metrics: lookup_exists_count=1, lookup_exists_avg_ms=1.488, lookup_exists_p90_ms=1.488, lookup_exists_total_keys=8, lookup_exists_total_bytes=0, lookup_exists_failed_keys=0, lookup_exists_error_count=0, save_exists_count=1, save_exists_avg_ms=0.551, save_exists_p90_ms=0.551, save_exists_total_keys=8, save_exists_total_bytes=0, save_exists_failed_keys=0, save_exists_error_count=0, save_put_count=1, save_put_avg_ms=10.845, save_put_p90_ms=10.845, save_put_total_keys=8, save_put_total_bytes=3145728, save_put_failed_keys=8, save_put_error_count=0 (APIServer pid=1) INFO 10-09 11:36:36 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 0.0% (APIServer pid=1) INFO 10-09 11:36:46 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 0.0% mooncake store info netstat -anp |grep 3221876 tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 3221876/python3 tcp 0 0 0.0.0.0:15860 0.0.0.0:* LISTEN 3221876/python3 tcp 0 0 10.62.XXX.48:34822 192.168.1.28:50051 ESTABLISHED 3221876/python3 tcp 0 0 10.62.XXX.48:34810 192.168.1.28:50051 ESTABLISHED 3221876/python3 netstat -anp |grep 2771186 tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 2771186/python3 tcp 0 0 0.0.0.0:15639 0.0.0.0:* LISTEN 2771186/python3 tcp 0 0 10.62.XXX.56:48110 192.168.1.28:50051 ESTABLISHED 2771186/python3 tcp 0 0 10.62.XXX.56:48094 192.168.1.28:50051 ESTABLISHED 2771186/python3 netstat -anp |grep 16973 tcp 0 0 0.0.0.0:16414 0.0.0.0:* LISTEN 16973/python3 tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 16973/python3 tcp 0 0 10.62.XXX.55:50668 192.168.1.28:50051 ESTABLISHED 16973/python3 tcp 0 0 10.62.XXX.55:50676 192.168.1.28:50051 ESTABLISHED 16973/python3 curl -s http://mooncake-master.mooncake:9003/metrics |grep XXX segment_allocated_bytes{segment="10.62.XXX.48:13190"} 0 segment_allocated_bytes{segment="10.62.XXX.55:13106"} 0 segment_allocated_bytes{segment="10.62.XXX.56:13867"} 0 segment_total_capacity_bytes{segment="10.62.XXX.48:13190"} 3355443200 segment_total_capacity_bytes{segment="10.62.XXX.55:13106"} 3355443200 segment_total_capacity_bytes{segment="10.62.XXX.56:13867"} 3355443200 curl -s http://mooncake-master.mooncake:9003/get_all_segments 10.62.XXX.55:13106 10.62.XXX.56:13867 10.62.XXX.48:13190 ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]
bsakaguchi commented 2026-10-29 21:23:26 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Oct 9, 2026):

Thanks for opening this issue, @artemus717!

Field Value
Issue #4536
GitHub user ID 16465173
Reporter @artemus717

A maintainer will triage this when possible. To help us respond faster, please include:

  • Mooncake version or commit SHA
  • Environment (OS, CUDA/driver, RDMA stack if relevant)
  • Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:6077199116 --> @github-actions[bot] commented on GitHub (Oct 9, 2026): Thanks for opening this issue, @artemus717! | Field | Value | |-------|-------| | **Issue** | #4536 | | **GitHub user ID** | `16465173` | | **Reporter** | @artemus717 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.
bsakaguchi commented 2026-10-29 21:23:26 +00:00
Author
Owner
Copy link

@sususama commented on GitHub (Oct 9, 2026):

Can I take this issue? Please assign it to me.

<!-- gh-comment-id:6083222955 --> @sususama commented on GitHub (Oct 9, 2026): Can I take this issue? Please assign it to me.
bsakaguchi commented 2026-10-29 21:23:26 +00:00
Author
Owner
Copy link

@artemus717 commented on GitHub (Oct 10, 2026):

mooncake master running in Kubernets as deployment using Kubernetes network. mooncake store running in Kubernetes as deployment using host network.

<!-- gh-comment-id:6092165583 --> @artemus717 commented on GitHub (Oct 10, 2026): mooncake master running in Kubernets as deployment using Kubernetes network. mooncake store running in Kubernetes as deployment using host network.
bsakaguchi commented 2026-10-29 21:23:27 +00:00
Author
Owner
Copy link

@Aionw commented on GitHub (Oct 10, 2026):

Diagnosis. This is a metadata-backend mismatch. The store nodes run with P2PHANDSHAKE (as the K8s guide's manifests configure). In that mode Client::MountSegment registers te_endpoint = <ip>:<random TE RPC port> (:15860, :15639, :16414 in the netstat output). The vLLM client uses the master's HTTP metadata server instead (new RPC mapping in its log), so it looks up mooncake/ram/<ip>:15860 there. P2P peers never publish anything to that server, so the lookup returns 404. The segment names shown in /metrics (:13190, ...) differ from the listening ports by design in P2P mode.

Workaround. Set the vLLM connector's metadata_server to P2PHANDSHAKE. In P2P mode the master's HTTP metadata server is not used at all, because peers exchange segment descriptors directly over their TE RPC ports. Alternatively, move the store nodes to the HTTP metadata server. Either way, every node must use the same backend.

Why logging alone won't help vLLM users. In vLLM, a nonzero return from store.setup() raises RuntimeError at startup (vllm/distributed/mooncake_store.py). A failed batch_put, by contrast, only logs a WARNING with codes={-800} and falls back to recompute. A runtime log line (#4544, #4545) helps whoever is already reading Mooncake's glog output, but the deployment still looks healthy and silently gets no KV-cache benefit. The check has to run inside setup() for vLLM to fail fast without any vLLM change.

Proposal: let the master own the TE metadata backend. The backend is a cluster-wide property, but today each process configures it independently, so a mismatch is always one typo away. Every client already connects to the master before initializing its Transfer Engine (ConnectToMaster → GetStorageConfig → InitTransferEngine in Client::Create), so the master can be the single source of truth:

  1. Add a master flag, for example --te_metadata_server=P2PHANDSHAKE or --te_metadata_server=http://mooncake-master.mooncake:8080/metadata. The value is the address clients should use, which in K8s means a Service name. The default is unset, which keeps today's behavior.
  2. Return it in GetStorageConfigResponse as a struct_pack::compatible<std::string, N> field, so old and new binaries interoperate. It is a plain connection string rather than an enum because clients must inherit the actual address, and the TE already derives the mode from that string (P2PHANDSHAKE vs. a plugin URL). A separate enum would duplicate that and could disagree with it. The P2P-vs-shared class used in step 4 comes from a single TE helper.
  3. In RealClient::setup_internal, an empty metadata_server inherits the master's value. Today an empty value against a separately deployed master is rejected with INVALID_PARAMS (real_client.cpp:997), so this only gives meaning to a currently invalid input. vLLM's default metadata_server is already "", so vLLM users get the right backend without configuring anything.
  4. If metadata_server is set explicitly and falls in a different class from the master's value (P2P vs. shared backend), setup() fails with a dedicated error code, and vLLM stops at startup with that error. Any mix of the two classes is broken in both directions: an HTTP reader cannot resolve a P2P writer (this issue), and a P2P reader handshakes to the writer's segment-name port, where nothing listens under the new RPC mapping. Rejecting the mix therefore never blocks a working setup.

With this in place, the K8s manifests can drop MOONCAKE_TE_META_DATA_SERVER from the store nodes and set it once on the master.

Scope / non-goals.

  • This only distinguishes P2P from shared backends. Two clients pointing at different shared backends (two etcd clusters, say) cannot be told apart by comparing connection strings, because the same service has many spellings. Catching that would require an active probe, which is out of scope here.
  • It does not make mixed modes work. That would need self-describing endpoints plus a direct-RPC fallback in the TE lookup path, which is a much larger protocol change.

@sususama @pjdurden, thanks for the quick diagnosis in #4545 and #4544. Would either of you like to take this on? It builds directly on what you've already traced, and it should fit in a single PR. Please coordinate here so the work isn't duplicated. In the meantime, #4544's doc fix and #4545's mount-time warning are still worth landing as the fallback for clusters that leave the flag unset.

<!-- gh-comment-id:6096513808 --> @Aionw commented on GitHub (Oct 10, 2026): **Diagnosis.** This is a metadata-backend mismatch. The store nodes run with `P2PHANDSHAKE` (as the K8s guide's manifests configure). In that mode `Client::MountSegment` registers `te_endpoint = <ip>:<random TE RPC port>` (`:15860`, `:15639`, `:16414` in the `netstat` output). The vLLM client uses the master's HTTP metadata server instead (`new RPC mapping` in its log), so it looks up `mooncake/ram/<ip>:15860` there. P2P peers never publish anything to that server, so the lookup returns 404. The segment names shown in `/metrics` (`:13190`, ...) differ from the listening ports by design in P2P mode. **Workaround.** Set the vLLM connector's `metadata_server` to `P2PHANDSHAKE`. In P2P mode the master's HTTP metadata server is not used at all, because peers exchange segment descriptors directly over their TE RPC ports. Alternatively, move the store nodes to the HTTP metadata server. Either way, every node must use the same backend. **Why logging alone won't help vLLM users.** In vLLM, a nonzero return from `store.setup()` raises `RuntimeError` at startup (`vllm/distributed/mooncake_store.py`). A failed `batch_put`, by contrast, only logs a WARNING with `codes={-800}` and falls back to recompute. A runtime log line (#4544, #4545) helps whoever is already reading Mooncake's glog output, but the deployment still looks healthy and silently gets no KV-cache benefit. The check has to run inside `setup()` for vLLM to fail fast without any vLLM change. **Proposal: let the master own the TE metadata backend.** The backend is a cluster-wide property, but today each process configures it independently, so a mismatch is always one typo away. Every client already connects to the master before initializing its Transfer Engine (`ConnectToMaster` → `GetStorageConfig` → `InitTransferEngine` in `Client::Create`), so the master can be the single source of truth: 1. Add a master flag, for example `--te_metadata_server=P2PHANDSHAKE` or `--te_metadata_server=http://mooncake-master.mooncake:8080/metadata`. The value is the address clients should use, which in K8s means a Service name. The default is unset, which keeps today's behavior. 2. Return it in `GetStorageConfigResponse` as a `struct_pack::compatible<std::string, N>` field, so old and new binaries interoperate. It is a plain connection string rather than an enum because clients must inherit the actual address, and the TE already derives the mode from that string (`P2PHANDSHAKE` vs. a plugin URL). A separate enum would duplicate that and could disagree with it. The P2P-vs-shared class used in step 4 comes from a single TE helper. 3. In `RealClient::setup_internal`, an empty `metadata_server` inherits the master's value. Today an empty value against a separately deployed master is rejected with `INVALID_PARAMS` (`real_client.cpp:997`), so this only gives meaning to a currently invalid input. vLLM's default `metadata_server` is already `""`, so vLLM users get the right backend without configuring anything. 4. If `metadata_server` is set explicitly and falls in a different class from the master's value (P2P vs. shared backend), `setup()` fails with a dedicated error code, and vLLM stops at startup with that error. Any mix of the two classes is broken in both directions: an HTTP reader cannot resolve a P2P writer (this issue), and a P2P reader handshakes to the writer's segment-name port, where nothing listens under the new RPC mapping. Rejecting the mix therefore never blocks a working setup. With this in place, the K8s manifests can drop `MOONCAKE_TE_META_DATA_SERVER` from the store nodes and set it once on the master. **Scope / non-goals.** - This only distinguishes P2P from shared backends. Two clients pointing at *different* shared backends (two etcd clusters, say) cannot be told apart by comparing connection strings, because the same service has many spellings. Catching that would require an active probe, which is out of scope here. - It does not make mixed modes work. That would need self-describing endpoints plus a direct-RPC fallback in the TE lookup path, which is a much larger protocol change. @sususama @pjdurden, thanks for the quick diagnosis in #4545 and #4544. Would either of you like to take this on? It builds directly on what you've already traced, and it should fit in a single PR. Please coordinate here so the work isn't duplicated. In the meantime, #4544's doc fix and #4545's mount-time warning are still worth landing as the fallback for clusters that leave the flag unset.
bsakaguchi commented 2026-10-29 21:23:27 +00:00
Author
Owner
Copy link

@sususama commented on GitHub (Oct 10, 2026):

I'll take this — it's a direct continuation of what #4545 traced, and I already have a build machine with the mixed-mode repro set up (single-host TCP: master + P2PHANDSHAKE writer + HTTP reader), so I can validate the new path end to end.

@pjdurden let me know if you'd rather own it; otherwise I'll start on it and keep #4544's doc fix as is.

Sequencing: #4545 touches real_client.cpp's empty-metadata_server fallback (the embedded-master branch), which this proposal changes. I'd like to land #4545 first as the fallback you described, then build this on top — unless you'd prefer I drop that hunk from #4545 and fold it into the new PR.

A few things I'd want to pin down before writing code:

  1. Where should the P2P-vs-shared classifier live — a TE helper next to the existing P2PHANDSHAKE string handling, or in the store?
  2. New ErrorCode for the class mismatch, or reuse an existing one?
  3. Should the K8s manifests (MOONCAKE_TE_META_DATA_SERVER) change in the same PR or follow separately?
  4. struct_pack::compatible<std::string, N> — what N should I use here?
<!-- gh-comment-id:6096573170 --> @sususama commented on GitHub (Oct 10, 2026): I'll take this — it's a direct continuation of what #4545 traced, and I already have a build machine with the mixed-mode repro set up (single-host TCP: master + P2PHANDSHAKE writer + HTTP reader), so I can validate the new path end to end. @pjdurden let me know if you'd rather own it; otherwise I'll start on it and keep #4544's doc fix as is. Sequencing: #4545 touches real_client.cpp's empty-metadata_server fallback (the embedded-master branch), which this proposal changes. I'd like to land #4545 first as the fallback you described, then build this on top — unless you'd prefer I drop that hunk from #4545 and fold it into the new PR. A few things I'd want to pin down before writing code: 1. Where should the P2P-vs-shared classifier live — a TE helper next to the existing P2PHANDSHAKE string handling, or in the store? 2. New ErrorCode for the class mismatch, or reuse an existing one? 3. Should the K8s manifests (MOONCAKE_TE_META_DATA_SERVER) change in the same PR or follow separately? 4. struct_pack::compatible<std::string, N> — what N should I use here?
bsakaguchi commented 2026-10-29 21:23:27 +00:00
Author
Owner
Copy link

@sususama commented on GitHub (Oct 12, 2026):

https://github.com/kvcache-ai/Mooncake/pull/4604
@Aionw Hi, I've implemented the code-level fix in this PR. Could you please review it?

<!-- gh-comment-id:6125747015 --> @sususama commented on GitHub (Oct 12, 2026): https://github.com/kvcache-ai/Mooncake/pull/4604 @Aionw Hi, I've implemented the code-level fix in this PR. Could you please review it?
bsakaguchi commented 2026-10-29 21:23:27 +00:00
Author
Owner
Copy link

@artemus717 commented on GitHub (Oct 28, 2026):

Will this be included in 0.3.14 ?

<!-- gh-comment-id:6458116897 --> @artemus717 commented on GitHub (Oct 28, 2026): Will this be included in 0.3.14 ?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#1158
No description provided.