kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2684] rdma+hip multi-protocol KV transfer fails on single-node disaggregation (Corrupted segment descriptor / Failed to get kvcache) #699

New issue

Closed

opened 2026-10-29 21:18:21 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:18:21 +00:00

Owner

Originally created by @Lzy17 on GitHub (Jun 30, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2684

Originally assigned to: @amd-arozanov on GitHub.

Summary

On a single node with both an RDMA NIC and the HIP intra-node transport installed, prefill/decode (PD) disaggregation fails to transfer KV cache. Every P/D session dies right after warmup and the decode side reports Failed to get kvcache from prefill instance. This is the configuration AMD single-node disaggregation runs in (ENABLE_MULTI_PROTOCOL=ON, ROCm/MI355X), where the device KV pool should move over HIP (intra-node XGMI) and the host-resident metadata/aux buffers over RDMA.

Environment

AMD MI355X (gfx950), single node, ROCm.
Mooncake built with USE_HIP=ON -DENABLE_MULTI_PROTOCOL=ON.

An RDMA NIC present, so topology discovery installs both transports:

Topology discovery complete. Found 1 HCAs.
installTransport, type=rdma
HIP transport installed for intra-node GPU P2P

Driver: SGLang PD disaggregation (1 prefill GPU0 + 1 decode GPU1), Mooncake transfer backend. The engine registers each buffer into all installed transports via a single batch batch_register.

Symptoms

Warmup succeeds, then the first real request fails. Logs:

decode side:

hip_transport.cpp:660] Unsupported memory type, 0x5555... 0
...
Decode transfer failed ... KVTransferError: Failed to get kvcache from prefill instance, it might be dead

prefill side:

transfer_metadata.cpp:785] Corrupted segment descriptor, name <peer> protocol hip
    buffer name hip:0 buffer addr ... buffer length 3832344320 buffer shm_name   (empty)
Session <peer> failed.
Prefill transfer failed ... remote mooncake session <peer> is not alive

The curl through the router returns:

{"error":{"type":"Internal Server Error","code":"decode_internal_error",
 "message":"... Failed to get kvcache from prefill instance, it might be dead"}}

Root cause

On a single node the transfer engine registers every buffer into all installed transports and publishes them under one shared local segment. With rdma+hip installed, three things break:

Host-memory registration rolls back the batch. The PD metadata/aux buffers (e.g. output_ids, bootstrap_room) are host (CPU) tensors — hipPointerGetAttributes returns memory type 0 (not Device), addresses in the 0x5555... range. HipTransport::registerLocalMemory rejects them with Unsupported memory type and returns -1, which rolls back the whole multi-transport batch and tears down sessions. (HIP IPC can only export device memory; these host buffers should simply be left to RDMA/TCP.)
The local segment protocol is overwritten. Each transport's install() overwrites LOCAL_SEGMENT_ID; HIP installs last, so the segment protocol collapses from rdma to a single hip instead of rdma,hip.
rdma+hip is not a recognized multi-protocol combination. The multi-protocol segment encode/decode allow-list is hardcoded to CXL+TCP / CXL+RDMA (max 2, CXL required) at encodeSegmentDesc / decodeSegmentDesc. So the node is not encoded via the multi-protocol path; it collapses to a single-protocol hip descriptor whose buffer list still contains the RDMA-registered KV buffers (these carry rkey/lkey, no shm_name). The single-protocol hip decoder requires every buffer to have a non-empty shm_name, so the rdma-origin KV entries trip Corrupted segment descriptor → the remote segment is unusable → KV transfer dies.

Note (3) is the primary blocker: even after fixing (1) so host buffers are skipped, end-to-end transfer still fails at the Corrupted segment descriptor check.

Why it wasn't seen before

Cross-node disaggregation uses the pure rdma path (RDMA can register host memory), which is unaffected. The failure only appears on a single node where the hip transport is invoked on the shared segment alongside rdma.

Expected behavior

A single node with rdma+hip installed should advertise both protocols on its local segment, register the device KV pool with HIP and the host aux buffers with RDMA, and route each transfer to the transport that owns the target buffer (device KV over HIP/XGMI, host aux over RDMA).

Fix

Proposed in #2682.

Downstream CI reproduction (SGLang)

This was first hit by SGLang PR sgl-project/sglang#27730 (bump Mooncake + enable ENABLE_MULTI_PROTOCOL for AMD single-node disaggregation). CI run 28375167699 (ROCm 7.2), suite stage-b-test-large-8-gpu-mi35x-disaggregation-amd, fails — both prefill and decode on the same node (10.42.9.21, single node):

Run test_disaggregation
  test/registered/amd/disaggregation/test_disaggregation_basic.py  -> FAILED (errors=1)  test_first_token_finish (line 178)
  test/registered/amd/disaggregation/test_disaggregation_pp.py     -> FAILED (errors=1)  test_gsm8k (line 293)

Transfer-engine errors (exactly the failure described above):

E hip_transport.cpp:671] Unsupported memory type, 0x5cc28f982ec0 0
E hip_transport.cpp:671] Unsupported memory type, 0x5ab7ec36fac0 0
W transfer_metadata.cpp:768] Corrupted segment descriptor, name 10.42.9.21:15882 protocol hip
    buffer name hip:1 buffer addr 134001715052544 buffer length 2684377088 buffer shm_name   (empty)
Prefill transfer failed ... KVTransferError: Failed to send kv chunk ... to 10.42.9.21:38649
Decode transfer failed ... KVTransferError: Failed to get kvcache from prefill instance, it might be dead
... remote mooncake session 10.42.9.21:15882 is not alive

surfaced to the client as:

openai.InternalServerError: Error code: 500 - {... 'message': "Decode transfer failed ...
  KVTransferError(...): Failed to get kvcache from prefill instance, it might be dead", ...}

The line numbers (hip_transport.cpp:671, transfer_metadata.cpp:768) differ slightly from the local repro above because of build/version offsets, but the signatures and ordering are identical: host (type-0) buffers rejected with Unsupported memory type, then the rdma+hip segment collapses to single-protocol hip and the rdma-origin KV buffers (no shm_name) trip Corrupted segment descriptor.

Originally created by @Lzy17 on GitHub (Jun 30, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2684 Originally assigned to: @amd-arozanov on GitHub. ## Summary On a single node with both an RDMA NIC and the HIP intra-node transport installed, prefill/decode (PD) disaggregation fails to transfer KV cache. Every P/D session dies right after warmup and the decode side reports `Failed to get kvcache from prefill instance`. This is the configuration AMD single-node disaggregation runs in (`ENABLE_MULTI_PROTOCOL=ON`, ROCm/MI355X), where the device KV pool should move over HIP (intra-node XGMI) and the host-resident metadata/aux buffers over RDMA. ## Environment - AMD MI355X (gfx950), single node, ROCm. - Mooncake built with `USE_HIP=ON -DENABLE_MULTI_PROTOCOL=ON`. - An RDMA NIC present, so topology discovery installs **both** transports: ``` Topology discovery complete. Found 1 HCAs. installTransport, type=rdma HIP transport installed for intra-node GPU P2P ``` - Driver: SGLang PD disaggregation (1 prefill GPU0 + 1 decode GPU1), Mooncake transfer backend. The engine registers each buffer into all installed transports via a single batch `batch_register`. ## Symptoms Warmup succeeds, then the first real request fails. Logs: decode side: ``` hip_transport.cpp:660] Unsupported memory type, 0x5555... 0 ... Decode transfer failed ... KVTransferError: Failed to get kvcache from prefill instance, it might be dead ``` prefill side: ``` transfer_metadata.cpp:785] Corrupted segment descriptor, name <peer> protocol hip buffer name hip:0 buffer addr ... buffer length 3832344320 buffer shm_name (empty) Session <peer> failed. Prefill transfer failed ... remote mooncake session <peer> is not alive ``` The curl through the router returns: ``` {"error":{"type":"Internal Server Error","code":"decode_internal_error", "message":"... Failed to get kvcache from prefill instance, it might be dead"}} ``` ## Root cause On a single node the transfer engine registers **every** buffer into **all** installed transports and publishes them under one shared local segment. With `rdma`+`hip` installed, three things break: 1. **Host-memory registration rolls back the batch.** The PD metadata/aux buffers (e.g. `output_ids`, `bootstrap_room`) are host (CPU) tensors — `hipPointerGetAttributes` returns memory type `0` (not Device), addresses in the `0x5555...` range. `HipTransport::registerLocalMemory` rejects them with `Unsupported memory type` and returns `-1`, which rolls back the whole multi-transport batch and tears down sessions. (HIP IPC can only export device memory; these host buffers should simply be left to RDMA/TCP.) 2. **The local segment protocol is overwritten.** Each transport's `install()` overwrites `LOCAL_SEGMENT_ID`; HIP installs last, so the segment protocol collapses from `rdma` to a single `hip` instead of `rdma,hip`. 3. **`rdma+hip` is not a recognized multi-protocol combination.** The multi-protocol segment encode/decode allow-list is hardcoded to `CXL+TCP` / `CXL+RDMA` (max 2, CXL required) at `encodeSegmentDesc` / `decodeSegmentDesc`. So the node is **not** encoded via the multi-protocol path; it collapses to a single-protocol `hip` descriptor whose buffer list still contains the RDMA-registered KV buffers (these carry rkey/lkey, **no** `shm_name`). The single-protocol `hip` decoder requires every buffer to have a non-empty `shm_name`, so the rdma-origin KV entries trip `Corrupted segment descriptor` → the remote segment is unusable → KV transfer dies. Note (3) is the primary blocker: even after fixing (1) so host buffers are skipped, end-to-end transfer still fails at the `Corrupted segment descriptor` check. ## Why it wasn't seen before Cross-node disaggregation uses the pure `rdma` path (RDMA can register host memory), which is unaffected. The failure only appears on a **single node** where the `hip` transport is invoked on the shared segment alongside `rdma`. ## Expected behavior A single node with `rdma`+`hip` installed should advertise both protocols on its local segment, register the device KV pool with HIP and the host aux buffers with RDMA, and route each transfer to the transport that owns the target buffer (device KV over HIP/XGMI, host aux over RDMA). ## Fix Proposed in #2682. --- ## Downstream CI reproduction (SGLang) This was first hit by SGLang PR [sgl-project/sglang#27730](https://github.com/sgl-project/sglang/pull/27730) (bump Mooncake + enable `ENABLE_MULTI_PROTOCOL` for AMD single-node disaggregation). CI run [28375167699](https://github.com/sgl-project/sglang/actions/runs/28375167699/job/84062853893) (ROCm 7.2), suite `stage-b-test-large-8-gpu-mi35x-disaggregation-amd`, fails — both prefill and decode on the same node (`10.42.9.21`, single node): ``` Run test_disaggregation test/registered/amd/disaggregation/test_disaggregation_basic.py -> FAILED (errors=1) test_first_token_finish (line 178) test/registered/amd/disaggregation/test_disaggregation_pp.py -> FAILED (errors=1) test_gsm8k (line 293) ``` Transfer-engine errors (exactly the failure described above): ``` E hip_transport.cpp:671] Unsupported memory type, 0x5cc28f982ec0 0 E hip_transport.cpp:671] Unsupported memory type, 0x5ab7ec36fac0 0 W transfer_metadata.cpp:768] Corrupted segment descriptor, name 10.42.9.21:15882 protocol hip buffer name hip:1 buffer addr 134001715052544 buffer length 2684377088 buffer shm_name (empty) Prefill transfer failed ... KVTransferError: Failed to send kv chunk ... to 10.42.9.21:38649 Decode transfer failed ... KVTransferError: Failed to get kvcache from prefill instance, it might be dead ... remote mooncake session 10.42.9.21:15882 is not alive ``` surfaced to the client as: ``` openai.InternalServerError: Error code: 500 - {... 'message': "Decode transfer failed ... KVTransferError(...): Failed to get kvcache from prefill instance, it might be dead", ...} ``` The line numbers (`hip_transport.cpp:671`, `transfer_metadata.cpp:768`) differ slightly from the local repro above because of build/version offsets, but the signatures and ordering are identical: host (type-0) buffers rejected with `Unsupported memory type`, then the `rdma+hip` segment collapses to single-protocol `hip` and the rdma-origin KV buffers (no `shm_name`) trip `Corrupted segment descriptor`.

bsakaguchi

2026-10-29 21:18:21 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 21:18:22 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 30, 2026):

Thanks for opening this issue, @Lzy17!

Field	Value
Issue	#2684
GitHub user ID	`36555117`
Reporter	@Lzy17

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jun 30, 2026): Thanks for opening this issue, @Lzy17! | Field | Value | |-------|-------| | **Issue** | #2684 | | **GitHub user ID** | `36555117` | | **Reporter** | @Lzy17 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:18:22 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 29, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Sep 29, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:18:22 +00:00

Author

Owner

@amd-arozanov commented on GitHub (Sep 29, 2026):

Closing the issue since https://github.com/kvcache-ai/Mooncake/pull/2682 was merged

@amd-arozanov commented on GitHub (Sep 29, 2026): Closing the issue since https://github.com/kvcache-ai/Mooncake/pull/2682 was merged