kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #2177] [Bug]: transfer_metadata: Failed to get segment descriptor for segment xxx #598

New issue
Closed
opened 2026-10-29 21:17:00 +00:00 by bsakaguchi · 11 comments
bsakaguchi commented 2026-10-29 21:17:00 +00:00
Owner
Copy link

Originally created by @thincal on GitHub (May 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2177

Originally assigned to: @alogfans on GitHub.

Bug Report

Describe the bug

SGLang v0.5.10 with mooncake transferengine v0.3.9 works well, but when I upgrade mooncake transfer-engine to latest version v0.3.11, it will report error as bellow:

...
E0521 16:22:21.694042   162 transfer_metadata.cpp:877] Local segment descriptor not found
...
I0521 16:41:11.701406  4064 transfer_metadata_dump.cpp:51] Failed to get segment descriptor for segment 10.112.116.43:16917 address 0x7152b4276000--0x7152b4288000
[2026-05-21 16:41:11 ATTN_CP7 TP7 EP7] Session 10.112.116.43:16917 failed.
[2026-05-21 16:41:11 ATTN_CP7 TP7 EP7] Prefill transfer failed for request rank=7 req.rid='20260521164110.335_yewi' req.bootstrap_room=666643570191291695 with exception KVTransferError(bootstrap_room=666643570191291695): Decode instance could be dead, remote mooncake session 10.112.116.43:16917 is not alive

Reproduction

  1. sglang v0.5.10 + mooncake transfer-engine v0.3.11, with pdd deployment
  2. send request

Before submitting...

  • Ensure you searched for relevant issues and read the [documentation]
Originally created by @thincal on GitHub (May 22, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2177 Originally assigned to: @alogfans on GitHub. ### Bug Report ### Describe the bug SGLang v0.5.10 with mooncake transferengine v0.3.9 works well, but when I upgrade mooncake transfer-engine to latest version v0.3.11, it will report error as bellow: ``` ... E0521 16:22:21.694042 162 transfer_metadata.cpp:877] Local segment descriptor not found ... I0521 16:41:11.701406 4064 transfer_metadata_dump.cpp:51] Failed to get segment descriptor for segment 10.112.116.43:16917 address 0x7152b4276000--0x7152b4288000 [2026-05-21 16:41:11 ATTN_CP7 TP7 EP7] Session 10.112.116.43:16917 failed. [2026-05-21 16:41:11 ATTN_CP7 TP7 EP7] Prefill transfer failed for request rank=7 req.rid='20260521164110.335_yewi' req.bootstrap_room=666643570191291695 with exception KVTransferError(bootstrap_room=666643570191291695): Decode instance could be dead, remote mooncake session 10.112.116.43:16917 is not alive ``` ### Reproduction 1. sglang v0.5.10 + mooncake transfer-engine v0.3.11, with pdd deployment 2. send request ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]
bsakaguchi 2026-10-29 21:17:00 +00:00
  • closed this issue
  • added the
    bug
         label
bsakaguchi commented 2026-10-29 21:17:02 +00:00
Author
Owner
Copy link

@alogfans commented on GitHub (May 22, 2026):

E0521 16:22:21.694042   162 transfer_metadata.cpp:877] Local segment descriptor not found
I0521 16:41:10.810740  4097 transfer_metadata_dump.cpp:51] Failed to get segment descriptor for segment 10.112.116.43:16917 address 0x715448048300--0x71544804a400

The first line message is expected, as allocateLocalSegmentID will first check whether getSegmentDesc existed. If not existed (normally the first call), it prints this message and allocates new SegmentDesc object. So it's safe to ignore.

The sedond message reflects that, potentially:

  • Not all node shares the same TE version,
  • 10.112.116.43:16917 is invalid

Does this break the service? If you can provide more logs we can further investigate it.

<!-- gh-comment-id:4515894222 --> @alogfans commented on GitHub (May 22, 2026): ``` E0521 16:22:21.694042 162 transfer_metadata.cpp:877] Local segment descriptor not found I0521 16:41:10.810740 4097 transfer_metadata_dump.cpp:51] Failed to get segment descriptor for segment 10.112.116.43:16917 address 0x715448048300--0x71544804a400 ``` The first line message is expected, as `allocateLocalSegmentID` will first check whether `getSegmentDesc` existed. If not existed (normally the first call), it prints this message and allocates new SegmentDesc object. So it's safe to ignore. The sedond message reflects that, potentially: - Not all node shares the same TE version, - 10.112.116.43:16917 is invalid Does this break the service? If you can provide more logs we can further investigate it.
bsakaguchi commented 2026-10-29 21:17:02 +00:00
Author
Owner
Copy link

@thincal commented on GitHub (May 22, 2026):

@alogfans Thanks for investigating this issue, more logs provided into the original post.

It seems that the error "Failed to get segment descriptor for segment" is happened during transfering kvcache by prefill install employing mooncake transfer engine, then it will cause the prefill transfer failed.

<!-- gh-comment-id:4516408724 --> @thincal commented on GitHub (May 22, 2026): @alogfans Thanks for investigating this issue, more logs provided into the original post. It seems that the error "Failed to get segment descriptor for segment" is happened during transfering kvcache by prefill install employing mooncake transfer engine, then it will cause the prefill transfer failed.
bsakaguchi commented 2026-10-29 21:17:02 +00:00
Author
Owner
Copy link

@thincal commented on GitHub (May 22, 2026):

Not all node shares the same TE version,

I would double check it, response later.

<!-- gh-comment-id:4516487969 --> @thincal commented on GitHub (May 22, 2026): > Not all node shares the same TE version, I would double check it, response later.
bsakaguchi commented 2026-10-29 21:17:02 +00:00
Author
Owner
Copy link

@thincal commented on GitHub (May 22, 2026):

Not all node shares the same TE version,

I would double check it, response later.

It is confirmed that both prefill and decode are with same version v0.3.11 of Mooncake TE.

<!-- gh-comment-id:4518427758 --> @thincal commented on GitHub (May 22, 2026): > > Not all node shares the same TE version, > > I would double check it, response later. It is confirmed that both prefill and decode are with same version v0.3.11 of Mooncake TE.
bsakaguchi commented 2026-10-29 21:17:02 +00:00
Author
Owner
Copy link

@thincal commented on GitHub (May 22, 2026):

I just have found that "Failed to get segment descriptor for segment" is caused by the error from decode side:

I0522 11:05:12.383546   474 rdma_context.cpp:153] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:0a:50:ba:14
E0522 11:05:12.383555   474 transfer_metadata.cpp:877] Local segment descriptor not found
I0522 11:05:12.383623   474 transfer_engine_impl.cpp:336] installTransport, type=rdma
E0522 11:09:03.174664   328 rdma_context.cpp:334] Failed to register memory 0x7244b2000000: Cannot allocate memory [12]
E0522 11:09:03.174690   328 rdma_transport.cpp:257] Failed to register memory region with context 0
W0522 11:09:03.174695   328 rdma_transport.cpp:393] RdmaTransport: Failed to register memory: addr 0x7244b2000000 length 968785920

@alogfans What's the possible reason with the register memory failure (Cannot allocate memory) ? I can provide more information if needed, thanks for help.

<!-- gh-comment-id:4518452508 --> @thincal commented on GitHub (May 22, 2026): I just have found that "Failed to get segment descriptor for segment" is caused by the error from decode side: ``` I0522 11:05:12.383546 474 rdma_context.cpp:153] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:0a:50:ba:14 E0522 11:05:12.383555 474 transfer_metadata.cpp:877] Local segment descriptor not found I0522 11:05:12.383623 474 transfer_engine_impl.cpp:336] installTransport, type=rdma E0522 11:09:03.174664 328 rdma_context.cpp:334] Failed to register memory 0x7244b2000000: Cannot allocate memory [12] E0522 11:09:03.174690 328 rdma_transport.cpp:257] Failed to register memory region with context 0 W0522 11:09:03.174695 328 rdma_transport.cpp:393] RdmaTransport: Failed to register memory: addr 0x7244b2000000 length 968785920 ``` @alogfans What's the possible reason with the register memory failure (Cannot allocate memory) ? I can provide more information if needed, thanks for help.
bsakaguchi commented 2026-10-29 21:17:02 +00:00
Author
Owner
Copy link

@thincal commented on GitHub (May 22, 2026):

From your settings we can see that you used two InfiniBand devices and tried to register 80G memory, in mooncake transfer engine, the settings led to register 80G one time for each device, accumulating to 160G registered, which is greater than the limit. https://github.com/kvcache-ai/Mooncake/issues/1382#issuecomment-3757287001

I have used 8 rdma devices to register with Mooncake TE, which might (guess) cause the memory register (cannot allocate memory) issue based on previous post. After I configure Mooncake TE with one single rdma device it works well again.

But it seems that this setting doesn't cause any issue in Mooncake TE version v0.3.9, could you help illustrate the behavior changes with this after version v0.3.9?

Many thanks @alogfans

<!-- gh-comment-id:4518560579 --> @thincal commented on GitHub (May 22, 2026): > From your settings we can see that you used two InfiniBand devices and tried to register 80G memory, in mooncake transfer engine, the settings led to register 80G one time for each device, accumulating to 160G registered, which is greater than the limit. https://github.com/kvcache-ai/Mooncake/issues/1382#issuecomment-3757287001 I have used 8 rdma devices to register with Mooncake TE, which might (guess) cause the memory register (cannot allocate memory) issue based on previous post. After I configure Mooncake TE with one single rdma device it works well again. But it seems that this setting doesn't cause any issue in Mooncake TE version v0.3.9, could you help illustrate the behavior changes with this after version v0.3.9? Many thanks @alogfans
bsakaguchi commented 2026-10-29 21:17:02 +00:00
Author
Owner
Copy link

@thincal commented on GitHub (May 22, 2026):

From your settings we can see that you used two InfiniBand devices and tried to register 80G memory, in mooncake transfer engine, the settings led to register 80G one time for each device, accumulating to 160G registered, which is greater than the limit. #1382 (comment)

I have used 8 rdma devices to register with Mooncake TE, which might cause the memory register (cannot allocate memory) issue based on previous post.

But it seems that this setting doesn't cause any issue in Mooncake TE version v0.3.9, could you help illustrate the behavior changes with this after version v0.3.9?

Many thanks @alogfans

Update: Sorry that I miss-understanding the behavior of "led to register 80G one time for each device", which it seems about the Moonecake Store? But after I update the Mooncake TE device register with one single device for each gpu id, it works well now.

Looking for any further information with this, thanks for help.

More info:

  • lock memory
# ulimit -a (inside container)
max locked memory           (kbytes, -l) 64

# ulimit -a (on host)
max locked memory           (kbytes, -l) 524288
  • mooncake setting
MC_TE_METRIC: "1"
MC_WORKERS_PER_CTX: "2"
MC_TRANSFER_TIMEOUT: "40"
MC_SLICE_SIZE: "262144"
MC_GID_INDEX: "3"
MC_MAX_EP_PER_CTX: "2048"
MC_LOG_LEVEL: INFO
  • sglang specifying mooncake devices
--disaggregation-ib-device=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
<!-- gh-comment-id:4519000904 --> @thincal commented on GitHub (May 22, 2026): > > From your settings we can see that you used two InfiniBand devices and tried to register 80G memory, in mooncake transfer engine, the settings led to register 80G one time for each device, accumulating to 160G registered, which is greater than the limit. [#1382 (comment)](https://github.com/kvcache-ai/Mooncake/issues/1382#issuecomment-3757287001) > > I have used 8 rdma devices to register with Mooncake TE, which might cause the memory register (cannot allocate memory) issue based on previous post. > > But it seems that this setting doesn't cause any issue in Mooncake TE version v0.3.9, could you help illustrate the behavior changes with this after version v0.3.9? > > Many thanks [@alogfans](https://github.com/alogfans) Update: Sorry that I miss-understanding the behavior of "led to register 80G one time for each device", which it seems about the Moonecake Store? But after I update the Mooncake TE device register with one single device for each gpu id, it works well now. Looking for any further information with this, thanks for help. > More info: - lock memory ``` # ulimit -a (inside container) max locked memory (kbytes, -l) 64 # ulimit -a (on host) max locked memory (kbytes, -l) 524288 ``` - mooncake setting ``` MC_TE_METRIC: "1" MC_WORKERS_PER_CTX: "2" MC_TRANSFER_TIMEOUT: "40" MC_SLICE_SIZE: "262144" MC_GID_INDEX: "3" MC_MAX_EP_PER_CTX: "2048" MC_LOG_LEVEL: INFO ``` - sglang specifying mooncake devices ``` --disaggregation-ib-device=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 ```
bsakaguchi commented 2026-10-29 21:17:02 +00:00
Author
Owner
Copy link

@thincal commented on GitHub (May 22, 2026):

It seems that I have found the root cause: https://github.com/kvcache-ai/Mooncake/pull/2066, which introduce the WITH_NVIDIA_PEERMEM with default value false to enable dmabuf by default.

But dmabuf with multiple devices reigster consumes massive lock memory etc. resources, I have verified with WITH_NVIDIA_PEERMEM: 1, it works well again with multi devices assigned.

Also it seems that this is fixed by https://github.com/kvcache-ai/Mooncake/pull/2192 from @ShangmingCai to turn dmabuf off by default.

cc @alogfans

<!-- gh-comment-id:4521015248 --> @thincal commented on GitHub (May 22, 2026): It seems that I have found the root cause: https://github.com/kvcache-ai/Mooncake/pull/2066, which introduce the `WITH_NVIDIA_PEERMEM` with default value `false` to enable dmabuf by default. But dmabuf with multiple devices reigster consumes massive lock memory etc. resources, I have verified with `WITH_NVIDIA_PEERMEM: 1`, it works well again with multi devices assigned. Also it seems that this is fixed by https://github.com/kvcache-ai/Mooncake/pull/2192 from @ShangmingCai to turn dmabuf off by default. cc @alogfans
bsakaguchi commented 2026-10-29 21:17:02 +00:00
Author
Owner
Copy link

@alogfans commented on GitHub (May 23, 2026):

@thincal After applying #2192, will the error still happen?

<!-- gh-comment-id:4524062139 --> @alogfans commented on GitHub (May 23, 2026): @thincal After applying #2192, will the error still happen?
bsakaguchi commented 2026-10-29 21:17:03 +00:00
Author
Owner
Copy link

@thincal commented on GitHub (May 23, 2026):

@thincal After applying #2192, will the error still happen?

I just add WITH_NVIDIA_PEERMEM: "1", it works well now.

<!-- gh-comment-id:4526053601 --> @thincal commented on GitHub (May 23, 2026): > [@thincal](https://github.com/thincal) After applying [#2192](https://github.com/kvcache-ai/Mooncake/pull/2192), will the error still happen? I just add `WITH_NVIDIA_PEERMEM: "1"`, it works well now.
bsakaguchi commented 2026-10-29 21:17:03 +00:00
Author
Owner
Copy link

@alogfans commented on GitHub (May 28, 2026):

@thincal Thanks for reporting!

<!-- gh-comment-id:4561482534 --> @alogfans commented on GitHub (May 28, 2026): @thincal Thanks for reporting!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#598
No description provided.