mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #851] [Bug] PD Disaggregation with Sglang and Mooncake: local protection error and Failed to get kvcache from prefill instance #261
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#261
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Night-Quiet on GitHub (Sep 17, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/851
Originally assigned to: @alogfans on GitHub.
Bug Report
Checklist
Describe the bug
Attempted to deploy Qwen-235B using PD Disaggregation, but encountered a bug.
Partial solutions that have been attempted but failed:
nvidia-peer-memory,gdrcopyThank you for the work of your team.
Please kindly check if some of my operations are causing deployment failure.
I have tried my best to solve it, but still failed.
prefill:
decode:
router:
curl:
Reproduction
prefill:
decode:
router:
curl:
Environment
python3 -m sglang.check_env:pip list | grep mooncake:ib_write_bw -d mlx5_7 --use_cuda=0 + ib_write_bw -d mlx5_7 --use_cuda=0 10.10.11.3:ibdev2netdev:lsmod | grep -E "nvidia|nv_peer_mem|gdrdrv":show_gids:lsmod | grep gdrdrv && ldconfig -p | grep libgdrapi:Before submitting...
@ShangmingCai commented on GitHub (Sep 18, 2025):
Sorry, I didn't encounter
local protection errorthis error before. @alogfans Any idea why this happens?Not sure if it is related to the CUDA version
NVCC: Cuda compilation tools, release 11.5, V11.5.119?@Night-Quiet commented on GitHub (Sep 19, 2025):
Thank you for the recovery. The nvcc version was caused by an environment variable. Even after I changed the order, the error message remained the same.
@alogfans commented on GitHub (Sep 25, 2025):
A common cause of
local protection errorsis that local memory addresses (such as0x7f4d4c000800) were not registered before use. However, this does not align with the observed output. IfregisterLocalMemorywas not called, the tool should display a different warning.Perhaps you can first try the built-in transfer-engine-bench to test the functionability?
@Night-Quiet commented on GitHub (Sep 26, 2025):
@alogfans Yes, the situation you mentioned initially occurred, but I have completed the repair in this regard, so there were no errors in this area during the sglang startup process. Thank you for your reply. I will try it later.
@github-actions[bot] commented on GitHub (Jun 27, 2026):
This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.