kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #351] [bug] Mooncake memory registration failed #128

New issue
Closed
opened 2026-10-29 21:09:47 +00:00 by bsakaguchi · 25 comments
bsakaguchi commented 2026-10-29 21:09:47 +00:00
Owner
Copy link

Originally created by @CSEEduanyu on GitHub (May 12, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/351

when i use mooncake in sglang:
srt/disaggregation/mooncake/transfer_engine.py line 36, in register
raise RuntimeError("Mooncake memory registration failed.")
E0511 07:42:36.355509 10240 rdma_context.cpp:198] Failed to register memory 0x2e1cbac200: Bad address [14]

Originally created by @CSEEduanyu on GitHub (May 12, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/351 when i use mooncake in sglang: srt/disaggregation/mooncake/transfer_engine.py line 36, in register raise RuntimeError("Mooncake memory registration failed.") E0511 07:42:36.355509 10240 rdma_context.cpp:198] Failed to register memory 0x2e1cbac200: Bad address [14]
bsakaguchi commented 2026-10-29 21:09:48 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (May 12, 2025):

Python: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H800
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda-12.4
NVCC: Cuda compilation tools, release 12.4, V12.4.131
CUDA Driver Version: 535.230.02
PyTorch: 2.6.0+cu124
sglang: 0.4.6.post2
sgl_kernel: 0.1.1
flashinfer_python: 0.2.5
triton: 3.2.0
transformers: 4.51.1
torchao: 0.10.0
numpy: 1.26.4
aiohttp: 3.11.18
fastapi: 0.115.12
hf_transfer: 0.1.9
huggingface_hub: 0.31.1
interegular: 0.3.3
modelscope: 1.25.0
orjson: 3.10.18
outlines: 0.1.11
packaging: 25.0
psutil: 7.0.0
pydantic: 2.11.4
python-multipart: 0.0.20
pyzmq: 26.4.0
uvicorn: 0.34.2
uvloop: 0.21.0
vllm: 0.8.2
xgrammar: 0.1.16
openai: 1.75.0
tiktoken: 0.9.0
anthropic: 0.51.0
litellm: 1.68.1
decord: 0.6.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV8 NV8 NV8 NV8 NV8 NV8 NV8 SYS PIX PHB PHB PHB SYS SYS SYS SYS 0-89 0 N/A
GPU1 NV8 X NV8 NV8 NV8 NV8 NV8 NV8 SYS PHB PIX PHB PHB SYS SYS SYS SYS 0-89 0 N/A
GPU2 NV8 NV8 X NV8 NV8 NV8 NV8 NV8 SYS PHB PHB PIX PHB SYS SYS SYS SYS 0-89 0 N/A
GPU3 NV8 NV8 NV8 X NV8 NV8 NV8 NV8 SYS PHB PHB PHB PIX SYS SYS SYS SYS 0-89 0 N/A
GPU4 NV8 NV8 NV8 NV8 X NV8 NV8 NV8 SYS SYS SYS SYS SYS PIX PHB PHB PHB 90-179 1 N/A
GPU5 NV8 NV8 NV8 NV8 NV8 X NV8 NV8 SYS SYS SYS SYS SYS PHB PIX PHB PHB 90-179 1 N/A
GPU6 NV8 NV8 NV8 NV8 NV8 NV8 X NV8 SYS SYS SYS SYS SYS PHB PHB PIX PHB 90-179 1 N/A
GPU7 NV8 NV8 NV8 NV8 NV8 NV8 NV8 X SYS SYS SYS SYS SYS PHB PHB PHB PIX 90-179 1 N/A
NIC0 SYS SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS
NIC1 PIX PHB PHB PHB SYS SYS SYS SYS SYS X PHB PHB PHB SYS SYS SYS SYS
NIC2 PHB PIX PHB PHB SYS SYS SYS SYS SYS PHB X PHB PHB SYS SYS SYS SYS
NIC3 PHB PHB PIX PHB SYS SYS SYS SYS SYS PHB PHB X PHB SYS SYS SYS SYS
NIC4 PHB PHB PHB PIX SYS SYS SYS SYS SYS PHB PHB PHB X SYS SYS SYS SYS
NIC5 SYS SYS SYS SYS PIX PHB PHB PHB SYS SYS SYS SYS SYS X PHB PHB PHB
NIC6 SYS SYS SYS SYS PHB PIX PHB PHB SYS SYS SYS SYS SYS PHB X PHB PHB
NIC7 SYS SYS SYS SYS PHB PHB PIX PHB SYS SYS SYS SYS SYS PHB PHB X PHB
NIC8 SYS SYS SYS SYS PHB PHB PHB PIX SYS SYS SYS SYS SYS PHB PHB PHB X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8

Hypervisor vendor: KVM
ulimit soft: 1048576

<!-- gh-comment-id:2870558671 --> @CSEEduanyu commented on GitHub (May 12, 2025): Python: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA H800 GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda-12.4 NVCC: Cuda compilation tools, release 12.4, V12.4.131 CUDA Driver Version: 535.230.02 PyTorch: 2.6.0+cu124 sglang: 0.4.6.post2 sgl_kernel: 0.1.1 flashinfer_python: 0.2.5 triton: 3.2.0 transformers: 4.51.1 torchao: 0.10.0 numpy: 1.26.4 aiohttp: 3.11.18 fastapi: 0.115.12 hf_transfer: 0.1.9 huggingface_hub: 0.31.1 interegular: 0.3.3 modelscope: 1.25.0 orjson: 3.10.18 outlines: 0.1.11 packaging: 25.0 psutil: 7.0.0 pydantic: 2.11.4 python-multipart: 0.0.20 pyzmq: 26.4.0 uvicorn: 0.34.2 uvloop: 0.21.0 vllm: 0.8.2 xgrammar: 0.1.16 openai: 1.75.0 tiktoken: 0.9.0 anthropic: 0.51.0 litellm: 1.68.1 decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV8 NV8 NV8 NV8 NV8 NV8 NV8 SYS PIX PHB PHB PHB SYS SYS SYS SYS 0-89 0 N/A GPU1 NV8 X NV8 NV8 NV8 NV8 NV8 NV8 SYS PHB PIX PHB PHB SYS SYS SYS SYS 0-89 0 N/A GPU2 NV8 NV8 X NV8 NV8 NV8 NV8 NV8 SYS PHB PHB PIX PHB SYS SYS SYS SYS 0-89 0 N/A GPU3 NV8 NV8 NV8 X NV8 NV8 NV8 NV8 SYS PHB PHB PHB PIX SYS SYS SYS SYS 0-89 0 N/A GPU4 NV8 NV8 NV8 NV8 X NV8 NV8 NV8 SYS SYS SYS SYS SYS PIX PHB PHB PHB 90-179 1 N/A GPU5 NV8 NV8 NV8 NV8 NV8 X NV8 NV8 SYS SYS SYS SYS SYS PHB PIX PHB PHB 90-179 1 N/A GPU6 NV8 NV8 NV8 NV8 NV8 NV8 X NV8 SYS SYS SYS SYS SYS PHB PHB PIX PHB 90-179 1 N/A GPU7 NV8 NV8 NV8 NV8 NV8 NV8 NV8 X SYS SYS SYS SYS SYS PHB PHB PHB PIX 90-179 1 N/A NIC0 SYS SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS NIC1 PIX PHB PHB PHB SYS SYS SYS SYS SYS X PHB PHB PHB SYS SYS SYS SYS NIC2 PHB PIX PHB PHB SYS SYS SYS SYS SYS PHB X PHB PHB SYS SYS SYS SYS NIC3 PHB PHB PIX PHB SYS SYS SYS SYS SYS PHB PHB X PHB SYS SYS SYS SYS NIC4 PHB PHB PHB PIX SYS SYS SYS SYS SYS PHB PHB PHB X SYS SYS SYS SYS NIC5 SYS SYS SYS SYS PIX PHB PHB PHB SYS SYS SYS SYS SYS X PHB PHB PHB NIC6 SYS SYS SYS SYS PHB PIX PHB PHB SYS SYS SYS SYS SYS PHB X PHB PHB NIC7 SYS SYS SYS SYS PHB PHB PIX PHB SYS SYS SYS SYS SYS PHB PHB X PHB NIC8 SYS SYS SYS SYS PHB PHB PHB PIX SYS SYS SYS SYS SYS PHB PHB PHB X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NIC8: mlx5_8 Hypervisor vendor: KVM ulimit soft: 1048576
bsakaguchi commented 2026-10-29 21:09:48 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (May 12, 2025):

gdrcopy_copybw :
GPU id:0; name: NVIDIA H800; Bus id: 0000:63:00
GPU id:1; name: NVIDIA H800; Bus id: 0000:67:00
GPU id:2; name: NVIDIA H800; Bus id: 0000:6b:00
GPU id:3; name: NVIDIA H800; Bus id: 0000:6f:00
GPU id:4; name: NVIDIA H800; Bus id: 0000:a3:00
GPU id:5; name: NVIDIA H800; Bus id: 0000:a7:00
GPU id:6; name: NVIDIA H800; Bus id: 0000🆎00
GPU id:7; name: NVIDIA H800; Bus id: 0000:af:00
selecting device 0
testing size: 131072
rounded size: 131072
gpu alloc fn: cuMemAlloc
device ptr: 7fbfb7e00000
map_d_ptr: 0x7fc1e81e7000
info.va: 7fbfb7e00000
info.mapped_size: 131072
info.page_size: 65536
info.mapped: 1
info.wc_mapping: 1
page offset: 0
user-space pointer:0x7fc1e81e7000
writing test, size=131072 offset=0 num_iters=10000
write BW: 17884.9MB/s
reading test, size=131072 offset=0 num_iters=100
read BW: 669.866MB/s
unmapping buffer
unpinning buffer
closing gdrdrv

<!-- gh-comment-id:2870558818 --> @CSEEduanyu commented on GitHub (May 12, 2025): gdrcopy_copybw : GPU id:0; name: NVIDIA H800; Bus id: 0000:63:00 GPU id:1; name: NVIDIA H800; Bus id: 0000:67:00 GPU id:2; name: NVIDIA H800; Bus id: 0000:6b:00 GPU id:3; name: NVIDIA H800; Bus id: 0000:6f:00 GPU id:4; name: NVIDIA H800; Bus id: 0000:a3:00 GPU id:5; name: NVIDIA H800; Bus id: 0000:a7:00 GPU id:6; name: NVIDIA H800; Bus id: 0000:ab:00 GPU id:7; name: NVIDIA H800; Bus id: 0000:af:00 selecting device 0 testing size: 131072 rounded size: 131072 gpu alloc fn: cuMemAlloc device ptr: 7fbfb7e00000 map_d_ptr: 0x7fc1e81e7000 info.va: 7fbfb7e00000 info.mapped_size: 131072 info.page_size: 65536 info.mapped: 1 info.wc_mapping: 1 page offset: 0 user-space pointer:0x7fc1e81e7000 writing test, size=131072 offset=0 num_iters=10000 write BW: 17884.9MB/s reading test, size=131072 offset=0 num_iters=100 read BW: 669.866MB/s unmapping buffer unpinning buffer closing gdrdrv
bsakaguchi commented 2026-10-29 21:09:48 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (May 12, 2025):

@CSEEduanyu can you provide the SGlang full log? It includes some important infos of mooncake.

<!-- gh-comment-id:2870818404 --> @ShangmingCai commented on GitHub (May 12, 2025): @CSEEduanyu can you provide the SGlang full log? It includes some important infos of mooncake.
bsakaguchi commented 2026-10-29 21:09:48 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (May 12, 2025):

@CSEEduanyu can you provide the SGlang full log? It includes some important infos of mooncake.

[2025-05-12 03:54:37 DP2 TP2] Init torch distributed ends. mem usage=1.62 GB
[2025-05-12 03:54:37 DP1 TP1] Init torch distributed ends. mem usage=1.66 GB
[2025-05-12 03:54:37 DP0 TP0] Init torch distributed ends. mem usage=1.62 GB
[2025-05-12 03:54:37 DP4 TP4] Init torch distributed ends. mem usage=1.62 GB
[2025-05-12 03:54:37 DP5 TP5] Init torch distributed ends. mem usage=1.66 GB
[2025-05-12 03:54:37 DP3 TP3] Init torch distributed ends. mem usage=1.66 GB
[2025-05-12 03:54:37 DP6 TP6] Init torch distributed ends. mem usage=1.62 GB
[2025-05-12 03:54:37 DP7 TP7] Init torch distributed ends. mem usage=1.66 GB
[2025-05-12 03:54:37 DP1 TP1] Load weight begin. avail mem=76.94 GB
[2025-05-12 03:54:37 DP2 TP2] Load weight begin. avail mem=76.97 GB
[2025-05-12 03:54:37 DP0 TP0] Load weight begin. avail mem=76.97 GB
[2025-05-12 03:54:37 DP4 TP4] Load weight begin. avail mem=76.97 GB
[2025-05-12 03:54:37 DP5 TP5] Load weight begin. avail mem=76.94 GB
[2025-05-12 03:54:37 DP7 TP7] Load weight begin. avail mem=76.94 GB
[2025-05-12 03:54:37 DP6 TP6] Load weight begin. avail mem=76.97 GB
[2025-05-12 03:54:37 DP3 TP3] Load weight begin. avail mem=76.94 GB
[2025-05-12 03:54:37 DP2 TP2] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-05-12 03:54:37 DP3 TP3] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-05-12 03:54:37 DP5 TP5] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-05-12 03:54:37 DP2 TP2] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled.
[2025-05-12 03:54:37 DP3 TP3] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled.
[2025-05-12 03:54:37 DP5 TP5] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled.
[2025-05-12 03:54:37 DP0 TP0] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-05-12 03:54:37 DP7 TP7] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-05-12 03:54:37 DP0 TP0] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled.
[2025-05-12 03:54:37 DP7 TP7] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled.
[2025-05-12 03:54:37 DP6 TP6] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-05-12 03:54:37 DP6 TP6] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled.
[2025-05-12 03:54:37 DP1 TP1] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-05-12 03:54:37 DP1 TP1] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled.
[2025-05-12 03:54:37 DP4 TP4] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-05-12 03:54:37 DP4 TP4] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled.
Loading safetensors checkpoint shards: 0% Completed | 0/163 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 7% Completed | 11/163 [00:00<00:01, 103.79it/s]
Loading safetensors checkpoint shards: 13% Completed | 22/163 [00:00<00:03, 39.21it/s]
Loading safetensors checkpoint shards: 20% Completed | 32/163 [00:00<00:02, 52.62it/s]
Loading safetensors checkpoint shards: 26% Completed | 42/163 [00:00<00:01, 62.67it/s]
Loading safetensors checkpoint shards: 33% Completed | 53/163 [00:00<00:01, 72.97it/s]
Loading safetensors checkpoint shards: 39% Completed | 63/163 [00:00<00:01, 77.80it/s]
Loading safetensors checkpoint shards: 45% Completed | 73/163 [00:01<00:01, 82.56it/s]
Loading safetensors checkpoint shards: 51% Completed | 83/163 [00:01<00:00, 84.92it/s]
Loading safetensors checkpoint shards: 57% Completed | 93/163 [00:01<00:00, 87.16it/s]
Loading safetensors checkpoint shards: 63% Completed | 103/163 [00:01<00:01, 47.33it/s]
Loading safetensors checkpoint shards: 69% Completed | 112/163 [00:01<00:00, 54.33it/s]
Loading safetensors checkpoint shards: 75% Completed | 122/163 [00:01<00:00, 62.85it/s]
Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7842.35it/s]
Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7582.27it/s]
Loading safetensors checkpoint shards: 80% Completed | 131/163 [00:02<00:00, 68.40it/s]
Loading safetensors checkpoint shards: 86% Completed | 140/163 [00:02<00:00, 72.86it/s]
Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7820.67it/s]
Loading safetensors checkpoint shards: 92% Completed | 150/163 [00:02<00:00, 77.65it/s]
Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7442.40it/s]
Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7723.83it/s]
Loading safetensors checkpoint shards: 98% Completed | 160/163 [00:02<00:00, 81.74it/s]
Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7490.29it/s]
Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:02<00:00, 68.99it/s]

Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7762.27it/s]
Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7921.77it/s]
[2025-05-12 03:55:18 DP6 TP6] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.77 GB, mem usage=54.20 GB.
[2025-05-12 03:55:18 DP2 TP2] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.77 GB, mem usage=54.20 GB.
[2025-05-12 03:55:18 DP5 TP5] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.74 GB, mem usage=54.20 GB.
[2025-05-12 03:55:22 DP7 TP7] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.74 GB, mem usage=54.20 GB.
[2025-05-12 03:55:25 DP0 TP0] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.77 GB, mem usage=54.20 GB.
[2025-05-12 03:55:25 DP3 TP3] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.74 GB, mem usage=54.20 GB.
[2025-05-12 03:55:25 DP1 TP1] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.74 GB, mem usage=54.20 GB.
[2025-05-12 03:55:25 DP4 TP4] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.77 GB, mem usage=54.20 GB.
[2025-05-12 03:55:25 DP1 TP1] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB
[2025-05-12 03:55:25 DP0 TP0] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB
[2025-05-12 03:55:25 DP1 TP1] Memory pool end. avail mem=14.26 GB
[2025-05-12 03:55:25 DP0 TP0] Memory pool end. avail mem=14.30 GB
[2025-05-12 03:55:25 DP7 TP7] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB
[2025-05-12 03:55:25 DP4 TP4] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB
[2025-05-12 03:55:25 DP5 TP5] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB
[2025-05-12 03:55:25 DP7 TP7] Memory pool end. avail mem=14.26 GB
[2025-05-12 03:55:25 DP3 TP3] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB
[2025-05-12 03:55:25 DP6 TP6] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB
[2025-05-12 03:55:25 DP4 TP4] Memory pool end. avail mem=14.30 GB
[2025-05-12 03:55:25 DP5 TP5] Memory pool end. avail mem=14.26 GB
[2025-05-12 03:55:25 DP6 TP6] Memory pool end. avail mem=14.30 GB
[2025-05-12 03:55:25 DP3 TP3] Memory pool end. avail mem=14.26 GB
[2025-05-12 03:55:25 DP2 TP2] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB
[2025-05-12 03:55:25 DP2 TP2] Memory pool end. avail mem=14.30 GB
[2025-05-12 03:55:26 DP2 TP2] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840
[2025-05-12 03:55:26 DP4 TP4] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840
[2025-05-12 03:55:26 DP5 TP5] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840
[2025-05-12 03:55:26 DP6 TP6] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840
[2025-05-12 03:55:26 DP0 TP0] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840
[2025-05-12 03:55:26 DP3 TP3] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840
[2025-05-12 03:55:26 DP7 TP7] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840
[2025-05-12 03:55:26 DP1 TP1] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0512 03:55:27.005717 5870 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0512 03:55:27.005769 5870 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 15919 for serving local TCP service
I0512 03:55:27.010327 5870 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0512 03:55:27.013155 5872 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0512 03:55:27.013209 5872 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 16562 for serving local TCP service
I0512 03:55:27.013949 5870 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0512 03:55:27.013974 5873 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0512 03:55:27.014008 5873 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 15335 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0512 03:55:27.014741 5874 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0512 03:55:27.014777 5874 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 16707 for serving local TCP service
I0512 03:55:27.017220 5872 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42
I0512 03:55:27.017621 5870 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72
I0512 03:55:27.018044 5873 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0512 03:55:27.018245 5868 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0512 03:55:27.018282 5868 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 15726 for serving local TCP service
I0512 03:55:27.018779 5874 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42
I0512 03:55:27.020735 5872 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2
I0512 03:55:27.021301 5870 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2
I0512 03:55:27.021633 5873 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2
I0512 03:55:27.022352 5868 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42
I0512 03:55:27.022379 5874 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2
I0512 03:55:27.024204 5872 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72
I0512 03:55:27.024894 5870 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72
I0512 03:55:27.025257 5873 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72
I0512 03:55:27.025880 5874 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72
I0512 03:55:27.025930 5868 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2
I0512 03:55:27.027941 5872 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2
I0512 03:55:27.028431 5870 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52
I0512 03:55:27.028836 5873 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2
I0512 03:55:27.029341 5874 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2
I0512 03:55:27.029419 5868 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72
I0512 03:55:27.031359 5872 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72
I0512 03:55:27.031904 5870 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2
I0512 03:55:27.032290 5873 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72
I0512 03:55:27.032727 5874 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72
I0512 03:55:27.032902 5868 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2
I0512 03:55:27.034749 5872 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52
I0512 03:55:27.035408 5870 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22
I0512 03:55:27.035735 5873 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52
I0512 03:55:27.036300 5874 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52
I0512 03:55:27.036355 5868 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0512 03:55:27.037158 5871 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0512 03:55:27.037205 5871 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 16192 for serving local TCP service
I0512 03:55:27.038232 5872 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2
I0512 03:55:27.039425 5873 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2
I0512 03:55:27.039816 5868 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52
I0512 03:55:27.039844 5874 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2
I0512 03:55:27.041188 5871 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42
I0512 03:55:27.041649 5872 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22
I0512 03:55:27.042852 5873 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22
I0512 03:55:27.043226 5868 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2
I0512 03:55:27.043341 5874 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0512 03:55:27.044507 5869 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0512 03:55:27.044554 5869 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 15268 for serving local TCP service
I0512 03:55:27.044806 5871 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2
I0512 03:55:27.046648 5868 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22
I0512 03:55:27.048596 5869 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42
I0512 03:55:27.049125 5871 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72
I0512 03:55:27.052249 5869 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2
I0512 03:55:27.053426 5871 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2
I0512 03:55:27.055922 5869 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72
I0512 03:55:27.057705 5871 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0512 03:55:27.059497 5875 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0512 03:55:27.059531 5875 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 16363 for serving local TCP service
I0512 03:55:27.059610 5869 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2
I0512 03:55:27.061990 5871 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52
I0512 03:55:27.063253 5869 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72
I0512 03:55:27.063684 5875 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42
I0512 03:55:27.066206 5871 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2
I0512 03:55:27.067595 5869 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52
I0512 03:55:27.068056 5875 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2
I0512 03:55:27.070348 5871 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22
I0512 03:55:27.071851 5869 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2
I0512 03:55:27.072325 5875 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72
I0512 03:55:27.076032 5869 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22
I0512 03:55:27.076732 5875 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2
I0512 03:55:27.080926 5875 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72
I0512 03:55:27.085055 5875 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52
I0512 03:55:27.089196 5875 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2
I0512 03:55:27.093334 5875 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22
E0512 03:55:27.776799 5872 rdma_context.cpp:198] Failed to register memory 0x2e1cba0200: Bad address [14]
[2025-05-12 03:55:27 DP4 TP4] Mooncake memory registration failed.
[2025-05-12 03:55:27 DP4 TP4] Scheduler hit an exception: Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 2215, in run_scheduler_process
scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, pp_rank, dp_rank)
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 449, in init
self.init_disaggregation()
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 618, in init_disaggregation
self.disagg_prefill_bootstrap_queue = PrefillBootstrapQueue(
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/disaggregation/prefill.py", line 80, in init
self.kv_manager = self._init_kv_manager()
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/disaggregation/prefill.py", line 114, in _init_kv_manager
kv_manager = kv_manager_class(
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/disaggregation/mooncake/conn.py", line 141, in init
self.register_buffer_to_engine()
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/disaggregation/mooncake/conn.py", line 167, in register_buffer_to_engine
self.engine.register(kv_data_ptr, kv_data_len)
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/disaggregation/mooncake/transfer_engine.py", line 36, in register
raise RuntimeError("Mooncake memory registration failed.")
RuntimeError: Mooncake memory registration failed.

<!-- gh-comment-id:2870912900 --> @CSEEduanyu commented on GitHub (May 12, 2025): > [@CSEEduanyu](https://github.com/CSEEduanyu) can you provide the SGlang full log? It includes some important infos of mooncake. [2025-05-12 03:54:37 DP2 TP2] Init torch distributed ends. mem usage=1.62 GB [2025-05-12 03:54:37 DP1 TP1] Init torch distributed ends. mem usage=1.66 GB [2025-05-12 03:54:37 DP0 TP0] Init torch distributed ends. mem usage=1.62 GB [2025-05-12 03:54:37 DP4 TP4] Init torch distributed ends. mem usage=1.62 GB [2025-05-12 03:54:37 DP5 TP5] Init torch distributed ends. mem usage=1.66 GB [2025-05-12 03:54:37 DP3 TP3] Init torch distributed ends. mem usage=1.66 GB [2025-05-12 03:54:37 DP6 TP6] Init torch distributed ends. mem usage=1.62 GB [2025-05-12 03:54:37 DP7 TP7] Init torch distributed ends. mem usage=1.66 GB [2025-05-12 03:54:37 DP1 TP1] Load weight begin. avail mem=76.94 GB [2025-05-12 03:54:37 DP2 TP2] Load weight begin. avail mem=76.97 GB [2025-05-12 03:54:37 DP0 TP0] Load weight begin. avail mem=76.97 GB [2025-05-12 03:54:37 DP4 TP4] Load weight begin. avail mem=76.97 GB [2025-05-12 03:54:37 DP5 TP5] Load weight begin. avail mem=76.94 GB [2025-05-12 03:54:37 DP7 TP7] Load weight begin. avail mem=76.94 GB [2025-05-12 03:54:37 DP6 TP6] Load weight begin. avail mem=76.97 GB [2025-05-12 03:54:37 DP3 TP3] Load weight begin. avail mem=76.94 GB [2025-05-12 03:54:37 DP2 TP2] Detected fp8 checkpoint. Please note that the format is experimental and subject to change. [2025-05-12 03:54:37 DP3 TP3] Detected fp8 checkpoint. Please note that the format is experimental and subject to change. [2025-05-12 03:54:37 DP5 TP5] Detected fp8 checkpoint. Please note that the format is experimental and subject to change. [2025-05-12 03:54:37 DP2 TP2] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled. [2025-05-12 03:54:37 DP3 TP3] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled. [2025-05-12 03:54:37 DP5 TP5] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled. [2025-05-12 03:54:37 DP0 TP0] Detected fp8 checkpoint. Please note that the format is experimental and subject to change. [2025-05-12 03:54:37 DP7 TP7] Detected fp8 checkpoint. Please note that the format is experimental and subject to change. [2025-05-12 03:54:37 DP0 TP0] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled. [2025-05-12 03:54:37 DP7 TP7] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled. [2025-05-12 03:54:37 DP6 TP6] Detected fp8 checkpoint. Please note that the format is experimental and subject to change. [2025-05-12 03:54:37 DP6 TP6] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled. [2025-05-12 03:54:37 DP1 TP1] Detected fp8 checkpoint. Please note that the format is experimental and subject to change. [2025-05-12 03:54:37 DP1 TP1] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled. [2025-05-12 03:54:37 DP4 TP4] Detected fp8 checkpoint. Please note that the format is experimental and subject to change. [2025-05-12 03:54:37 DP4 TP4] Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled. Loading safetensors checkpoint shards: 0% Completed | 0/163 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 7% Completed | 11/163 [00:00<00:01, 103.79it/s] Loading safetensors checkpoint shards: 13% Completed | 22/163 [00:00<00:03, 39.21it/s] Loading safetensors checkpoint shards: 20% Completed | 32/163 [00:00<00:02, 52.62it/s] Loading safetensors checkpoint shards: 26% Completed | 42/163 [00:00<00:01, 62.67it/s] Loading safetensors checkpoint shards: 33% Completed | 53/163 [00:00<00:01, 72.97it/s] Loading safetensors checkpoint shards: 39% Completed | 63/163 [00:00<00:01, 77.80it/s] Loading safetensors checkpoint shards: 45% Completed | 73/163 [00:01<00:01, 82.56it/s] Loading safetensors checkpoint shards: 51% Completed | 83/163 [00:01<00:00, 84.92it/s] Loading safetensors checkpoint shards: 57% Completed | 93/163 [00:01<00:00, 87.16it/s] Loading safetensors checkpoint shards: 63% Completed | 103/163 [00:01<00:01, 47.33it/s] Loading safetensors checkpoint shards: 69% Completed | 112/163 [00:01<00:00, 54.33it/s] Loading safetensors checkpoint shards: 75% Completed | 122/163 [00:01<00:00, 62.85it/s] Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7842.35it/s] Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7582.27it/s] Loading safetensors checkpoint shards: 80% Completed | 131/163 [00:02<00:00, 68.40it/s] Loading safetensors checkpoint shards: 86% Completed | 140/163 [00:02<00:00, 72.86it/s] Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7820.67it/s] Loading safetensors checkpoint shards: 92% Completed | 150/163 [00:02<00:00, 77.65it/s] Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7442.40it/s] Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7723.83it/s] Loading safetensors checkpoint shards: 98% Completed | 160/163 [00:02<00:00, 81.74it/s] Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7490.29it/s] Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:02<00:00, 68.99it/s] Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7762.27it/s] Cloning 16 replicas of the shared expert into MoE: 100%|████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 7921.77it/s] [2025-05-12 03:55:18 DP6 TP6] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.77 GB, mem usage=54.20 GB. [2025-05-12 03:55:18 DP2 TP2] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.77 GB, mem usage=54.20 GB. [2025-05-12 03:55:18 DP5 TP5] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.74 GB, mem usage=54.20 GB. [2025-05-12 03:55:22 DP7 TP7] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.74 GB, mem usage=54.20 GB. [2025-05-12 03:55:25 DP0 TP0] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.77 GB, mem usage=54.20 GB. [2025-05-12 03:55:25 DP3 TP3] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.74 GB, mem usage=54.20 GB. [2025-05-12 03:55:25 DP1 TP1] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.74 GB, mem usage=54.20 GB. [2025-05-12 03:55:25 DP4 TP4] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=22.77 GB, mem usage=54.20 GB. [2025-05-12 03:55:25 DP1 TP1] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB [2025-05-12 03:55:25 DP0 TP0] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB [2025-05-12 03:55:25 DP1 TP1] Memory pool end. avail mem=14.26 GB [2025-05-12 03:55:25 DP0 TP0] Memory pool end. avail mem=14.30 GB [2025-05-12 03:55:25 DP7 TP7] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB [2025-05-12 03:55:25 DP4 TP4] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB [2025-05-12 03:55:25 DP5 TP5] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB [2025-05-12 03:55:25 DP7 TP7] Memory pool end. avail mem=14.26 GB [2025-05-12 03:55:25 DP3 TP3] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB [2025-05-12 03:55:25 DP6 TP6] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB [2025-05-12 03:55:25 DP4 TP4] Memory pool end. avail mem=14.30 GB [2025-05-12 03:55:25 DP5 TP5] Memory pool end. avail mem=14.26 GB [2025-05-12 03:55:25 DP6 TP6] Memory pool end. avail mem=14.30 GB [2025-05-12 03:55:25 DP3 TP3] Memory pool end. avail mem=14.26 GB [2025-05-12 03:55:25 DP2 TP2] KV Cache is allocated. #tokens: 110356, KV size: 7.22 GB [2025-05-12 03:55:25 DP2 TP2] Memory pool end. avail mem=14.30 GB [2025-05-12 03:55:26 DP2 TP2] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840 [2025-05-12 03:55:26 DP4 TP4] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840 [2025-05-12 03:55:26 DP5 TP5] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840 [2025-05-12 03:55:26 DP6 TP6] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840 [2025-05-12 03:55:26 DP0 TP0] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840 [2025-05-12 03:55:26 DP3 TP3] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840 [2025-05-12 03:55:26 DP7 TP7] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840 [2025-05-12 03:55:26 DP1 TP1] max_total_num_tokens=110356, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2049, context_len=163840 WARNING: Logging before InitGoogleLogging() is written to STDERR I0512 03:55:27.005717 5870 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0512 03:55:27.005769 5870 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 15919 for serving local TCP service I0512 03:55:27.010327 5870 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42 WARNING: Logging before InitGoogleLogging() is written to STDERR I0512 03:55:27.013155 5872 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0512 03:55:27.013209 5872 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 16562 for serving local TCP service I0512 03:55:27.013949 5870 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2 WARNING: Logging before InitGoogleLogging() is written to STDERR I0512 03:55:27.013974 5873 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0512 03:55:27.014008 5873 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 15335 for serving local TCP service WARNING: Logging before InitGoogleLogging() is written to STDERR I0512 03:55:27.014741 5874 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0512 03:55:27.014777 5874 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 16707 for serving local TCP service I0512 03:55:27.017220 5872 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42 I0512 03:55:27.017621 5870 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72 I0512 03:55:27.018044 5873 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42 WARNING: Logging before InitGoogleLogging() is written to STDERR I0512 03:55:27.018245 5868 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0512 03:55:27.018282 5868 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 15726 for serving local TCP service I0512 03:55:27.018779 5874 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42 I0512 03:55:27.020735 5872 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2 I0512 03:55:27.021301 5870 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2 I0512 03:55:27.021633 5873 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2 I0512 03:55:27.022352 5868 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42 I0512 03:55:27.022379 5874 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2 I0512 03:55:27.024204 5872 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72 I0512 03:55:27.024894 5870 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72 I0512 03:55:27.025257 5873 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72 I0512 03:55:27.025880 5874 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72 I0512 03:55:27.025930 5868 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2 I0512 03:55:27.027941 5872 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2 I0512 03:55:27.028431 5870 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52 I0512 03:55:27.028836 5873 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2 I0512 03:55:27.029341 5874 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2 I0512 03:55:27.029419 5868 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72 I0512 03:55:27.031359 5872 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72 I0512 03:55:27.031904 5870 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2 I0512 03:55:27.032290 5873 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72 I0512 03:55:27.032727 5874 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72 I0512 03:55:27.032902 5868 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2 I0512 03:55:27.034749 5872 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52 I0512 03:55:27.035408 5870 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22 I0512 03:55:27.035735 5873 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52 I0512 03:55:27.036300 5874 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52 I0512 03:55:27.036355 5868 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72 WARNING: Logging before InitGoogleLogging() is written to STDERR I0512 03:55:27.037158 5871 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0512 03:55:27.037205 5871 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 16192 for serving local TCP service I0512 03:55:27.038232 5872 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2 I0512 03:55:27.039425 5873 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2 I0512 03:55:27.039816 5868 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52 I0512 03:55:27.039844 5874 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2 I0512 03:55:27.041188 5871 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42 I0512 03:55:27.041649 5872 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22 I0512 03:55:27.042852 5873 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22 I0512 03:55:27.043226 5868 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2 I0512 03:55:27.043341 5874 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22 WARNING: Logging before InitGoogleLogging() is written to STDERR I0512 03:55:27.044507 5869 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0512 03:55:27.044554 5869 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 15268 for serving local TCP service I0512 03:55:27.044806 5871 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2 I0512 03:55:27.046648 5868 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22 I0512 03:55:27.048596 5869 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42 I0512 03:55:27.049125 5871 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72 I0512 03:55:27.052249 5869 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2 I0512 03:55:27.053426 5871 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2 I0512 03:55:27.055922 5869 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72 I0512 03:55:27.057705 5871 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72 WARNING: Logging before InitGoogleLogging() is written to STDERR I0512 03:55:27.059497 5875 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0512 03:55:27.059531 5875 transfer_engine.cpp:91] Transfer Engine uses address 10.92.130.219 and port 16363 for serving local TCP service I0512 03:55:27.059610 5869 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2 I0512 03:55:27.061990 5871 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52 I0512 03:55:27.063253 5869 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72 I0512 03:55:27.063684 5875 rdma_context.cpp:125] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:2d:42 I0512 03:55:27.066206 5871 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2 I0512 03:55:27.067595 5869 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52 I0512 03:55:27.068056 5875 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:34:b2 I0512 03:55:27.070348 5871 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22 I0512 03:55:27.071851 5869 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2 I0512 03:55:27.072325 5875 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:36:72 I0512 03:55:27.076032 5869 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22 I0512 03:55:27.076732 5875 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:37:c2 I0512 03:55:27.080926 5875 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:10:72 I0512 03:55:27.085055 5875 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:13:52 I0512 03:55:27.089196 5875 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:19:f2 I0512 03:55:27.093334 5875 rdma_context.cpp:125] RDMA device: mlx5_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:19:16:26:22 E0512 03:55:27.776799 5872 rdma_context.cpp:198] Failed to register memory 0x2e1cba0200: Bad address [14] [2025-05-12 03:55:27 DP4 TP4] Mooncake memory registration failed. [2025-05-12 03:55:27 DP4 TP4] Scheduler hit an exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 2215, in run_scheduler_process scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, pp_rank, dp_rank) File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 449, in __init__ self.init_disaggregation() File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 618, in init_disaggregation self.disagg_prefill_bootstrap_queue = PrefillBootstrapQueue( File "/usr/local/lib/python3.10/dist-packages/sglang/srt/disaggregation/prefill.py", line 80, in __init__ self.kv_manager = self._init_kv_manager() File "/usr/local/lib/python3.10/dist-packages/sglang/srt/disaggregation/prefill.py", line 114, in _init_kv_manager kv_manager = kv_manager_class( File "/usr/local/lib/python3.10/dist-packages/sglang/srt/disaggregation/mooncake/conn.py", line 141, in __init__ self.register_buffer_to_engine() File "/usr/local/lib/python3.10/dist-packages/sglang/srt/disaggregation/mooncake/conn.py", line 167, in register_buffer_to_engine self.engine.register(kv_data_ptr, kv_data_len) File "/usr/local/lib/python3.10/dist-packages/sglang/srt/disaggregation/mooncake/transfer_engine.py", line 36, in register raise RuntimeError("Mooncake memory registration failed.") RuntimeError: Mooncake memory registration failed.
bsakaguchi commented 2026-10-29 21:09:49 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (May 12, 2025):

@ShangmingCai Is there any additional information I need to provide? I'm using a container rather than a physical machine, but I was able to run deepep normally before

<!-- gh-comment-id:2870930007 --> @CSEEduanyu commented on GitHub (May 12, 2025): @ShangmingCai Is there any additional information I need to provide? I'm using a container rather than a physical machine, but I was able to run deepep normally before
bsakaguchi commented 2026-10-29 21:09:49 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (May 12, 2025):

@alogfans do you have any idea of this? The init logs seem normal to me. This is very odd.

<!-- gh-comment-id:2870940957 --> @ShangmingCai commented on GitHub (May 12, 2025): @alogfans do you have any idea of this? The init logs seem normal to me. This is very odd.
bsakaguchi commented 2026-10-29 21:09:49 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (May 12, 2025):

https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html
This indeed solved my problem, but I don't know why。

<!-- gh-comment-id:2871003343 --> @CSEEduanyu commented on GitHub (May 12, 2025): https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html This indeed solved my problem, but I don't know why。
bsakaguchi commented 2026-10-29 21:09:49 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (May 12, 2025):

https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html This indeed solved my problem, but I don't know why。

@CSEEduanyu This could relate to the root permission. If you are using docker, you might need to use sudo to start the container with 3rd-party library GDR support.

<!-- gh-comment-id:2871041533 --> @ShangmingCai commented on GitHub (May 12, 2025): > https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html This indeed solved my problem, but I don't know why。 @CSEEduanyu This could relate to the root permission. If you are using docker, you might need to use sudo to start the container with 3rd-party library GDR support.
bsakaguchi commented 2026-10-29 21:09:49 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (May 12, 2025):

https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html This indeed solved my problem, but I don't know why。

@CSEEduanyu This could relate to the root permission. If you are using docker, you might need to use sudo to start the container with 3rd-party library GDR support.

But strangely, the test for gdrcopy_copybw was normal

<!-- gh-comment-id:2871072979 --> @CSEEduanyu commented on GitHub (May 12, 2025): > > https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html This indeed solved my problem, but I don't know why。 > > [@CSEEduanyu](https://github.com/CSEEduanyu) This could relate to the root permission. If you are using docker, you might need to use sudo to start the container with 3rd-party library GDR support. But strangely, the test for gdrcopy_copybw was normal
bsakaguchi commented 2026-10-29 21:09:49 +00:00
Author
Owner
Copy link

@YosanHo commented on GitHub (May 13, 2025):

same error for me

I0513 13:21:20.363459 72079 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.363724 72079 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15436 for serving local TCP service
I0513 13:21:20.374280 72079 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/
I0513 13:21:20.375334 72079 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.395785 72080 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.395951 72080 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15986 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.403113 72076 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.403254 72076 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15505 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.405055 72078 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.405176 72078 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16252 for serving local TCP service
I0513 13:21:20.406009 72080 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/
I0513 13:21:20.406737 72080 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.413020 72076 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/
I0513 13:21:20.413455 72078 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/
I0513 13:21:20.413678 72076 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.414115 72078 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.418197 72082 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.418345 72082 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16068 for serving local TCP service
I0513 13:21:20.428444 72082 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/
I0513 13:21:20.429111 72082 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.430963 72075 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.431088 72075 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16536 for serving local TCP service
I0513 13:21:20.440829 72075 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/
I0513 13:21:20.441489 72075 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.462790 72081 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.462942 72081 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15336 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.469861 72077 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.470021 72077 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16851 for serving local TCP service
I0513 13:21:20.474447 72081 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/
I0513 13:21:20.475225 72081 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.480049 72077 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/
I0513 13:21:20.480682 72077 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
E0513 13:21:20.956470 72079 rdma_context.cpp:198] Failed to register memory 0x7f555a000000: Bad address [14]
[2025-05-13 13:21:20 TP4] Mooncake memory registration failed.
[2025-05-13 13:21:20 TP4] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2145, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 435, in __init__
    self.init_disaggregation()
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 605, in init_disaggregation
    self.disagg_prefill_bootstrap_queue = PrefillBootstrapQueue(
                                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 80, in __init__
    self.kv_manager = self._init_kv_manager()
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 114, in _init_kv_manager
    kv_manager = kv_manager_class(
                 ^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 141, in __init__
    self.register_buffer_to_engine()
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 167, in register_buffer_to_engine
    self.engine.register(kv_data_ptr, kv_data_len)
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/transfer_engine.py", line 36, in register
    raise RuntimeError("Mooncake memory registration failed.")
RuntimeError: Mooncake memory registration failed.

check_env:

Python: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H20
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.8, V12.8.93
CUDA Driver Version: 550.144.03
PyTorch: 2.6.0+cu124
sglang: 0.4.6.post3
sgl_kernel: 0.1.2.post1
flashinfer_python: 0.2.5
triton: 3.2.0
transformers: 4.51.1
torchao: 0.11.0
numpy: 2.2.5
aiohttp: 3.11.18
fastapi: 0.115.12
hf_transfer: 0.1.9
huggingface_hub: 0.31.1
interegular: 0.3.3
modelscope: 1.25.0
orjson: 3.10.18
outlines: 0.1.11
packaging: 24.1
psutil: 7.0.0
pydantic: 2.11.4
python-multipart: 0.0.20
pyzmq: 26.4.0
uvicorn: 0.34.2
uvloop: 0.21.0
vllm: 0.8.5
xgrammar: 0.1.19
openai: 1.75.0
tiktoken: 0.9.0
anthropic: 0.51.0
litellm: 1.69.1
decord: 0.6.0
NVIDIA Topology: 
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     PIX     SYS     NODE    0-105   0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     PIX     SYS     NODE    0-105   0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     NODE    SYS     PIX     0-105   0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     SYS     SYS     NODE    SYS     PIX     0-105   0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    NODE    NODE    NODE    SYS     PIX     SYS     106-211 1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    NODE    NODE    NODE    SYS     PIX     SYS     106-211 1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    PIX     NODE    NODE    SYS     NODE    SYS     106-211 1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      PIX     NODE    NODE    SYS     NODE    SYS     106-211 1               N/A
NIC0    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PIX      X      NODE    NODE    SYS     NODE    SYS
NIC1    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE     X      PIX     SYS     NODE    SYS
NIC2    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE    PIX      X      SYS     NODE    SYS
NIC3    PIX     PIX     NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     NODE
NIC4    SYS     SYS     SYS     SYS     PIX     PIX     NODE    NODE    NODE    NODE    NODE    SYS      X      SYS
NIC5    NODE    NODE    PIX     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    SYS      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4"temp"
  NIC5: mlx5_5"temp"


Hypervisor vendor: KVM
ulimit soft: 1048576
<!-- gh-comment-id:2874939438 --> @YosanHo commented on GitHub (May 13, 2025): same error for me ``` I0513 13:21:20.363459 72079 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0513 13:21:20.363724 72079 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15436 for serving local TCP service I0513 13:21:20.374280 72079 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/ I0513 13:21:20.375334 72079 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 WARNING: Logging before InitGoogleLogging() is written to STDERR I0513 13:21:20.395785 72080 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0513 13:21:20.395951 72080 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15986 for serving local TCP service WARNING: Logging before InitGoogleLogging() is written to STDERR I0513 13:21:20.403113 72076 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0513 13:21:20.403254 72076 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15505 for serving local TCP service WARNING: Logging before InitGoogleLogging() is written to STDERR I0513 13:21:20.405055 72078 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0513 13:21:20.405176 72078 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16252 for serving local TCP service I0513 13:21:20.406009 72080 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/ I0513 13:21:20.406737 72080 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 I0513 13:21:20.413020 72076 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/ I0513 13:21:20.413455 72078 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/ I0513 13:21:20.413678 72076 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 I0513 13:21:20.414115 72078 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 WARNING: Logging before InitGoogleLogging() is written to STDERR I0513 13:21:20.418197 72082 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0513 13:21:20.418345 72082 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16068 for serving local TCP service I0513 13:21:20.428444 72082 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/ I0513 13:21:20.429111 72082 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 WARNING: Logging before InitGoogleLogging() is written to STDERR I0513 13:21:20.430963 72075 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0513 13:21:20.431088 72075 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16536 for serving local TCP service I0513 13:21:20.440829 72075 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/ I0513 13:21:20.441489 72075 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 WARNING: Logging before InitGoogleLogging() is written to STDERR I0513 13:21:20.462790 72081 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0513 13:21:20.462942 72081 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15336 for serving local TCP service WARNING: Logging before InitGoogleLogging() is written to STDERR I0513 13:21:20.469861 72077 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0513 13:21:20.470021 72077 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16851 for serving local TCP service I0513 13:21:20.474447 72081 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/ I0513 13:21:20.475225 72081 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 I0513 13:21:20.480049 72077 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/ I0513 13:21:20.480682 72077 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 E0513 13:21:20.956470 72079 rdma_context.cpp:198] Failed to register memory 0x7f555a000000: Bad address [14] [2025-05-13 13:21:20 TP4] Mooncake memory registration failed. [2025-05-13 13:21:20 TP4] Scheduler hit an exception: Traceback (most recent call last): File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2145, in run_scheduler_process scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 435, in __init__ self.init_disaggregation() File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 605, in init_disaggregation self.disagg_prefill_bootstrap_queue = PrefillBootstrapQueue( ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 80, in __init__ self.kv_manager = self._init_kv_manager() ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 114, in _init_kv_manager kv_manager = kv_manager_class( ^^^^^^^^^^^^^^^^^ File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 141, in __init__ self.register_buffer_to_engine() File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 167, in register_buffer_to_engine self.engine.register(kv_data_ptr, kv_data_len) File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/transfer_engine.py", line 36, in register raise RuntimeError("Mooncake memory registration failed.") RuntimeError: Mooncake memory registration failed. ``` check_env: ``` Python: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb 6 2025, 18:56:27) [GCC 11.2.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA H20 GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.8, V12.8.93 CUDA Driver Version: 550.144.03 PyTorch: 2.6.0+cu124 sglang: 0.4.6.post3 sgl_kernel: 0.1.2.post1 flashinfer_python: 0.2.5 triton: 3.2.0 transformers: 4.51.1 torchao: 0.11.0 numpy: 2.2.5 aiohttp: 3.11.18 fastapi: 0.115.12 hf_transfer: 0.1.9 huggingface_hub: 0.31.1 interegular: 0.3.3 modelscope: 1.25.0 orjson: 3.10.18 outlines: 0.1.11 packaging: 24.1 psutil: 7.0.0 pydantic: 2.11.4 python-multipart: 0.0.20 pyzmq: 26.4.0 uvicorn: 0.34.2 uvloop: 0.21.0 vllm: 0.8.5 xgrammar: 0.1.19 openai: 1.75.0 tiktoken: 0.9.0 anthropic: 0.51.0 litellm: 1.69.1 decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS PIX SYS NODE 0-105 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS PIX SYS NODE 0-105 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS SYS NODE SYS PIX 0-105 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS SYS NODE SYS PIX 0-105 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 NODE NODE NODE SYS PIX SYS 106-211 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 NODE NODE NODE SYS PIX SYS 106-211 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 PIX NODE NODE SYS NODE SYS 106-211 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X PIX NODE NODE SYS NODE SYS 106-211 1 N/A NIC0 SYS SYS SYS SYS NODE NODE PIX PIX X NODE NODE SYS NODE SYS NIC1 SYS SYS SYS SYS NODE NODE NODE NODE NODE X PIX SYS NODE SYS NIC2 SYS SYS SYS SYS NODE NODE NODE NODE NODE PIX X SYS NODE SYS NIC3 PIX PIX NODE NODE SYS SYS SYS SYS SYS SYS SYS X SYS NODE NIC4 SYS SYS SYS SYS PIX PIX NODE NODE NODE NODE NODE SYS X SYS NIC5 NODE NODE PIX PIX SYS SYS SYS SYS SYS SYS SYS NODE SYS X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4"temp" NIC5: mlx5_5"temp" Hypervisor vendor: KVM ulimit soft: 1048576 ```
bsakaguchi commented 2026-10-29 21:09:50 +00:00
Author
Owner
Copy link

@stmatengss commented on GitHub (May 13, 2025):

same error for me

I0513 13:21:20.363459 72079 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.363724 72079 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15436 for serving local TCP service
I0513 13:21:20.374280 72079 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.375334 72079 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.395785 72080 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.395951 72080 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15986 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.403113 72076 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.403254 72076 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15505 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.405055 72078 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.405176 72078 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16252 for serving local TCP service
I0513 13:21:20.406009 72080 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.406737 72080 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.413020 72076 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.413455 72078 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.413678 72076 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.414115 72078 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.418197 72082 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.418345 72082 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16068 for serving local TCP service
I0513 13:21:20.428444 72082 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.429111 72082 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.430963 72075 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.431088 72075 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16536 for serving local TCP service
I0513 13:21:20.440829 72075 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.441489 72075 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.462790 72081 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.462942 72081 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15336 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.469861 72077 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.470021 72077 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16851 for serving local TCP service
I0513 13:21:20.474447 72081 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.475225 72081 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.480049 72077 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.480682 72077 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
E0513 13:21:20.956470 72079 rdma_context.cpp:198] Failed to register memory 0x7f555a000000: Bad address [14]
[2025-05-13 13:21:20 TP4] Mooncake memory registration failed.
[2025-05-13 13:21:20 TP4] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2145, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 435, in __init__
    self.init_disaggregation()
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 605, in init_disaggregation
    self.disagg_prefill_bootstrap_queue = PrefillBootstrapQueue(
                                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 80, in __init__
    self.kv_manager = self._init_kv_manager()
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 114, in _init_kv_manager
    kv_manager = kv_manager_class(
                 ^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 141, in __init__
    self.register_buffer_to_engine()
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 167, in register_buffer_to_engine
    self.engine.register(kv_data_ptr, kv_data_len)
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/transfer_engine.py", line 36, in register
    raise RuntimeError("Mooncake memory registration failed.")
RuntimeError: Mooncake memory registration failed.

check_env:

Python: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H20
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.8, V12.8.93
CUDA Driver Version: 550.144.03
PyTorch: 2.6.0+cu124
sglang: 0.4.6.post3
sgl_kernel: 0.1.2.post1
flashinfer_python: 0.2.5
triton: 3.2.0
transformers: 4.51.1
torchao: 0.11.0
numpy: 2.2.5
aiohttp: 3.11.18
fastapi: 0.115.12
hf_transfer: 0.1.9
huggingface_hub: 0.31.1
interegular: 0.3.3
modelscope: 1.25.0
orjson: 3.10.18
outlines: 0.1.11
packaging: 24.1
psutil: 7.0.0
pydantic: 2.11.4
python-multipart: 0.0.20
pyzmq: 26.4.0
uvicorn: 0.34.2
uvloop: 0.21.0
vllm: 0.8.5
xgrammar: 0.1.19
openai: 1.75.0
tiktoken: 0.9.0
anthropic: 0.51.0
litellm: 1.69.1
decord: 0.6.0
NVIDIA Topology: 
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     PIX     SYS     NODE    0-105   0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     PIX     SYS     NODE    0-105   0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     NODE    SYS     PIX     0-105   0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     SYS     SYS     NODE    SYS     PIX     0-105   0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    NODE    NODE    NODE    SYS     PIX     SYS     106-211 1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    NODE    NODE    NODE    SYS     PIX     SYS     106-211 1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    PIX     NODE    NODE    SYS     NODE    SYS     106-211 1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      PIX     NODE    NODE    SYS     NODE    SYS     106-211 1               N/A
NIC0    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PIX      X      NODE    NODE    SYS     NODE    SYS
NIC1    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE     X      PIX     SYS     NODE    SYS
NIC2    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE    PIX      X      SYS     NODE    SYS
NIC3    PIX     PIX     NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     NODE
NIC4    SYS     SYS     SYS     SYS     PIX     PIX     NODE    NODE    NODE    NODE    NODE    SYS      X      SYS
NIC5    NODE    NODE    PIX     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    SYS      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4"temp"
  NIC5: mlx5_5"temp"


Hypervisor vendor: KVM
ulimit soft: 1048576

Could you use tools to test the availability of GDR (e..g., gdrcopy)?

<!-- gh-comment-id:2875508528 --> @stmatengss commented on GitHub (May 13, 2025): > same error for me > > ``` > I0513 13:21:20.363459 72079 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > I0513 13:21:20.363724 72079 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15436 for serving local TCP service > I0513 13:21:20.374280 72079 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > I0513 13:21:20.375334 72079 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > WARNING: Logging before InitGoogleLogging() is written to STDERR > I0513 13:21:20.395785 72080 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > I0513 13:21:20.395951 72080 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15986 for serving local TCP service > WARNING: Logging before InitGoogleLogging() is written to STDERR > I0513 13:21:20.403113 72076 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > I0513 13:21:20.403254 72076 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15505 for serving local TCP service > WARNING: Logging before InitGoogleLogging() is written to STDERR > I0513 13:21:20.405055 72078 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > I0513 13:21:20.405176 72078 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16252 for serving local TCP service > I0513 13:21:20.406009 72080 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > I0513 13:21:20.406737 72080 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > I0513 13:21:20.413020 72076 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > I0513 13:21:20.413455 72078 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > I0513 13:21:20.413678 72076 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > I0513 13:21:20.414115 72078 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > WARNING: Logging before InitGoogleLogging() is written to STDERR > I0513 13:21:20.418197 72082 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > I0513 13:21:20.418345 72082 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16068 for serving local TCP service > I0513 13:21:20.428444 72082 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > I0513 13:21:20.429111 72082 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > WARNING: Logging before InitGoogleLogging() is written to STDERR > I0513 13:21:20.430963 72075 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > I0513 13:21:20.431088 72075 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16536 for serving local TCP service > I0513 13:21:20.440829 72075 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > I0513 13:21:20.441489 72075 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > WARNING: Logging before InitGoogleLogging() is written to STDERR > I0513 13:21:20.462790 72081 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > I0513 13:21:20.462942 72081 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15336 for serving local TCP service > WARNING: Logging before InitGoogleLogging() is written to STDERR > I0513 13:21:20.469861 72077 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > I0513 13:21:20.470021 72077 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16851 for serving local TCP service > I0513 13:21:20.474447 72081 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > I0513 13:21:20.475225 72081 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > I0513 13:21:20.480049 72077 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > I0513 13:21:20.480682 72077 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > E0513 13:21:20.956470 72079 rdma_context.cpp:198] Failed to register memory 0x7f555a000000: Bad address [14] > [2025-05-13 13:21:20 TP4] Mooncake memory registration failed. > [2025-05-13 13:21:20 TP4] Scheduler hit an exception: Traceback (most recent call last): > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2145, in run_scheduler_process > scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 435, in __init__ > self.init_disaggregation() > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 605, in init_disaggregation > self.disagg_prefill_bootstrap_queue = PrefillBootstrapQueue( > ^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 80, in __init__ > self.kv_manager = self._init_kv_manager() > ^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 114, in _init_kv_manager > kv_manager = kv_manager_class( > ^^^^^^^^^^^^^^^^^ > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 141, in __init__ > self.register_buffer_to_engine() > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 167, in register_buffer_to_engine > self.engine.register(kv_data_ptr, kv_data_len) > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/transfer_engine.py", line 36, in register > raise RuntimeError("Mooncake memory registration failed.") > RuntimeError: Mooncake memory registration failed. > ``` > > check_env: > > ``` > Python: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb 6 2025, 18:56:27) [GCC 11.2.0] > CUDA available: True > GPU 0,1,2,3,4,5,6,7: NVIDIA H20 > GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 > CUDA_HOME: /usr/local/cuda > NVCC: Cuda compilation tools, release 12.8, V12.8.93 > CUDA Driver Version: 550.144.03 > PyTorch: 2.6.0+cu124 > sglang: 0.4.6.post3 > sgl_kernel: 0.1.2.post1 > flashinfer_python: 0.2.5 > triton: 3.2.0 > transformers: 4.51.1 > torchao: 0.11.0 > numpy: 2.2.5 > aiohttp: 3.11.18 > fastapi: 0.115.12 > hf_transfer: 0.1.9 > huggingface_hub: 0.31.1 > interegular: 0.3.3 > modelscope: 1.25.0 > orjson: 3.10.18 > outlines: 0.1.11 > packaging: 24.1 > psutil: 7.0.0 > pydantic: 2.11.4 > python-multipart: 0.0.20 > pyzmq: 26.4.0 > uvicorn: 0.34.2 > uvloop: 0.21.0 > vllm: 0.8.5 > xgrammar: 0.1.19 > openai: 1.75.0 > tiktoken: 0.9.0 > anthropic: 0.51.0 > litellm: 1.69.1 > decord: 0.6.0 > NVIDIA Topology: > GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 CPU Affinity NUMA Affinity GPU NUMA ID > GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS PIX SYS NODE 0-105 0 N/A > GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS PIX SYS NODE 0-105 0 N/A > GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS SYS NODE SYS PIX 0-105 0 N/A > GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS SYS NODE SYS PIX 0-105 0 N/A > GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 NODE NODE NODE SYS PIX SYS 106-211 1 N/A > GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 NODE NODE NODE SYS PIX SYS 106-211 1 N/A > GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 PIX NODE NODE SYS NODE SYS 106-211 1 N/A > GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X PIX NODE NODE SYS NODE SYS 106-211 1 N/A > NIC0 SYS SYS SYS SYS NODE NODE PIX PIX X NODE NODE SYS NODE SYS > NIC1 SYS SYS SYS SYS NODE NODE NODE NODE NODE X PIX SYS NODE SYS > NIC2 SYS SYS SYS SYS NODE NODE NODE NODE NODE PIX X SYS NODE SYS > NIC3 PIX PIX NODE NODE SYS SYS SYS SYS SYS SYS SYS X SYS NODE > NIC4 SYS SYS SYS SYS PIX PIX NODE NODE NODE NODE NODE SYS X SYS > NIC5 NODE NODE PIX PIX SYS SYS SYS SYS SYS SYS SYS NODE SYS X > > Legend: > > X = Self > SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) > NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node > PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) > PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) > PIX = Connection traversing at most a single PCIe bridge > NV# = Connection traversing a bonded set of # NVLinks > > NIC Legend: > > NIC0: mlx5_0 > NIC1: mlx5_1 > NIC2: mlx5_2 > NIC3: mlx5_3 > NIC4: mlx5_4"temp" > NIC5: mlx5_5"temp" > > > Hypervisor vendor: KVM > ulimit soft: 1048576 > ``` Could you use tools to test the availability of GDR (e..g., gdrcopy)?
bsakaguchi commented 2026-10-29 21:09:50 +00:00
Author
Owner
Copy link

@YosanHo commented on GitHub (May 14, 2025):

same error for me

I0513 13:21:20.363459 72079 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.363724 72079 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15436 for serving local TCP service
I0513 13:21:20.374280 72079 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.375334 72079 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.395785 72080 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.395951 72080 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15986 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.403113 72076 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.403254 72076 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15505 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.405055 72078 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.405176 72078 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16252 for serving local TCP service
I0513 13:21:20.406009 72080 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.406737 72080 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.413020 72076 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.413455 72078 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.413678 72076 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.414115 72078 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.418197 72082 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.418345 72082 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16068 for serving local TCP service
I0513 13:21:20.428444 72082 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.429111 72082 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.430963 72075 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.431088 72075 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16536 for serving local TCP service
I0513 13:21:20.440829 72075 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.441489 72075 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.462790 72081 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.462942 72081 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15336 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.469861 72077 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.470021 72077 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16851 for serving local TCP service
I0513 13:21:20.474447 72081 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.475225 72081 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.480049 72077 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.480682 72077 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
E0513 13:21:20.956470 72079 rdma_context.cpp:198] Failed to register memory 0x7f555a000000: Bad address [14]
[2025-05-13 13:21:20 TP4] Mooncake memory registration failed.
[2025-05-13 13:21:20 TP4] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2145, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 435, in __init__
    self.init_disaggregation()
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 605, in init_disaggregation
    self.disagg_prefill_bootstrap_queue = PrefillBootstrapQueue(
                                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 80, in __init__
    self.kv_manager = self._init_kv_manager()
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 114, in _init_kv_manager
    kv_manager = kv_manager_class(
                 ^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 141, in __init__
    self.register_buffer_to_engine()
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 167, in register_buffer_to_engine
    self.engine.register(kv_data_ptr, kv_data_len)
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/transfer_engine.py", line 36, in register
    raise RuntimeError("Mooncake memory registration failed.")
RuntimeError: Mooncake memory registration failed.

check_env:

Python: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H20
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.8, V12.8.93
CUDA Driver Version: 550.144.03
PyTorch: 2.6.0+cu124
sglang: 0.4.6.post3
sgl_kernel: 0.1.2.post1
flashinfer_python: 0.2.5
triton: 3.2.0
transformers: 4.51.1
torchao: 0.11.0
numpy: 2.2.5
aiohttp: 3.11.18
fastapi: 0.115.12
hf_transfer: 0.1.9
huggingface_hub: 0.31.1
interegular: 0.3.3
modelscope: 1.25.0
orjson: 3.10.18
outlines: 0.1.11
packaging: 24.1
psutil: 7.0.0
pydantic: 2.11.4
python-multipart: 0.0.20
pyzmq: 26.4.0
uvicorn: 0.34.2
uvloop: 0.21.0
vllm: 0.8.5
xgrammar: 0.1.19
openai: 1.75.0
tiktoken: 0.9.0
anthropic: 0.51.0
litellm: 1.69.1
decord: 0.6.0
NVIDIA Topology: 
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     PIX     SYS     NODE    0-105   0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     PIX     SYS     NODE    0-105   0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     NODE    SYS     PIX     0-105   0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     SYS     SYS     NODE    SYS     PIX     0-105   0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    NODE    NODE    NODE    SYS     PIX     SYS     106-211 1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    NODE    NODE    NODE    SYS     PIX     SYS     106-211 1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    PIX     NODE    NODE    SYS     NODE    SYS     106-211 1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      PIX     NODE    NODE    SYS     NODE    SYS     106-211 1               N/A
NIC0    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PIX      X      NODE    NODE    SYS     NODE    SYS
NIC1    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE     X      PIX     SYS     NODE    SYS
NIC2    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE    PIX      X      SYS     NODE    SYS
NIC3    PIX     PIX     NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     NODE
NIC4    SYS     SYS     SYS     SYS     PIX     PIX     NODE    NODE    NODE    NODE    NODE    SYS      X      SYS
NIC5    NODE    NODE    PIX     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    SYS      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4"temp"
  NIC5: mlx5_5"temp"


Hypervisor vendor: KVM
ulimit soft: 1048576

Could you use tools to test the availability of GDR (e..g., gdrcopy)?

@stmatengss It's configuration problem on host , it works well now, tks

<!-- gh-comment-id:2878654176 --> @YosanHo commented on GitHub (May 14, 2025): > > same error for me > > ``` > > I0513 13:21:20.363459 72079 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > I0513 13:21:20.363724 72079 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15436 for serving local TCP service > > I0513 13:21:20.374280 72079 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > I0513 13:21:20.375334 72079 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > I0513 13:21:20.395785 72080 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > I0513 13:21:20.395951 72080 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15986 for serving local TCP service > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > I0513 13:21:20.403113 72076 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > I0513 13:21:20.403254 72076 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15505 for serving local TCP service > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > I0513 13:21:20.405055 72078 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > I0513 13:21:20.405176 72078 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16252 for serving local TCP service > > I0513 13:21:20.406009 72080 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > I0513 13:21:20.406737 72080 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > I0513 13:21:20.413020 72076 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > I0513 13:21:20.413455 72078 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > I0513 13:21:20.413678 72076 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > I0513 13:21:20.414115 72078 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > I0513 13:21:20.418197 72082 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > I0513 13:21:20.418345 72082 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16068 for serving local TCP service > > I0513 13:21:20.428444 72082 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > I0513 13:21:20.429111 72082 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > I0513 13:21:20.430963 72075 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > I0513 13:21:20.431088 72075 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16536 for serving local TCP service > > I0513 13:21:20.440829 72075 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > I0513 13:21:20.441489 72075 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > I0513 13:21:20.462790 72081 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > I0513 13:21:20.462942 72081 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15336 for serving local TCP service > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > I0513 13:21:20.469861 72077 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > I0513 13:21:20.470021 72077 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16851 for serving local TCP service > > I0513 13:21:20.474447 72081 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > I0513 13:21:20.475225 72081 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > I0513 13:21:20.480049 72077 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > I0513 13:21:20.480682 72077 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > E0513 13:21:20.956470 72079 rdma_context.cpp:198] Failed to register memory 0x7f555a000000: Bad address [14] > > [2025-05-13 13:21:20 TP4] Mooncake memory registration failed. > > [2025-05-13 13:21:20 TP4] Scheduler hit an exception: Traceback (most recent call last): > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2145, in run_scheduler_process > > scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank) > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 435, in __init__ > > self.init_disaggregation() > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 605, in init_disaggregation > > self.disagg_prefill_bootstrap_queue = PrefillBootstrapQueue( > > ^^^^^^^^^^^^^^^^^^^^^^ > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 80, in __init__ > > self.kv_manager = self._init_kv_manager() > > ^^^^^^^^^^^^^^^^^^^^^^^ > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 114, in _init_kv_manager > > kv_manager = kv_manager_class( > > ^^^^^^^^^^^^^^^^^ > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 141, in __init__ > > self.register_buffer_to_engine() > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 167, in register_buffer_to_engine > > self.engine.register(kv_data_ptr, kv_data_len) > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/transfer_engine.py", line 36, in register > > raise RuntimeError("Mooncake memory registration failed.") > > RuntimeError: Mooncake memory registration failed. > > ``` > > > > > > > > > > > > > > > > > > > > > > > > check_env: > > ``` > > Python: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb 6 2025, 18:56:27) [GCC 11.2.0] > > CUDA available: True > > GPU 0,1,2,3,4,5,6,7: NVIDIA H20 > > GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 > > CUDA_HOME: /usr/local/cuda > > NVCC: Cuda compilation tools, release 12.8, V12.8.93 > > CUDA Driver Version: 550.144.03 > > PyTorch: 2.6.0+cu124 > > sglang: 0.4.6.post3 > > sgl_kernel: 0.1.2.post1 > > flashinfer_python: 0.2.5 > > triton: 3.2.0 > > transformers: 4.51.1 > > torchao: 0.11.0 > > numpy: 2.2.5 > > aiohttp: 3.11.18 > > fastapi: 0.115.12 > > hf_transfer: 0.1.9 > > huggingface_hub: 0.31.1 > > interegular: 0.3.3 > > modelscope: 1.25.0 > > orjson: 3.10.18 > > outlines: 0.1.11 > > packaging: 24.1 > > psutil: 7.0.0 > > pydantic: 2.11.4 > > python-multipart: 0.0.20 > > pyzmq: 26.4.0 > > uvicorn: 0.34.2 > > uvloop: 0.21.0 > > vllm: 0.8.5 > > xgrammar: 0.1.19 > > openai: 1.75.0 > > tiktoken: 0.9.0 > > anthropic: 0.51.0 > > litellm: 1.69.1 > > decord: 0.6.0 > > NVIDIA Topology: > > GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 CPU Affinity NUMA Affinity GPU NUMA ID > > GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS PIX SYS NODE 0-105 0 N/A > > GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS PIX SYS NODE 0-105 0 N/A > > GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS SYS NODE SYS PIX 0-105 0 N/A > > GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS SYS NODE SYS PIX 0-105 0 N/A > > GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 NODE NODE NODE SYS PIX SYS 106-211 1 N/A > > GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 NODE NODE NODE SYS PIX SYS 106-211 1 N/A > > GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 PIX NODE NODE SYS NODE SYS 106-211 1 N/A > > GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X PIX NODE NODE SYS NODE SYS 106-211 1 N/A > > NIC0 SYS SYS SYS SYS NODE NODE PIX PIX X NODE NODE SYS NODE SYS > > NIC1 SYS SYS SYS SYS NODE NODE NODE NODE NODE X PIX SYS NODE SYS > > NIC2 SYS SYS SYS SYS NODE NODE NODE NODE NODE PIX X SYS NODE SYS > > NIC3 PIX PIX NODE NODE SYS SYS SYS SYS SYS SYS SYS X SYS NODE > > NIC4 SYS SYS SYS SYS PIX PIX NODE NODE NODE NODE NODE SYS X SYS > > NIC5 NODE NODE PIX PIX SYS SYS SYS SYS SYS SYS SYS NODE SYS X > > > > Legend: > > > > X = Self > > SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) > > NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node > > PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) > > PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) > > PIX = Connection traversing at most a single PCIe bridge > > NV# = Connection traversing a bonded set of # NVLinks > > > > NIC Legend: > > > > NIC0: mlx5_0 > > NIC1: mlx5_1 > > NIC2: mlx5_2 > > NIC3: mlx5_3 > > NIC4: mlx5_4"temp" > > NIC5: mlx5_5"temp" > > > > > > Hypervisor vendor: KVM > > ulimit soft: 1048576 > > ``` > > Could you use tools to test the availability of GDR (e..g., gdrcopy)? @stmatengss It's configuration problem on host , it works well now, tks
bsakaguchi commented 2026-10-29 21:09:50 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (May 14, 2025):

same error for me

I0513 13:21:20.363459 72079 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.363724 72079 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15436 for serving local TCP service
I0513 13:21:20.374280 72079 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.375334 72079 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.395785 72080 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.395951 72080 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15986 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.403113 72076 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.403254 72076 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15505 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.405055 72078 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.405176 72078 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16252 for serving local TCP service
I0513 13:21:20.406009 72080 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.406737 72080 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.413020 72076 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.413455 72078 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.413678 72076 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.414115 72078 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.418197 72082 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.418345 72082 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16068 for serving local TCP service
I0513 13:21:20.428444 72082 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.429111 72082 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.430963 72075 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.431088 72075 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16536 for serving local TCP service
I0513 13:21:20.440829 72075 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.441489 72075 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.462790 72081 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.462942 72081 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15336 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.469861 72077 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.470021 72077 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16851 for serving local TCP service
I0513 13:21:20.474447 72081 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.475225 72081 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.480049 72077 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.480682 72077 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
E0513 13:21:20.956470 72079 rdma_context.cpp:198] Failed to register memory 0x7f555a000000: Bad address [14]
[2025-05-13 13:21:20 TP4] Mooncake memory registration failed.
[2025-05-13 13:21:20 TP4] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2145, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 435, in __init__
    self.init_disaggregation()
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 605, in init_disaggregation
    self.disagg_prefill_bootstrap_queue = PrefillBootstrapQueue(
                                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 80, in __init__
    self.kv_manager = self._init_kv_manager()
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 114, in _init_kv_manager
    kv_manager = kv_manager_class(
                 ^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 141, in __init__
    self.register_buffer_to_engine()
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 167, in register_buffer_to_engine
    self.engine.register(kv_data_ptr, kv_data_len)
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/transfer_engine.py", line 36, in register
    raise RuntimeError("Mooncake memory registration failed.")
RuntimeError: Mooncake memory registration failed.

check_env:

Python: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H20
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.8, V12.8.93
CUDA Driver Version: 550.144.03
PyTorch: 2.6.0+cu124
sglang: 0.4.6.post3
sgl_kernel: 0.1.2.post1
flashinfer_python: 0.2.5
triton: 3.2.0
transformers: 4.51.1
torchao: 0.11.0
numpy: 2.2.5
aiohttp: 3.11.18
fastapi: 0.115.12
hf_transfer: 0.1.9
huggingface_hub: 0.31.1
interegular: 0.3.3
modelscope: 1.25.0
orjson: 3.10.18
outlines: 0.1.11
packaging: 24.1
psutil: 7.0.0
pydantic: 2.11.4
python-multipart: 0.0.20
pyzmq: 26.4.0
uvicorn: 0.34.2
uvloop: 0.21.0
vllm: 0.8.5
xgrammar: 0.1.19
openai: 1.75.0
tiktoken: 0.9.0
anthropic: 0.51.0
litellm: 1.69.1
decord: 0.6.0
NVIDIA Topology: 
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     PIX     SYS     NODE    0-105   0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     PIX     SYS     NODE    0-105   0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     NODE    SYS     PIX     0-105   0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     SYS     SYS     NODE    SYS     PIX     0-105   0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    NODE    NODE    NODE    SYS     PIX     SYS     106-211 1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    NODE    NODE    NODE    SYS     PIX     SYS     106-211 1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    PIX     NODE    NODE    SYS     NODE    SYS     106-211 1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      PIX     NODE    NODE    SYS     NODE    SYS     106-211 1               N/A
NIC0    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PIX      X      NODE    NODE    SYS     NODE    SYS
NIC1    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE     X      PIX     SYS     NODE    SYS
NIC2    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE    PIX      X      SYS     NODE    SYS
NIC3    PIX     PIX     NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     NODE
NIC4    SYS     SYS     SYS     SYS     PIX     PIX     NODE    NODE    NODE    NODE    NODE    SYS      X      SYS
NIC5    NODE    NODE    PIX     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    SYS      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4"temp"
  NIC5: mlx5_5"temp"


Hypervisor vendor: KVM
ulimit soft: 1048576

Could you use tools to test the availability of GDR (e..g., gdrcopy)?

@stmatengss It's configuration problem on host , it works well now, tks

Hello, could you please specify which configuration it is? @YosanHo

<!-- gh-comment-id:2878735686 --> @CSEEduanyu commented on GitHub (May 14, 2025): > > > same error for me > > > ``` > > > I0513 13:21:20.363459 72079 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.363724 72079 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15436 for serving local TCP service > > > I0513 13:21:20.374280 72079 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.375334 72079 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.395785 72080 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.395951 72080 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15986 for serving local TCP service > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.403113 72076 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.403254 72076 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15505 for serving local TCP service > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.405055 72078 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.405176 72078 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16252 for serving local TCP service > > > I0513 13:21:20.406009 72080 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.406737 72080 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > I0513 13:21:20.413020 72076 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.413455 72078 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.413678 72076 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > I0513 13:21:20.414115 72078 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.418197 72082 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.418345 72082 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16068 for serving local TCP service > > > I0513 13:21:20.428444 72082 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.429111 72082 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.430963 72075 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.431088 72075 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16536 for serving local TCP service > > > I0513 13:21:20.440829 72075 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.441489 72075 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.462790 72081 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.462942 72081 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15336 for serving local TCP service > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.469861 72077 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.470021 72077 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16851 for serving local TCP service > > > I0513 13:21:20.474447 72081 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.475225 72081 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > I0513 13:21:20.480049 72077 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.480682 72077 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > E0513 13:21:20.956470 72079 rdma_context.cpp:198] Failed to register memory 0x7f555a000000: Bad address [14] > > > [2025-05-13 13:21:20 TP4] Mooncake memory registration failed. > > > [2025-05-13 13:21:20 TP4] Scheduler hit an exception: Traceback (most recent call last): > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2145, in run_scheduler_process > > > scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank) > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 435, in __init__ > > > self.init_disaggregation() > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 605, in init_disaggregation > > > self.disagg_prefill_bootstrap_queue = PrefillBootstrapQueue( > > > ^^^^^^^^^^^^^^^^^^^^^^ > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 80, in __init__ > > > self.kv_manager = self._init_kv_manager() > > > ^^^^^^^^^^^^^^^^^^^^^^^ > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 114, in _init_kv_manager > > > kv_manager = kv_manager_class( > > > ^^^^^^^^^^^^^^^^^ > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 141, in __init__ > > > self.register_buffer_to_engine() > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 167, in register_buffer_to_engine > > > self.engine.register(kv_data_ptr, kv_data_len) > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/transfer_engine.py", line 36, in register > > > raise RuntimeError("Mooncake memory registration failed.") > > > RuntimeError: Mooncake memory registration failed. > > > ``` > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > check_env: > > > ``` > > > Python: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb 6 2025, 18:56:27) [GCC 11.2.0] > > > CUDA available: True > > > GPU 0,1,2,3,4,5,6,7: NVIDIA H20 > > > GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 > > > CUDA_HOME: /usr/local/cuda > > > NVCC: Cuda compilation tools, release 12.8, V12.8.93 > > > CUDA Driver Version: 550.144.03 > > > PyTorch: 2.6.0+cu124 > > > sglang: 0.4.6.post3 > > > sgl_kernel: 0.1.2.post1 > > > flashinfer_python: 0.2.5 > > > triton: 3.2.0 > > > transformers: 4.51.1 > > > torchao: 0.11.0 > > > numpy: 2.2.5 > > > aiohttp: 3.11.18 > > > fastapi: 0.115.12 > > > hf_transfer: 0.1.9 > > > huggingface_hub: 0.31.1 > > > interegular: 0.3.3 > > > modelscope: 1.25.0 > > > orjson: 3.10.18 > > > outlines: 0.1.11 > > > packaging: 24.1 > > > psutil: 7.0.0 > > > pydantic: 2.11.4 > > > python-multipart: 0.0.20 > > > pyzmq: 26.4.0 > > > uvicorn: 0.34.2 > > > uvloop: 0.21.0 > > > vllm: 0.8.5 > > > xgrammar: 0.1.19 > > > openai: 1.75.0 > > > tiktoken: 0.9.0 > > > anthropic: 0.51.0 > > > litellm: 1.69.1 > > > decord: 0.6.0 > > > NVIDIA Topology: > > > GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 CPU Affinity NUMA Affinity GPU NUMA ID > > > GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS PIX SYS NODE 0-105 0 N/A > > > GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS PIX SYS NODE 0-105 0 N/A > > > GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS SYS NODE SYS PIX 0-105 0 N/A > > > GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS SYS NODE SYS PIX 0-105 0 N/A > > > GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 NODE NODE NODE SYS PIX SYS 106-211 1 N/A > > > GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 NODE NODE NODE SYS PIX SYS 106-211 1 N/A > > > GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 PIX NODE NODE SYS NODE SYS 106-211 1 N/A > > > GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X PIX NODE NODE SYS NODE SYS 106-211 1 N/A > > > NIC0 SYS SYS SYS SYS NODE NODE PIX PIX X NODE NODE SYS NODE SYS > > > NIC1 SYS SYS SYS SYS NODE NODE NODE NODE NODE X PIX SYS NODE SYS > > > NIC2 SYS SYS SYS SYS NODE NODE NODE NODE NODE PIX X SYS NODE SYS > > > NIC3 PIX PIX NODE NODE SYS SYS SYS SYS SYS SYS SYS X SYS NODE > > > NIC4 SYS SYS SYS SYS PIX PIX NODE NODE NODE NODE NODE SYS X SYS > > > NIC5 NODE NODE PIX PIX SYS SYS SYS SYS SYS SYS SYS NODE SYS X > > > > > > Legend: > > > > > > X = Self > > > SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) > > > NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node > > > PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) > > > PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) > > > PIX = Connection traversing at most a single PCIe bridge > > > NV# = Connection traversing a bonded set of # NVLinks > > > > > > NIC Legend: > > > > > > NIC0: mlx5_0 > > > NIC1: mlx5_1 > > > NIC2: mlx5_2 > > > NIC3: mlx5_3 > > > NIC4: mlx5_4"temp" > > > NIC5: mlx5_5"temp" > > > > > > > > > Hypervisor vendor: KVM > > > ulimit soft: 1048576 > > > ``` > > > > > > Could you use tools to test the availability of GDR (e..g., gdrcopy)? > > [@stmatengss](https://github.com/stmatengss) It's configuration problem on host , it works well now, tks Hello, could you please specify which configuration it is? @YosanHo
bsakaguchi commented 2026-10-29 21:09:50 +00:00
Author
Owner
Copy link

@taohui commented on GitHub (May 16, 2025):

same error for me

I0513 13:21:20.363459 72079 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.363724 72079 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15436 for serving local TCP service
I0513 13:21:20.374280 72079 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.375334 72079 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.395785 72080 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.395951 72080 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15986 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.403113 72076 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.403254 72076 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15505 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.405055 72078 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.405176 72078 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16252 for serving local TCP service
I0513 13:21:20.406009 72080 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.406737 72080 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.413020 72076 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.413455 72078 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.413678 72076 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.414115 72078 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.418197 72082 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.418345 72082 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16068 for serving local TCP service
I0513 13:21:20.428444 72082 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.429111 72082 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.430963 72075 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.431088 72075 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16536 for serving local TCP service
I0513 13:21:20.440829 72075 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.441489 72075 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.462790 72081 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.462942 72081 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15336 for serving local TCP service
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0513 13:21:20.469861 72077 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0513 13:21:20.470021 72077 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16851 for serving local TCP service
I0513 13:21:20.474447 72081 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.475225 72081 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
I0513 13:21:20.480049 72077 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/�
I0513 13:21:20.480682 72077 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92
E0513 13:21:20.956470 72079 rdma_context.cpp:198] Failed to register memory 0x7f555a000000: Bad address [14]
[2025-05-13 13:21:20 TP4] Mooncake memory registration failed.
[2025-05-13 13:21:20 TP4] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2145, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 435, in __init__
    self.init_disaggregation()
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 605, in init_disaggregation
    self.disagg_prefill_bootstrap_queue = PrefillBootstrapQueue(
                                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 80, in __init__
    self.kv_manager = self._init_kv_manager()
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 114, in _init_kv_manager
    kv_manager = kv_manager_class(
                 ^^^^^^^^^^^^^^^^^
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 141, in __init__
    self.register_buffer_to_engine()
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 167, in register_buffer_to_engine
    self.engine.register(kv_data_ptr, kv_data_len)
  File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/transfer_engine.py", line 36, in register
    raise RuntimeError("Mooncake memory registration failed.")
RuntimeError: Mooncake memory registration failed.

check_env:

Python: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H20
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.8, V12.8.93
CUDA Driver Version: 550.144.03
PyTorch: 2.6.0+cu124
sglang: 0.4.6.post3
sgl_kernel: 0.1.2.post1
flashinfer_python: 0.2.5
triton: 3.2.0
transformers: 4.51.1
torchao: 0.11.0
numpy: 2.2.5
aiohttp: 3.11.18
fastapi: 0.115.12
hf_transfer: 0.1.9
huggingface_hub: 0.31.1
interegular: 0.3.3
modelscope: 1.25.0
orjson: 3.10.18
outlines: 0.1.11
packaging: 24.1
psutil: 7.0.0
pydantic: 2.11.4
python-multipart: 0.0.20
pyzmq: 26.4.0
uvicorn: 0.34.2
uvloop: 0.21.0
vllm: 0.8.5
xgrammar: 0.1.19
openai: 1.75.0
tiktoken: 0.9.0
anthropic: 0.51.0
litellm: 1.69.1
decord: 0.6.0
NVIDIA Topology: 
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     PIX     SYS     NODE    0-105   0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     PIX     SYS     NODE    0-105   0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     NODE    SYS     PIX     0-105   0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     SYS     SYS     NODE    SYS     PIX     0-105   0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    NODE    NODE    NODE    SYS     PIX     SYS     106-211 1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    NODE    NODE    NODE    SYS     PIX     SYS     106-211 1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    PIX     NODE    NODE    SYS     NODE    SYS     106-211 1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      PIX     NODE    NODE    SYS     NODE    SYS     106-211 1               N/A
NIC0    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PIX      X      NODE    NODE    SYS     NODE    SYS
NIC1    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE     X      PIX     SYS     NODE    SYS
NIC2    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE    PIX      X      SYS     NODE    SYS
NIC3    PIX     PIX     NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     NODE
NIC4    SYS     SYS     SYS     SYS     PIX     PIX     NODE    NODE    NODE    NODE    NODE    SYS      X      SYS
NIC5    NODE    NODE    PIX     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    SYS      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4"temp"
  NIC5: mlx5_5"temp"


Hypervisor vendor: KVM
ulimit soft: 1048576

Could you use tools to test the availability of GDR (e..g., gdrcopy)?

@stmatengss It's configuration problem on host , it works well now, tks

I have the same error, how do you solve it?

<!-- gh-comment-id:2885884663 --> @taohui commented on GitHub (May 16, 2025): > > > same error for me > > > ``` > > > I0513 13:21:20.363459 72079 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.363724 72079 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15436 for serving local TCP service > > > I0513 13:21:20.374280 72079 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.375334 72079 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.395785 72080 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.395951 72080 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15986 for serving local TCP service > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.403113 72076 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.403254 72076 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15505 for serving local TCP service > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.405055 72078 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.405176 72078 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16252 for serving local TCP service > > > I0513 13:21:20.406009 72080 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.406737 72080 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > I0513 13:21:20.413020 72076 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.413455 72078 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.413678 72076 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > I0513 13:21:20.414115 72078 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.418197 72082 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.418345 72082 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16068 for serving local TCP service > > > I0513 13:21:20.428444 72082 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.429111 72082 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.430963 72075 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.431088 72075 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16536 for serving local TCP service > > > I0513 13:21:20.440829 72075 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.441489 72075 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.462790 72081 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.462942 72081 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 15336 for serving local TCP service > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > I0513 13:21:20.469861 72077 transfer_engine.cpp:338] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) > > > I0513 13:21:20.470021 72077 transfer_engine.cpp:91] Transfer Engine uses address 10.148.0.107 and port 16851 for serving local TCP service > > > I0513 13:21:20.474447 72081 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.475225 72081 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > I0513 13:21:20.480049 72077 rdma_context.cpp:411] Find best gid index: 0 on mlx5_3/� > > > I0513 13:21:20.480682 72077 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 31, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:2a:13:92 > > > E0513 13:21:20.956470 72079 rdma_context.cpp:198] Failed to register memory 0x7f555a000000: Bad address [14] > > > [2025-05-13 13:21:20 TP4] Mooncake memory registration failed. > > > [2025-05-13 13:21:20 TP4] Scheduler hit an exception: Traceback (most recent call last): > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2145, in run_scheduler_process > > > scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank) > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 435, in __init__ > > > self.init_disaggregation() > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 605, in init_disaggregation > > > self.disagg_prefill_bootstrap_queue = PrefillBootstrapQueue( > > > ^^^^^^^^^^^^^^^^^^^^^^ > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 80, in __init__ > > > self.kv_manager = self._init_kv_manager() > > > ^^^^^^^^^^^^^^^^^^^^^^^ > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/prefill.py", line 114, in _init_kv_manager > > > kv_manager = kv_manager_class( > > > ^^^^^^^^^^^^^^^^^ > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 141, in __init__ > > > self.register_buffer_to_engine() > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 167, in register_buffer_to_engine > > > self.engine.register(kv_data_ptr, kv_data_len) > > > File "/opt/app/python3.12/lib/python3.12/site-packages/sglang/srt/disaggregation/mooncake/transfer_engine.py", line 36, in register > > > raise RuntimeError("Mooncake memory registration failed.") > > > RuntimeError: Mooncake memory registration failed. > > > ``` > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > check_env: > > > ``` > > > Python: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb 6 2025, 18:56:27) [GCC 11.2.0] > > > CUDA available: True > > > GPU 0,1,2,3,4,5,6,7: NVIDIA H20 > > > GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 > > > CUDA_HOME: /usr/local/cuda > > > NVCC: Cuda compilation tools, release 12.8, V12.8.93 > > > CUDA Driver Version: 550.144.03 > > > PyTorch: 2.6.0+cu124 > > > sglang: 0.4.6.post3 > > > sgl_kernel: 0.1.2.post1 > > > flashinfer_python: 0.2.5 > > > triton: 3.2.0 > > > transformers: 4.51.1 > > > torchao: 0.11.0 > > > numpy: 2.2.5 > > > aiohttp: 3.11.18 > > > fastapi: 0.115.12 > > > hf_transfer: 0.1.9 > > > huggingface_hub: 0.31.1 > > > interegular: 0.3.3 > > > modelscope: 1.25.0 > > > orjson: 3.10.18 > > > outlines: 0.1.11 > > > packaging: 24.1 > > > psutil: 7.0.0 > > > pydantic: 2.11.4 > > > python-multipart: 0.0.20 > > > pyzmq: 26.4.0 > > > uvicorn: 0.34.2 > > > uvloop: 0.21.0 > > > vllm: 0.8.5 > > > xgrammar: 0.1.19 > > > openai: 1.75.0 > > > tiktoken: 0.9.0 > > > anthropic: 0.51.0 > > > litellm: 1.69.1 > > > decord: 0.6.0 > > > NVIDIA Topology: > > > GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 CPU Affinity NUMA Affinity GPU NUMA ID > > > GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS PIX SYS NODE 0-105 0 N/A > > > GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS PIX SYS NODE 0-105 0 N/A > > > GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS SYS NODE SYS PIX 0-105 0 N/A > > > GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS SYS NODE SYS PIX 0-105 0 N/A > > > GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 NODE NODE NODE SYS PIX SYS 106-211 1 N/A > > > GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 NODE NODE NODE SYS PIX SYS 106-211 1 N/A > > > GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 PIX NODE NODE SYS NODE SYS 106-211 1 N/A > > > GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X PIX NODE NODE SYS NODE SYS 106-211 1 N/A > > > NIC0 SYS SYS SYS SYS NODE NODE PIX PIX X NODE NODE SYS NODE SYS > > > NIC1 SYS SYS SYS SYS NODE NODE NODE NODE NODE X PIX SYS NODE SYS > > > NIC2 SYS SYS SYS SYS NODE NODE NODE NODE NODE PIX X SYS NODE SYS > > > NIC3 PIX PIX NODE NODE SYS SYS SYS SYS SYS SYS SYS X SYS NODE > > > NIC4 SYS SYS SYS SYS PIX PIX NODE NODE NODE NODE NODE SYS X SYS > > > NIC5 NODE NODE PIX PIX SYS SYS SYS SYS SYS SYS SYS NODE SYS X > > > > > > Legend: > > > > > > X = Self > > > SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) > > > NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node > > > PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) > > > PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) > > > PIX = Connection traversing at most a single PCIe bridge > > > NV# = Connection traversing a bonded set of # NVLinks > > > > > > NIC Legend: > > > > > > NIC0: mlx5_0 > > > NIC1: mlx5_1 > > > NIC2: mlx5_2 > > > NIC3: mlx5_3 > > > NIC4: mlx5_4"temp" > > > NIC5: mlx5_5"temp" > > > > > > > > > Hypervisor vendor: KVM > > > ulimit soft: 1048576 > > > ``` > > > > > > Could you use tools to test the availability of GDR (e..g., gdrcopy)? > > [@stmatengss](https://github.com/stmatengss) It's configuration problem on host , it works well now, tks I have the same error, how do you solve it?
bsakaguchi commented 2026-10-29 21:09:50 +00:00
Author
Owner
Copy link

@YosanHo commented on GitHub (May 16, 2025):

@CSEEduanyu @taohui I'm not aware of the specifics, but after our operations team performed a kernel upgrade on the host, there appeared to be some issues related to the Infiniband configuration that were resolved upon rolling back the changes.

<!-- gh-comment-id:2885946823 --> @YosanHo commented on GitHub (May 16, 2025): @CSEEduanyu @taohui I'm not aware of the specifics, but after our operations team performed a kernel upgrade on the host, there appeared to be some issues related to the Infiniband configuration that were resolved upon rolling back the changes.
bsakaguchi commented 2026-10-29 21:09:50 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (May 18, 2025):

After investigation, I found that the root cause is that our environment is configured with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which generates a virtual address for RDMA, but RDMA requires a physical address @ShangmingCai

<!-- gh-comment-id:2888725354 --> @CSEEduanyu commented on GitHub (May 18, 2025): After investigation, I found that the root cause is that our environment is configured with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which generates a virtual address for RDMA, but RDMA requires a physical address @ShangmingCai
bsakaguchi commented 2026-10-29 21:09:51 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (May 18, 2025):

@CSEEduanyu Oh, that's new to me. Thank you for sharing this info!

<!-- gh-comment-id:2888729558 --> @ShangmingCai commented on GitHub (May 18, 2025): @CSEEduanyu Oh, that's new to me. Thank you for sharing this info!
bsakaguchi commented 2026-10-29 21:09:51 +00:00
Author
Owner
Copy link

@WingEdge777 commented on GitHub (May 19, 2025):

After investigation, I found that the root cause is that our environment is configured with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which generates a virtual address for RDMA, but RDMA requires a physical address @ShangmingCai

@CSEEduanyu hi, do you mean that you resolved the issue by set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False ? I still got same error with this config.

<!-- gh-comment-id:2890929592 --> @WingEdge777 commented on GitHub (May 19, 2025): > After investigation, I found that the root cause is that our environment is configured with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which generates a virtual address for RDMA, but RDMA requires a physical address [@ShangmingCai](https://github.com/ShangmingCai) @CSEEduanyu hi, do you mean that you resolved the issue by set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False ? I still got same error with this config.
bsakaguchi commented 2026-10-29 21:09:51 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (May 19, 2025):

After investigation, I found that the root cause is that our environment is configured with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which generates a virtual address for RDMA, but RDMA requires a physical address @ShangmingCai

@CSEEduanyu hi, do you mean that you resolved the issue by set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False ? I still got same error with this config.

Yes, I directly removed this environment variable rather than setting it to False

<!-- gh-comment-id:2890977106 --> @CSEEduanyu commented on GitHub (May 19, 2025): > > After investigation, I found that the root cause is that our environment is configured with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which generates a virtual address for RDMA, but RDMA requires a physical address [@ShangmingCai](https://github.com/ShangmingCai) > > [@CSEEduanyu](https://github.com/CSEEduanyu) hi, do you mean that you resolved the issue by set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False ? I still got same error with this config. Yes, I directly removed this environment variable rather than setting it to False
bsakaguchi commented 2026-10-29 21:09:51 +00:00
Author
Owner
Copy link

@qiang-w-ruijie commented on GitHub (May 26, 2025):

After investigation, I found that the root cause is that our environment is configured with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which generates a virtual address for RDMA, but RDMA requires a physical address @ShangmingCai

@CSEEduanyu hi, do you mean that you resolved the issue by set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False ? I still got same error with this config.

Yes, I directly removed this environment variable rather than setting it to False

@CSEEduanyu I use PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, But it is still have the same error.
#https://github.com/sgl-project/sglang/issues/4655#issuecomment-2908370753

<!-- gh-comment-id:2908383914 --> @qiang-w-ruijie commented on GitHub (May 26, 2025): > > > After investigation, I found that the root cause is that our environment is configured with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which generates a virtual address for RDMA, but RDMA requires a physical address [@ShangmingCai](https://github.com/ShangmingCai) > > > > > > [@CSEEduanyu](https://github.com/CSEEduanyu) hi, do you mean that you resolved the issue by set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False ? I still got same error with this config. > > Yes, I directly removed this environment variable rather than setting it to False @CSEEduanyu I use PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, But it is still have the same error. #https://github.com/sgl-project/sglang/issues/4655#issuecomment-2908370753
bsakaguchi commented 2026-10-29 21:09:51 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (May 26, 2025):

After investigation, I found that the root cause is that our environment is configured with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which generates a virtual address for RDMA, but RDMA requires a physical address @ShangmingCai

@CSEEduanyu hi, do you mean that you resolved the issue by set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False ? I still got same error with this config.

Yes, I directly removed this environment variable rather than setting it to False

@CSEEduanyu I use PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, But it is still have the same error. #sgl-project/sglang#4655 (comment)

Do not set the environment variable PYTORCH_CUDA_ALLOC_CONF

<!-- gh-comment-id:2909880028 --> @CSEEduanyu commented on GitHub (May 26, 2025): > > > > After investigation, I found that the root cause is that our environment is configured with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which generates a virtual address for RDMA, but RDMA requires a physical address [@ShangmingCai](https://github.com/ShangmingCai) > > > > > > > > > [@CSEEduanyu](https://github.com/CSEEduanyu) hi, do you mean that you resolved the issue by set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False ? I still got same error with this config. > > > > > > Yes, I directly removed this environment variable rather than setting it to False > > [@CSEEduanyu](https://github.com/CSEEduanyu) I use PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, But it is still have the same error. #[sgl-project/sglang#4655 (comment)](https://github.com/sgl-project/sglang/issues/4655#issuecomment-2908370753) Do not set the environment variable PYTORCH_CUDA_ALLOC_CONF
bsakaguchi commented 2026-10-29 21:09:51 +00:00
Author
Owner
Copy link

@WingEdge777 commented on GitHub (May 28, 2025):

To be honest, generally nobody set PYTORCH_CUDA_ALLOC_CONF env before run sgl. This error probably involve some host linux kernel mode config. My SRE installed kernel with no source/header and nvidia-driver with --no-kernel-module, that should be a problem to use gdrcopy. One might try restall linux kernel with source and reinstall nvidia driver. If you are capable to run gdrcopy, then your env config should be ready for running sgl mooncake within docker container.

<!-- gh-comment-id:2914753242 --> @WingEdge777 commented on GitHub (May 28, 2025): To be honest, generally nobody set PYTORCH_CUDA_ALLOC_CONF env before run sgl. This error probably involve some host linux kernel mode config. My SRE installed kernel with no source/header and nvidia-driver with --no-kernel-module, that should be a problem to use gdrcopy. One might try restall linux kernel with source and reinstall nvidia driver. If you are capable to run gdrcopy, then your env config should be ready for running sgl mooncake within docker container.
bsakaguchi commented 2026-10-29 21:09:51 +00:00
Author
Owner
Copy link

@Prayer3th commented on GitHub (May 28, 2025):

Maybe you should check if nvidia_peermem exists in lsmod and if not then load it with modprobe nvidia_peermem.

The ibv_reg_mr verb cannot obtain the correct GPU memory address mapping when the nvidia_peermem module is not loaded. The RDMA registration process attempts to access GPU memory, but cannot correctly map the GPU memory into an accessible address space without the appropriate kernel module support.

<!-- gh-comment-id:2915187866 --> @Prayer3th commented on GitHub (May 28, 2025): Maybe you should check if **nvidia_peermem** exists in lsmod and if not then load it with **modprobe nvidia_peermem**. The ibv_reg_mr verb cannot obtain the correct GPU memory address mapping when the nvidia_peermem module is not loaded. The RDMA registration process attempts to access GPU memory, but cannot correctly map the GPU memory into an accessible address space without the appropriate kernel module support.
bsakaguchi commented 2026-10-29 21:09:51 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (May 29, 2025):

To be honest, generally nobody set PYTORCH_CUDA_ALLOC_CONF env before run sgl. This error probably involve some host linux kernel mode config. My SRE installed kernel with no source/header and nvidia-driver with --no-kernel-module, that should be a problem to use gdrcopy. One might try restall linux kernel with source and reinstall nvidia driver. If you are capable to run gdrcopy, then your env config should be ready for running sgl mooncake within docker container.

The root cause of this error is that the address passed to ibv_reg_mr() cannot be resolved, and expandable_segments is just one possible reason for this. By the way, expandable_segments is very important because it can reduce GPU memory fragmentation. This modification(https://github.com/pytorch/pytorch/pull/153261) in PyTorch makes it compatible with RDMA, but I haven't tried it yet.

<!-- gh-comment-id:2918698559 --> @CSEEduanyu commented on GitHub (May 29, 2025): > To be honest, generally nobody set PYTORCH_CUDA_ALLOC_CONF env before run sgl. This error probably involve some host linux kernel mode config. My SRE installed kernel with no source/header and nvidia-driver with --no-kernel-module, that should be a problem to use gdrcopy. One might try restall linux kernel with source and reinstall nvidia driver. If you are capable to run gdrcopy, then your env config should be ready for running sgl mooncake within docker container. The root cause of this error is that the address passed to ibv_reg_mr() cannot be resolved, and expandable_segments is just one possible reason for this. By the way, expandable_segments is very important because it can reduce GPU memory fragmentation. This modification(https://github.com/pytorch/pytorch/pull/153261) in PyTorch makes it compatible with RDMA, but I haven't tried it yet.
bsakaguchi commented 2026-10-29 21:09:52 +00:00
Author
Owner
Copy link

@Lanezzz commented on GitHub (Jun 24, 2025):

Maybe you should check if nvidia_peermem exists in lsmod and if not then load it with modprobe nvidia_peermem.

The ibv_reg_mr verb cannot obtain the correct GPU memory address mapping when the nvidia_peermem module is not loaded. The RDMA registration process attempts to access GPU memory, but cannot correctly map the GPU memory into an accessible address space without the appropriate kernel module support.

I met the same error and i have resolved it by loading peermem module.

<!-- gh-comment-id:3000054493 --> @Lanezzz commented on GitHub (Jun 24, 2025): > Maybe you should check if **nvidia_peermem** exists in lsmod and if not then load it with **modprobe nvidia_peermem**. > > The ibv_reg_mr verb cannot obtain the correct GPU memory address mapping when the nvidia_peermem module is not loaded. The RDMA registration process attempts to access GPU memory, but cannot correctly map the GPU memory into an accessible address space without the appropriate kernel module support. I met the same error and i have resolved it by loading peermem module.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#128
No description provided.