[GH-ISSUE #257] sglang使用mooncake报错

bsakaguchi commented

2026-10-29 21:09:20 +00:00

Owner

Originally created by @phoenix-zhu-16206 on GitHub (Apr 16, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/257

sglang使用main分支最新代码
mooncake使用main分支最新代码

1 启动etcd
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379

2 mooncake.json
{
"local_hostname": "172.16.0.64",
"metadata_server": "etcd://172.16.0.64:2379",
"protocol": "tcp",
"device_name": ""
}

3 启动prefill
MOONCAKE_CONFIG_PATH=./mooncake.json python -m sglang.launch_server --model-path /data/models/models/deepseek-ai/DeepSeek-R1-Distill-Llama-8B --disaggregation-mode prefill --port 30000 --host 172.16.0.64 --tp-size 2

4 启动decode
MOONCAKE_CONFIG_PATH=./mooncake.json python -m sglang.launch_server --model-path /data/models/models/deepseek-ai/DeepSeek-R1-Distill-Llama-8B --disaggregation-mode decode --port 30001 --base-gpu-id 2 --host 172.16.0.64 --tp-size 2

5 启动proxy
python3 -m sglang.srt.disaggregation.mini_lb --prefill http://172.16.0.64:30000 --decode http://172.16.0.64:30001 --host 0.0.0.0 --port 8000

6 测试
curl -X POST http://172.16.0.64:8000/generate -H "Content-Type: application/json" -d '{
"text": "Let me tell you a lonnng story ",
"sampling_params": {
"temperature": 0
}
}'

Originally created by @phoenix-zhu-16206 on GitHub (Apr 16, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/257 sglang使用main分支最新代码 mooncake使用main分支最新代码 1 启动etcd etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379 2 mooncake.json { "local_hostname": "172.16.0.64", "metadata_server": "etcd://172.16.0.64:2379", "protocol": "tcp", "device_name": "" } 3 启动prefill MOONCAKE_CONFIG_PATH=./mooncake.json python -m sglang.launch_server --model-path /data/models/models/deepseek-ai/DeepSeek-R1-Distill-Llama-8B --disaggregation-mode prefill --port 30000 --host 172.16.0.64 --tp-size 2 4 启动decode MOONCAKE_CONFIG_PATH=./mooncake.json python -m sglang.launch_server --model-path /data/models/models/deepseek-ai/DeepSeek-R1-Distill-Llama-8B --disaggregation-mode decode --port 30001 --base-gpu-id 2 --host 172.16.0.64 --tp-size 2 5 启动proxy python3 -m sglang.srt.disaggregation.mini_lb --prefill http://172.16.0.64:30000 --decode http://172.16.0.64:30001 --host 0.0.0.0 --port 8000 6 测试 curl -X POST http://172.16.0.64:8000/generate -H "Content-Type: application/json" -d '{ "text": "Let me tell you a lonnng story ", "sampling_params": { "temperature": 0 } }' <img width="1136" alt="Image" src="https://github.com/user-attachments/assets/a752d2a4-3702-4f41-95b2-05b73ebec97c" />

bsakaguchi

2026-10-29 21:09:20 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:09:21 +00:00

Author

Owner

@phoenix-zhu-16206 commented on GitHub (Apr 16, 2025):

加了行日志打印了下错误码，输出如下 E0416 20:07:04.836551 759348 tcp_transport.cpp:128] write body finish, system:14 E0416 20:07:04.836566 759362 tcp_transport.cpp:128] write body finish, system:14

@phoenix-zhu-16206 commented on GitHub (Apr 16, 2025): <img width="686" alt="Image" src="https://github.com/user-attachments/assets/62f30afa-35a6-457f-944c-ca4f0a3ded1d" /> 加了行日志打印了下错误码，输出如下 E0416 20:07:04.836551 759348 tcp_transport.cpp:128] write body finish, system:14 E0416 20:07:04.836566 759362 tcp_transport.cpp:128] write body finish, system:14

bsakaguchi commented

2026-10-29 21:09:21 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Apr 17, 2025):

2 mooncake.json { "local_hostname": "172.16.0.64", "metadata_server": "etcd://172.16.0.64:2379", "protocol": "tcp", "device_name": "" }

If you don't have the RDMA device, then it is not runnable. Currently, the disaggregation feature of SGLang only supports GDR right now. Please note that.

TCP is not supported currently, or maybe will never be supported in the future, since kv transfer requires a high-speed network. You will not get any performance improvement if you insist on using TCP with disaggregated serving.

@ShangmingCai commented on GitHub (Apr 17, 2025): > 2 mooncake.json { "local_hostname": "172.16.0.64", "metadata_server": "etcd://172.16.0.64:2379", "protocol": "tcp", "device_name": "" } If you don't have the RDMA device, then it is not runnable. Currently, the disaggregation feature of SGLang only supports GDR right now. Please note that. TCP is not supported currently, or maybe will never be supported in the future, since kv transfer requires a high-speed network. You will not get any performance improvement if you insist on using TCP with disaggregated serving.

bsakaguchi commented

2026-10-29 21:09:21 +00:00

Author

Owner

@ictzyqq commented on GitHub (May 18, 2025):

If you don't have the RDMA device, then it is not runnable. Currently, the disaggregation feature of SGLang only supports GDR right now. Please note that.

TCP is not supported currently, or maybe will never be supported in the future, since kv transfer requires a high-speed network. You will not get any performance improvement if you insist on using TCP with disaggregated serving.

大佬您好，我有些疑惑，readme里面不是说MoonCake的transfer engine支持RDMA、TCP等多种协议吗，为什么这里您又说不支持TCP？还是说只有sglang不支持TCP？那我如果没有RDMA设备的话就无法跑PD分离了吗

@ictzyqq commented on GitHub (May 18, 2025): > If you don't have the RDMA device, then it is not runnable. Currently, the disaggregation feature of SGLang only supports GDR right now. Please note that. > > TCP is not supported currently, or maybe will never be supported in the future, since kv transfer requires a high-speed network. You will not get any performance improvement if you insist on using TCP with disaggregated serving. 大佬您好，我有些疑惑，readme里面不是说MoonCake的transfer engine支持RDMA、TCP等多种协议吗，为什么这里您又说不支持TCP？还是说只有sglang不支持TCP？那我如果没有RDMA设备的话就无法跑PD分离了吗

bsakaguchi commented

2026-10-29 21:09:21 +00:00

Author

Owner

@ShangmingCai commented on GitHub (May 18, 2025):

@ictzyqq SGLang的PD实现不支持tcp、因为需要注册gpu mem buffer，只有支持gdr才能注册成功。我们有fallback to cpu mem的实现，但即使能run性能不一定好，暂时不是高优推进的方向。会在各种高优性能优化都做完之后再回来考虑。Transfer engine和Store组件本身支持tcp。

@ShangmingCai commented on GitHub (May 18, 2025): @ictzyqq SGLang的PD实现不支持tcp、因为需要注册gpu mem buffer，只有支持gdr才能注册成功。我们有fallback to cpu mem的实现，但即使能run性能不一定好，暂时不是高优推进的方向。会在各种高优性能优化都做完之后再回来考虑。Transfer engine和Store组件本身支持tcp。

bsakaguchi commented

2026-10-29 21:09:21 +00:00

Author

Owner

@ictzyqq commented on GitHub (May 18, 2025):

@ictzyqq SGLang的PD实现不支持tcp、因为需要注册gpu mem buffer，只有支持gdr才能注册成功。我们有fallback to cpu mem的实现，但即使能run性能不一定好，暂时不是高优推进的方向。会在各种高优性能优化都做完之后再回来考虑。Transfer engine和Store组件本身支持tcp。

谢谢您的回复！请问这个fallback to cpu mem的实现是已经合入到sglang主分支里了吗？
我现在就想基于sglang v0.4.6.post4实现机内PD分离的功能（不追求性能），但由于没有RDMA，prefill看起来是自动选择了TCP传输，并报错：

I20250516 09:00:52.199445 45697 transfer_engine.cpp:44] Transfer Engine starting. Server: 10.8.59.192, Metadata: P2PHANDSHAKE, ip_or_host_name: , rpc_port: 0
I20250516 09:00:52.199507 45697 transfer_engine.cpp:100] Transfer Engine RPC using P2P handshake, listening on 10.8.59.192:16217
I20250516 09:00:52.199564 45697 transfer_engine.cpp:112] Auto-discovering topology...
I20250516 09:00:52.199565 46826 transfer_engine.cpp:357] Metrics reporting thread started (interval: 5s)
W20250516 09:00:52.240088 45697 topology.cpp:55] No RDMA devices found, check your device installation
I20250516 09:00:52.240196 45697 transfer_engine.cpp:127] Topology discovery complete. Found 0 HCAs.
I20250516 09:00:52.240238 45697 tcp_transport.cpp:241] TcpTransport: listen on port 16218
[2025-05-16 09:00:52] 10.8.59.192 [16/May/2025:09:00:52 +0000] "PUT /route HTTP/1.1" 200 155 "-" "python-requests/2.32.3"
[2025-05-16 09:00:52] INFO: Started server process [45012]
[2025-05-16 09:00:52] INFO: Waiting for application startup.
[2025-05-16 09:00:52] INFO: Application startup complete.
[2025-05-16 09:00:52] INFO: Uvicorn running on http://10.8.59.192:30011 (Press CTRL+C to quit)
[2025-05-16 09:00:53] INFO: 10.8.59.192:57062 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-05-16 09:00:53] Start of prefill warmup ...
[2025-05-16 09:00:53] FakeKVSender init with kv_indices: 4, aux_index: 0, dest_ranks: None
[2025-05-16 09:00:53] Prefill batch. #new-seq: 1, #new-token: 4, #cached-token: 0, token usage: 0.00, #running-req: 0, #unbootstrapped-req: 0, #queue-req: 0, #transferring-req: 0
2025-05-16 09:00:54,835 - INFO - flashinfer.jit: Loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_Fal
se
2025-05-16 09:00:54,874 - INFO - flashinfer.jit: Finished loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_
f16qk_False
[2025-05-16 09:00:55] FakeKVSender send with kv_indices: [1 2 3 4], index_slice: slice(0, 4, None), is_last: True
[2025-05-16 09:00:55] FakeKVSender send success
[2025-05-16 09:00:55] FakeKVSender poll success
[2025-05-16 09:00:55] INFO: 10.8.59.192:57064 - "POST /generate HTTP/1.1" 200 OK
[2025-05-16 09:00:55] End of prefill warmup with status 200, resp: [{'text': '#', 'meta_info': {'id': '043e96d7aa724360b1f2a4165cecf549', 'finish_reason': {'type': 'length', 'length': 0}, 'prompt_tokens': 4, 'completion_tokens': 1, '
cached_tokens': 0, 'e2e_latency': 1.7231113910675049}}]
[2025-05-16 09:00:55] The server is fired up and ready to roll!
[2025-05-16 09:01:13] 10.8.59.192 [16/May/2025:09:01:13 +0000] "GET /route?engine_rank=-1&target_dp_group=-1 HTTP/1.1" 200 204 "-" "python-requests/2.32.3"
[2025-05-16 09:01:13] 10.8.59.192 [16/May/2025:09:01:13 +0000] "GET /route?engine_rank=0&target_dp_group=0 HTTP/1.1" 200 206 "-" "python-requests/2.32.3"
[2025-05-16 09:01:13] Prefill batch. #new-seq: 1, #new-token: 9, #cached-token: 0, token usage: 0.00, #running-req: 0, #unbootstrapped-req: 0, #queue-req: 0, #transferring-req: 0
E20250516 09:01:13.107141 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E20250516 09:01:13.107199 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E20250516 09:01:13.107254 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E20250516 09:01:13.107276 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E20250516 09:01:13.107299 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E20250516 09:01:13.107321 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E20250516 09:01:13.107403 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E20250516 09:01:13.107425 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E20250516 09:01:13.107445 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
[2025-05-16 09:01:13] Mooncake Transfer Engine Return Error.
[2025-05-16 09:01:13] Mooncake Transfer Engine Return Error.

这个报错的原因就是您说的gpu mem buffer没有注册成功吧？对于没有RDMA的设备如何实现PD分离的功能（例如使用fallback to cpu mem的实现），请问您有什么建议吗？

@ictzyqq commented on GitHub (May 18, 2025): > [@ictzyqq](https://github.com/ictzyqq) SGLang的PD实现不支持tcp、因为需要注册gpu mem buffer，只有支持gdr才能注册成功。我们有fallback to cpu mem的实现，但即使能run性能不一定好，暂时不是高优推进的方向。会在各种高优性能优化都做完之后再回来考虑。Transfer engine和Store组件本身支持tcp。谢谢您的回复！请问这个fallback to cpu mem的实现是已经合入到sglang主分支里了吗？我现在就想基于sglang v0.4.6.post4实现机内PD分离的功能（不追求性能），但由于没有RDMA，prefill看起来是自动选择了TCP传输，并报错： I20250516 09:00:52.199445 45697 transfer_engine.cpp:44] Transfer Engine starting. Server: 10.8.59.192, Metadata: P2PHANDSHAKE, ip_or_host_name: , rpc_port: 0 I20250516 09:00:52.199507 45697 transfer_engine.cpp:100] Transfer Engine RPC using P2P handshake, listening on 10.8.59.192:16217 I20250516 09:00:52.199564 45697 transfer_engine.cpp:112] Auto-discovering topology... I20250516 09:00:52.199565 46826 transfer_engine.cpp:357] Metrics reporting thread started (interval: 5s) W20250516 09:00:52.240088 45697 topology.cpp:55] **No RDMA devices found, check your device installation** I20250516 09:00:52.240196 45697 transfer_engine.cpp:127] **Topology discovery complete. Found 0 HCAs.** I20250516 09:00:52.240238 45697 tcp_transport.cpp:241] **TcpTransport: listen on port 16218** [2025-05-16 09:00:52] 10.8.59.192 [16/May/2025:09:00:52 +0000] "PUT /route HTTP/1.1" 200 155 "-" "python-requests/2.32.3" [2025-05-16 09:00:52] INFO: Started server process [45012] [2025-05-16 09:00:52] INFO: Waiting for application startup. [2025-05-16 09:00:52] INFO: Application startup complete. [2025-05-16 09:00:52] INFO: Uvicorn running on http://10.8.59.192:30011 (Press CTRL+C to quit) [2025-05-16 09:00:53] INFO: 10.8.59.192:57062 - "GET /get_model_info HTTP/1.1" 200 OK [2025-05-16 09:00:53] Start of prefill warmup ... [2025-05-16 09:00:53] FakeKVSender init with kv_indices: 4, aux_index: 0, dest_ranks: None [2025-05-16 09:00:53] Prefill batch. #new-seq: 1, #new-token: 4, #cached-token: 0, token usage: 0.00, #running-req: 0, #unbootstrapped-req: 0, #queue-req: 0, #transferring-req: 0 2025-05-16 09:00:54,835 - INFO - flashinfer.jit: Loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_Fal se 2025-05-16 09:00:54,874 - INFO - flashinfer.jit: Finished loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_ f16qk_False [2025-05-16 09:00:55] FakeKVSender send with kv_indices: [1 2 3 4], index_slice: slice(0, 4, None), is_last: True [2025-05-16 09:00:55] FakeKVSender send success [2025-05-16 09:00:55] FakeKVSender poll success [2025-05-16 09:00:55] INFO: 10.8.59.192:57064 - "POST /generate HTTP/1.1" 200 OK [2025-05-16 09:00:55] End of prefill warmup with status 200, resp: [{'text': '#', 'meta_info': {'id': '043e96d7aa724360b1f2a4165cecf549', 'finish_reason': {'type': 'length', 'length': 0}, 'prompt_tokens': 4, 'completion_tokens': 1, ' cached_tokens': 0, 'e2e_latency': 1.7231113910675049}}] [2025-05-16 09:00:55] The server is fired up and ready to roll! [2025-05-16 09:01:13] 10.8.59.192 [16/May/2025:09:01:13 +0000] "GET /route?engine_rank=-1&target_dp_group=-1 HTTP/1.1" 200 204 "-" "python-requests/2.32.3" [2025-05-16 09:01:13] 10.8.59.192 [16/May/2025:09:01:13 +0000] "GET /route?engine_rank=0&target_dp_group=0 HTTP/1.1" 200 206 "-" "python-requests/2.32.3" [2025-05-16 09:01:13] Prefill batch. #new-seq: 1, #new-token: 9, #cached-token: 0, token usage: 0.00, #running-req: 0, #unbootstrapped-req: 0, #queue-req: 0, #transferring-req: 0 E20250516 09:01:13.107141 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E20250516 09:01:13.107199 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E20250516 09:01:13.107254 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E20250516 09:01:13.107276 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E20250516 09:01:13.107299 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E20250516 09:01:13.107321 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E20250516 09:01:13.107403 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E20250516 09:01:13.107425 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E20250516 09:01:13.107445 46830 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 [2025-05-16 09:01:13] Mooncake Transfer Engine Return Error. [2025-05-16 09:01:13] Mooncake Transfer Engine Return Error. 这个报错的原因就是您说的gpu mem buffer没有注册成功吧？对于没有RDMA的设备如何实现PD分离的功能（例如使用fallback to cpu mem的实现），请问您有什么建议吗？

bsakaguchi commented

2026-10-29 21:09:22 +00:00

Author

Owner

@ShangmingCai commented on GitHub (May 18, 2025):

@ictzyqq 试试这个pr https://github.com/sgl-project/sglang/pull/5664 ，但是可能目前有conflict需要自己fix一下。如果是机内，不建议走mooncake tcp，你也许可以自己写一个nccl的wrapper来传输。

@ShangmingCai commented on GitHub (May 18, 2025): @ictzyqq 试试这个pr https://github.com/sgl-project/sglang/pull/5664 ，但是可能目前有conflict需要自己fix一下。如果是机内，不建议走mooncake tcp，你也许可以自己写一个nccl的wrapper来传输。

bsakaguchi commented

2026-10-29 21:09:22 +00:00

Author

Owner

@ictzyqq commented on GitHub (May 18, 2025):

@ShangmingCai 好的，谢谢大佬！

@ictzyqq commented on GitHub (May 18, 2025): @ShangmingCai 好的，谢谢大佬！

bsakaguchi commented

2026-10-29 21:09:22 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 15, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 15, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:09:22 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 22, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 22, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

Rows
Columns

[GH-ISSUE #257] sglang使用mooncake报错 #104