[GH-ISSUE #519] PD Diaggragate with sglang failed by【tcp_transport Session::writeBody failed. Error: Bad address】 #179

New issue

Closed

opened 2026-10-29 21:10:34 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:10:34 +00:00

Owner

Originally created by @mugglew on GitHub (Jun 19, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/519

hello all, here is my command

python3 -m sglang.launch_server \
  --model almp_bucket_malaysia/prod/models/QWEN3-32B/Qwen3-32B \
  --trust-remote-code \
  --disaggregation-mode prefill \
  --port 30000 \
  --disaggregation-ib-device mlx5_1 \
  --host 0.0.0.0 > /prefiller.log 2>&1 &

python3 -m sglang.launch_server \
  --model almp_bucket_malaysia/prod/models/QWEN3-32B/Qwen3-32B \
  --trust-remote-code \
  --disaggregation-mode decode \
  --port 30001 \
  --base-gpu-id 1 \
  --disaggregation-ib-device mlx5_1 \
  --host 0.0.0.0 > /decoder.log 2>&1 &

python3 -m sglang.srt.disaggregation.mini_lb \
  --prefill http://0.0.0.0:30000 \
  --decode http://0.0.0.0:30001 \
  --host 0.0.0.0 \
  --port 8000

it starts success but reports error when testing by api chat

prefill.log
{"text":"","meta_info":{"id":"c564d0c547f945aa987620f1255b838d","finish_reason":{"type":"abort","message":"Decode transfer failed for request rank=0 decode_req.req.rid='c564d0c547f945aa987620f1255b838d' decode_req.req.bootstrap_room=2170015846053279415 with exception KVTransferError(bootstrap_room=2170015846053279415): Failed to get kvcache from prefill instance, it might be dead","status_code":500,"err_type":null},"prompt_tokens":8,"completion_tokens":0,"cached_tokens":0,"e2e_latency":0.07976341247558594}}root@hd19nh7b2md0c7iunpfng-77c96f889b-7hls2:/# tail -f prefiller.log
E0619 02:47:42.441629 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.441717 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.442241 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.442361 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.442468 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.442691 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.442728 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
[2025-06-19 02:47:42] Session 192.168.31.14:16002 failed.
[2025-06-19 02:47:42] Prefill transfer failed for request rank=0 req.rid='d14876c8385443e9ab44dca283c28ce9' req.bootstrap_room=2170015846053279415 with exception KVTransferError(bootstrap_room=2170015846053279415): Failed to send kv chunk of 2170015846053279415 to 192.168.31.14:35457
[2025-06-19 02:47:42] INFO: 127.0.0.1:46246 - "POST /generate HTTP/1.1" 200 OK
^C

decode.log
root@hd19nh7b2md0c7iunpfng-77c96f889b-7hls2:/# tail -f decoder.log
E0619 02:47:42.441622 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.441694 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.441780 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.442312 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.442481 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.442536 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.442739 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0619 02:47:42.442775 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
[2025-06-19 02:47:42] Decode transfer failed for request rank=0 decode_req.req.rid='c564d0c547f945aa987620f1255b838d' decode_req.req.bootstrap_room=2170015846053279415 with exception KVTransferError(bootstrap_room=2170015846053279415): Failed to get kvcache from prefill instance, it might be dead

here is my nvidai-smi topo -m

I Just really want to use pd disaggragate by sglang-mooncake
CAN SOMEONE HELP ME? THANK YOU SO MUCH

Originally created by @mugglew on GitHub (Jun 19, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/519 hello all, here is my command ```shell python3 -m sglang.launch_server \ --model almp_bucket_malaysia/prod/models/QWEN3-32B/Qwen3-32B \ --trust-remote-code \ --disaggregation-mode prefill \ --port 30000 \ --disaggregation-ib-device mlx5_1 \ --host 0.0.0.0 > /prefiller.log 2>&1 & python3 -m sglang.launch_server \ --model almp_bucket_malaysia/prod/models/QWEN3-32B/Qwen3-32B \ --trust-remote-code \ --disaggregation-mode decode \ --port 30001 \ --base-gpu-id 1 \ --disaggregation-ib-device mlx5_1 \ --host 0.0.0.0 > /decoder.log 2>&1 & python3 -m sglang.srt.disaggregation.mini_lb \ --prefill http://0.0.0.0:30000 \ --decode http://0.0.0.0:30001 \ --host 0.0.0.0 \ --port 8000 ``` it starts success but reports error when testing by api chat prefill.log {"text":"","meta_info":{"id":"c564d0c547f945aa987620f1255b838d","finish_reason":{"type":"abort","message":"Decode transfer failed for request rank=0 decode_req.req.rid='c564d0c547f945aa987620f1255b838d' decode_req.req.bootstrap_room=2170015846053279415 with exception KVTransferError(bootstrap_room=2170015846053279415): Failed to get kvcache from prefill instance, it might be dead","status_code":500,"err_type":null},"prompt_tokens":8,"completion_tokens":0,"cached_tokens":0,"e2e_latency":0.07976341247558594}}root@hd19nh7b2md0c7iunpfng-77c96f889b-7hls2:/# tail -f prefiller.log E0619 02:47:42.441629 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.441717 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.442241 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.442361 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.442468 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.442691 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.442728 1555 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 [2025-06-19 02:47:42] Session 192.168.31.14:16002 failed. [2025-06-19 02:47:42] Prefill transfer failed for request rank=0 req.rid='d14876c8385443e9ab44dca283c28ce9' req.bootstrap_room=2170015846053279415 with exception KVTransferError(bootstrap_room=2170015846053279415): Failed to send kv chunk of 2170015846053279415 to 192.168.31.14:35457 [2025-06-19 02:47:42] INFO: 127.0.0.1:46246 - "POST /generate HTTP/1.1" 200 OK ^C decode.log root@hd19nh7b2md0c7iunpfng-77c96f889b-7hls2:/# tail -f decoder.log E0619 02:47:42.441622 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.441694 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.441780 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.442312 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.442481 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.442536 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.442739 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0619 02:47:42.442775 1620 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 [2025-06-19 02:47:42] Decode transfer failed for request rank=0 decode_req.req.rid='c564d0c547f945aa987620f1255b838d' decode_req.req.bootstrap_room=2170015846053279415 with exception KVTransferError(bootstrap_room=2170015846053279415): Failed to get kvcache from prefill instance, it might be dead here is my nvidai-smi topo -m ![Image](https://github.com/user-attachments/assets/7a251cae-264e-4f72-b35d-cd79ff00ee0c) I Just really want to use pd disaggragate by sglang-mooncake CAN SOMEONE HELP ME? THANK YOU SO MUCH

bsakaguchi closed this issue

2026-10-29 21:10:34 +00:00

bsakaguchi commented

2026-10-29 21:10:34 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Jun 19, 2025):

Users should install the gdr through this command: sudo apt-get install -y nvidia-peer-memory or from source, and need to make sure the RDMA driver version supports gdr. After modprobe, users can check it through gdrcopy . If it runs in the Docker container, please grant it with --privileged, and make sure the container has the root permission. After these processes, we should be able to register the GPU memory with Mooncake and do the transfer successfully.

@ShangmingCai commented on GitHub (Jun 19, 2025): Users should install the gdr through this command: sudo apt-get install -y nvidia-peer-memory or from [source](https://github.com/Mellanox/nv_peer_memory), and need to make sure the RDMA driver version supports gdr. After modprobe, users can check it through [gdrcopy ](https://github.com/NVIDIA/gdrcopy). If it runs in the Docker container, please grant it with --privileged, and make sure the container has the root permission. After these processes, we should be able to register the GPU memory with Mooncake and do the transfer successfully.

bsakaguchi referenced this issue

2026-10-29 21:26:52 +00:00

[PR #179] [MERGED] [Refactor] [MooncakeStore] Migrate Master Service from gRPC to coro_rpc #1374