kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #256] Network problem about SGLang PD disaggregation #105

New issue

Closed

opened 2026-10-29 21:09:20 +00:00 by bsakaguchi · 8 comments

bsakaguchi commented

2026-10-29 21:09:20 +00:00

Owner

Originally created by @LJL36 on GitHub (Apr 16, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/256

Hi, I want to continue the disccusion about the problem I encountered while using the PD disaggregation by SGLang.

Mooncake config and commands

node-1(29.163.184.201)

# the prefill node
MC_TCP_BIND_ADDRESS=29.163.184.201 MOONCAKE_CONFIG_PATH=./mooncake.json python -m sglang.launch_server --model-path /dockerdata/bin/19967/DeepSeek_R1_671B_model/1 --disaggregation-mode prefill --port 30000 --host 29.163.184.201 --tp-size 8 --mem-fraction-static 0.9 --enable-flashinfer-mla

# the proxy server
python3 -m sglang.srt.disaggregation.mini_lb --prefill http://29.163.184.201:30000 --decode http://29.163.178.171:30001 --host 0.0.0.0 --port 8000

{
    "local_hostname": "29.163.184.201",
    "metadata_server": "http://29.163.184.201:8998/metadata",
    "protocol": "rdma",
    "device_name": "mlx5_bond_1,mlx5_bond_2,mlx5_bond_3,mlx5_bond_4,mlx5_bond_5,mlx5_bond_6,mlx5_bond_7,mlx5_bond_8"
}

node-2(29.163.178.171)

# the decode node
MC_TCP_BIND_ADDRESS=29.163.178.171 MOONCAKE_CONFIG_PATH=./mooncake.json python -m sglang.launch_server --model-path /dockerdata/bin/19967/DeepSeek_R1_671B_model/1 --disaggregation-mode decode --port 30001 --host 29.163.178.171 --tp-size 8 --mem-fraction-static 0.9 --enable-flashinfer-mla

{
    "local_hostname": "29.163.178.171",
    "metadata_server": "http://29.163.184.201:8998/metadata",
    "protocol": "rdma",
    "device_name": "mlx5_bond_1,mlx5_bond_2,mlx5_bond_3,mlx5_bond_4,mlx5_bond_5,mlx5_bond_6,mlx5_bond_7,mlx5_bond_8"
}

Introduction

Some running log
As mentioned here, the several requests at the beginnning is OK. After i issued the pressure test, the following error will be raised uncertainly.

All of the above are the situations encountered yesterday.

New discoveries

Today, I switch to the latest main branch of SGLang and Mooncake. And I found the number of connections between the two nodes is always increasing.
After all servers started(prefill node/decode node/proxy server), the number of connections is 8 keeping the connection between the metadata server and the decode node. For each request, the number of connections will be increased by 9, and after the request is completed, one is reduced by. In total 8 connections will be increased for each request and will not be released. When the number of connections increases more than 1000, the pressure test task will stuck. If restarting the task, the number of connections will increase to nearly 2000 after running for a period of time, and the network card will fail.

Originally created by @LJL36 on GitHub (Apr 16, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/256 Hi, I want to continue the disccusion about the problem I encountered while using the PD disaggregation by SGLang. ## Mooncake config and commands 1. node-1(29.163.184.201) ```bash # the prefill node MC_TCP_BIND_ADDRESS=29.163.184.201 MOONCAKE_CONFIG_PATH=./mooncake.json python -m sglang.launch_server --model-path /dockerdata/bin/19967/DeepSeek_R1_671B_model/1 --disaggregation-mode prefill --port 30000 --host 29.163.184.201 --tp-size 8 --mem-fraction-static 0.9 --enable-flashinfer-mla # the proxy server python3 -m sglang.srt.disaggregation.mini_lb --prefill http://29.163.184.201:30000 --decode http://29.163.178.171:30001 --host 0.0.0.0 --port 8000 ``` ```json { "local_hostname": "29.163.184.201", "metadata_server": "http://29.163.184.201:8998/metadata", "protocol": "rdma", "device_name": "mlx5_bond_1,mlx5_bond_2,mlx5_bond_3,mlx5_bond_4,mlx5_bond_5,mlx5_bond_6,mlx5_bond_7,mlx5_bond_8" } ``` 2. node-2(29.163.178.171) ```bash # the decode node MC_TCP_BIND_ADDRESS=29.163.178.171 MOONCAKE_CONFIG_PATH=./mooncake.json python -m sglang.launch_server --model-path /dockerdata/bin/19967/DeepSeek_R1_671B_model/1 --disaggregation-mode decode --port 30001 --host 29.163.178.171 --tp-size 8 --mem-fraction-static 0.9 --enable-flashinfer-mla ``` ```json { "local_hostname": "29.163.178.171", "metadata_server": "http://29.163.184.201:8998/metadata", "protocol": "rdma", "device_name": "mlx5_bond_1,mlx5_bond_2,mlx5_bond_3,mlx5_bond_4,mlx5_bond_5,mlx5_bond_6,mlx5_bond_7,mlx5_bond_8" } ``` ## Introduction Some running log As mentioned [here](https://github.com/sgl-project/sglang/pull/4880#issuecomment-2805174018), the several requests at the beginnning is OK. After i issued the pressure test, the following error will be raised uncertainly. <img width="1885" alt="Image" src="https://github.com/user-attachments/assets/107f1668-0e11-428d-8730-6bee9520d3d0" /> <img width="1879" alt="Image" src="https://github.com/user-attachments/assets/fef7f571-f2fa-4bda-aa91-5725a9d8d897" /> All of the above are the situations encountered yesterday. ## New discoveries Today, I switch to the latest main branch of SGLang and Mooncake. And I found the number of connections between the two nodes is always increasing. After all servers started(prefill node/decode node/proxy server), the number of connections is 8 keeping the connection between the metadata server and the decode node. For each request, the number of connections will be increased by 9, and after the request is completed, one is reduced by. In total 8 connections will be increased for each request and will not be released. When the number of connections increases more than 1000, the pressure test task will stuck. If restarting the task, the number of connections will increase to nearly 2000 after running for a period of time, and the network card will fail.

bsakaguchi closed this issue

2026-10-29 21:09:20 +00:00

bsakaguchi commented

2026-10-29 21:09:21 +00:00

Author

Owner

@LJL36 commented on GitHub (Apr 16, 2025):

We found that the number of connections will always be increasing and takes up most of the decode node's memory. Then some connections will be released. I guess that is why the pressure test will stuck for corresponding requests have been cancelled. And at the time the network card fails, the memory exhausted. Have you ever encountered a situation where the number of connections only increases and does not decrease?

In addition, I want to ask that why as the number of connections increases, the memory usage of the decode node increases, but the memory usage of the prefill node remains the same.

@LJL36 commented on GitHub (Apr 16, 2025): We found that the number of connections will always be increasing and takes up most of the decode node's memory. Then some connections will be released. I guess that is why the pressure test will stuck for corresponding requests have been cancelled. And at the time the network card fails, the memory exhausted. Have you ever encountered a situation where the number of connections only increases and does not decrease? In addition, I want to ask that why as the number of connections increases, the memory usage of the decode node increases, but the memory usage of the prefill node remains the same.

bsakaguchi commented

2026-10-29 21:09:21 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Apr 16, 2025):

the number of connections is 8 keeping the connection between the metadata server and the decode node. For each request, the number of connections will be increased by 9, and after the request is completed, one is reduced by. In total 8 connections will be increased for each request and will not be released. When the number of connections increases more than 1000, the pressure test task will stuck. If restarting the task, the number of connections will increase to nearly 2000 after running for a period of time, and the network card will fail.

We reuse the connection at the engine level. Can you show us what type of connection number increase? Did you mean the underlying transfer sync processes of the transfer engine?

CC: @alogfans

@ShangmingCai commented on GitHub (Apr 16, 2025): > the number of connections is 8 keeping the connection between the metadata server and the decode node. For each request, the number of connections will be increased by 9, and after the request is completed, one is reduced by. In total 8 connections will be increased for each request and will not be released. When the number of connections increases more than 1000, the pressure test task will stuck. If restarting the task, the number of connections will increase to nearly 2000 after running for a period of time, and the network card will fail. > We reuse the connection at the engine level. Can you show us what type of connection number increase? Did you mean the underlying transfer sync processes of the transfer engine? CC: @alogfans

bsakaguchi commented

2026-10-29 21:09:21 +00:00

Author

Owner

@LJL36 commented on GitHub (Apr 17, 2025):

the number of connections is 8 keeping the connection between the metadata server and the decode node. For each request, the number of connections will be increased by 9, and after the request is completed, one is reduced by. In total 8 connections will be increased for each request and will not be released. When the number of connections increases more than 1000, the pressure test task will stuck. If restarting the task, the number of connections will increase to nearly 2000 after running for a period of time, and the network card will fail.

We reuse the connection at the engine level. Can you show us what type of connection number increase? Did you mean the underlying transfer sync processes of the transfer engine?

CC: @alogfans

How can I distinguish the type of connection? I can only observe the pair of ip:port between two nodes. And these ports are held by the sglang::scheduler process

Here we changed two other nodes for testing

@LJL36 commented on GitHub (Apr 17, 2025): > > the number of connections is 8 keeping the connection between the metadata server and the decode node. For each request, the number of connections will be increased by 9, and after the request is completed, one is reduced by. In total 8 connections will be increased for each request and will not be released. When the number of connections increases more than 1000, the pressure test task will stuck. If restarting the task, the number of connections will increase to nearly 2000 after running for a period of time, and the network card will fail. > > We reuse the connection at the engine level. Can you show us what type of connection number increase? Did you mean the underlying transfer sync processes of the transfer engine? > > CC: [@alogfans](https://github.com/alogfans) How can I distinguish the type of connection? I can only observe the pair of ip:port between two nodes. And these ports are held by the sglang::scheduler process <img width="829" alt="Image" src="https://github.com/user-attachments/assets/cefdadd9-9982-4598-b49b-0807003481e4" /> Here we changed two other nodes for testing

bsakaguchi commented

2026-10-29 21:09:21 +00:00

Author

Owner

@Zhou-sx commented on GitHub (Apr 17, 2025):

I run successfully using one RDMA nic. But failed when I set four RDMA nics.

@Zhou-sx commented on GitHub (Apr 17, 2025): I run successfully using one RDMA nic. But failed when I set four RDMA nics.

bsakaguchi commented

2026-10-29 21:09:22 +00:00

Author

Owner

@LJL36 commented on GitHub (Apr 21, 2025):

After I use the latest version of SGLang and Mooncake, I encoutered the following problem

It seems mooncake try to use all the nic including the ones I don't include in the mooncake.json and how can i address it?

@LJL36 commented on GitHub (Apr 21, 2025): After I use the latest version of SGLang and Mooncake, I encoutered the following problem <img width="1876" alt="Image" src="https://github.com/user-attachments/assets/e9c0a274-bae7-4b74-b602-1007fceb05cc" /> It seems mooncake try to use all the nic including the ones I don't include in the mooncake.json and how can i address it?

bsakaguchi commented

2026-10-29 21:09:22 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Apr 21, 2025):

After I use the latest version of SGLang and Mooncake, I encoutered the following problem
It seems mooncake try to use all the nic including the ones I don't include in the mooncake.json and how can i address it?

Try --disaggregation-ib-device, append the ib-device that used to be in the JSON file.

@ShangmingCai commented on GitHub (Apr 21, 2025): > After I use the latest version of SGLang and Mooncake, I encoutered the following problem > > <img alt="Image" width="1876" src="https://private-user-images.githubusercontent.com/41277042/435619677-e9c0a274-bae7-4b74-b602-1007fceb05cc.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDUyMjc4MTUsIm5iZiI6MTc0NTIyNzUxNSwicGF0aCI6Ii80MTI3NzA0Mi80MzU2MTk2NzctZTljMGEyNzQtYmFlNy00Yjc0LWI2MDItMTAwN2ZjZWIwNWNjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA0MjElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNDIxVDA5MjUxNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJkMDYwMmExYzQwYWQ3MDMzMmE4MDUzNmZkYjI0MTE3ZDFjYjkzMWExZjc1YjBiMDI1NDdiMjdlY2Y4NjQxYzImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.Qm7f7sU8VScsdonv-vFWPjb77QjxE6E396Pdxyd0tEQ"> > It seems mooncake try to use all the nic including the ones I don't include in the mooncake.json and how can i address it? Try `--disaggregation-ib-device`, append the ib-device that used to be in the JSON file.

bsakaguchi commented

2026-10-29 21:09:22 +00:00

Author

Owner

@LJL36 commented on GitHub (Apr 21, 2025):

After I use the latest version of SGLang and Mooncake, I encoutered the following problem

It seems mooncake try to use all the nic including the ones I don't include in the mooncake.json and how can i address it?

Try --disaggregation-ib-device, append the ib-device that used to be in the JSON file.

can you show me how to run the latest PD disaggregation on two nodes? --disaggregation-ib-device replaces the device_name and which args replaces the metadata_server? the --dist-init-addr or --host?

@LJL36 commented on GitHub (Apr 21, 2025): > > After I use the latest version of SGLang and Mooncake, I encoutered the following problem > > <img alt="Image" width="1876" src="https://private-user-images.githubusercontent.com/41277042/435619677-e9c0a274-bae7-4b74-b602-1007fceb05cc.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDUyMjc4MTUsIm5iZiI6MTc0NTIyNzUxNSwicGF0aCI6Ii80MTI3NzA0Mi80MzU2MTk2NzctZTljMGEyNzQtYmFlNy00Yjc0LWI2MDItMTAwN2ZjZWIwNWNjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA0MjElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNDIxVDA5MjUxNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJkMDYwMmExYzQwYWQ3MDMzMmE4MDUzNmZkYjI0MTE3ZDFjYjkzMWExZjc1YjBiMDI1NDdiMjdlY2Y4NjQxYzImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.Qm7f7sU8VScsdonv-vFWPjb77QjxE6E396Pdxyd0tEQ"> > > It seems mooncake try to use all the nic including the ones I don't include in the mooncake.json and how can i address it? > > Try `--disaggregation-ib-device`, append the ib-device that used to be in the JSON file. can you show me how to run the latest PD disaggregation on two nodes? --disaggregation-ib-device replaces the device_name and which args replaces the metadata_server? the --dist-init-addr or --host?

bsakaguchi commented

2026-10-29 21:09:22 +00:00

Author

Owner

@LJL36 commented on GitHub (Apr 21, 2025):

After I use the latest version of SGLang and Mooncake, I encoutered the following problem

It seems mooncake try to use all the nic including the ones I don't include in the mooncake.json and how can i address it?

Try --disaggregation-ib-device, append the ib-device that used to be in the JSON file.

can you show me how to run the latest PD disaggregation on two nodes? --disaggregation-ib-device replaces the device_name and which args replaces the metadata_server? the --dist-init-addr or --host?

I run it successfully by not assigning the boostrap server ip maually, thanks

@LJL36 commented on GitHub (Apr 21, 2025): > > > After I use the latest version of SGLang and Mooncake, I encoutered the following problem > > > <img alt="Image" width="1876" src="https://private-user-images.githubusercontent.com/41277042/435619677-e9c0a274-bae7-4b74-b602-1007fceb05cc.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDUyMjc4MTUsIm5iZiI6MTc0NTIyNzUxNSwicGF0aCI6Ii80MTI3NzA0Mi80MzU2MTk2NzctZTljMGEyNzQtYmFlNy00Yjc0LWI2MDItMTAwN2ZjZWIwNWNjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA0MjElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNDIxVDA5MjUxNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJkMDYwMmExYzQwYWQ3MDMzMmE4MDUzNmZkYjI0MTE3ZDFjYjkzMWExZjc1YjBiMDI1NDdiMjdlY2Y4NjQxYzImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.Qm7f7sU8VScsdonv-vFWPjb77QjxE6E396Pdxyd0tEQ"> > > > It seems mooncake try to use all the nic including the ones I don't include in the mooncake.json and how can i address it? > > > > > > Try `--disaggregation-ib-device`, append the ib-device that used to be in the JSON file. > > can you show me how to run the latest PD disaggregation on two nodes? --disaggregation-ib-device replaces the device_name and which args replaces the metadata_server? the --dist-init-addr or --host? I run it successfully by not assigning the boostrap server ip maually, thanks