[GH-ISSUE #532] VLLM多节点部署Deepseek-R1出现Ray集群崩溃的错误 #420

New issue

Closed

opened 2026-10-26 21:21:03 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:21:03 +00:00

Owner

Originally created by @loki1017 on GitHub (Mar 4, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/532

我想使用Vllm提供的脚本部署满血版Deepseek-R1(671B), 首先使用vllm官方提供的run_cluster脚本：

#!/bin/bash

if [ $# -lt 4 ]; then
echo "Usage: $0 docker_image head_node_address --head|--worker path_to_hf_home [additional_args...]"
exit 1
fi

DOCKER_IMAGE="$1"
HEAD_NODE_ADDRESS="$2"
NODE_TYPE="$3" # Should be --head or --worker
PATH_TO_HF_HOME="$4"
shift 4

ADDITIONAL_ARGS=("$@")

if [ "${NODE_TYPE}" != "--head" ] && [ "${NODE_TYPE}" != "--worker" ]; then
echo "Error: Node type must be --head or --worker"
exit 1
fi

cleanup() {
docker stop node
docker rm node
}
trap cleanup EXIT

RAY_START_CMD="ray start --block"
if [ "${NODE_TYPE}" == "--head" ]; then
RAY_START_CMD+=" --head --port=6379"
else
RAY_START_CMD+=" --address=${HEAD_NODE_ADDRESS}:6379"
fi

docker run
--entrypoint /bin/bash
--network host
--name node
--shm-size 10.24g
--gpus all
-v "${PATH_TO_HF_HOME}:/root/.cache/huggingface"
"${ADDITIONAL_ARGS[@]}"
"${DOCKER_IMAGE}" -c "${RAY_START_CMD}"

对脚本做了修改：

因为Head节点中6379端口号被占用了，所以我将将端口号修改成了3338
在RAY_START_CMD+=" --address=${HEAD_NODE_ADDRESS}:6379"中硬编码指定了Worker的IP地址，增加命令 --node-ip-address=XXX.XXX.XXX.101
run命令中指定显卡 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7。两个节点加起来16张卡

我将服务器100作为Head节点，服务器101作为Worker节点，总共16张80G A800，执行启动命令：
服务器100：
bash run_cluster.sh
vllm/vllm-openai:latest
XXX.XXX.XXX.100
--head
/path/to/the/huggingface/home/in/this/node
-e VLLM_HOST_IP=XXX.XXX.XXX.100

服务器101：
bash run_cluster.sh
vllm/vllm-openai:latest
XXX.XXX.XXX.100
--worker
/path/to/the/huggingface/home/in/this/node
-e VLLM_HOST_IP=XXX.XXX.XXX.101

执行之后，进入Head节点的node容器中：
docker exec -it node /bin/bash

查看Ray status
开始显示两个节点16张GPU，但是再执行一次就只能发现一个节点 😅:
root@hxrj-X640-G40:/vllm-workspace# ray status
======== Autoscaler status: 2025-03-03 19:43:53.393690 ========
Node status

Active:
1 node_941be23a6aa1c5ceddd78cde63be3c5edfcd3fa9b40c9c608589462e
1 node_b9506642f73df1a75b4ba7bb216e45a1026fbae4e06cece133db8236
Pending:
(no pending nodes)
Recent failures:
(no failures)

Resources

Usage:
0.0/224.0 CPU
0.0/16.0 GPU
0B/1.94TiB memory
0B/19.46GiB object_store_memory

Demands:
(no resource demands)

root@hxrj-X640-G40:/vllm-workspace# ray status
======== Autoscaler status: 2025-03-03 19:44:18.425174 ========
Node status

Active:
1 node_941be23a6aa1c5ceddd78cde63be3c5edfcd3fa9b40c9c608589462e
Pending:
(no pending nodes)
Recent failures:
(no failures)

Resources

Usage:
0.0/112.0 CPU
0.0/8.0 GPU
0B/987.88GiB memory
0B/9.73GiB object_store_memory

Demands:
(no resource demands)

最后，我发现是101服务器中容器闪退的问题：
(vllm0.7.2) zzz@hxrj:/Work/zzz/vllm$ docker logs vllm
Local node IP: 192.168.141.239
[2025-03-03 19:43:40,951 W 1 1] global_state_accessor.cc:463: Retrying to get node with node ID b9506642f73df1a75b4ba7bb216e45a1026fbae4e06cece133db8236

Ray runtime started.

To terminate the Ray runtime, run
ray stop

--block
This command will now block forever until terminated by a signal.
Running subprocesses are monitored and a message will be printed if any of them terminate unexpectedly. Subprocesses exit with SIGTERM will be treated as graceful, thus NOT reported.

Some Ray subprocesses exited unexpectedly:
raylet [exit code=1]

Remaining processes will be killed.

我不知道为什么这样，希望大家解惑，给亲们磕头了！😭

Originally created by @loki1017 on GitHub (Mar 4, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/532 我想使用Vllm提供的脚本部署满血版Deepseek-R1(671B), 首先使用vllm官方提供的run_cluster脚本： #!/bin/bash if [ $# -lt 4 ]; then echo "Usage: $0 docker_image head_node_address --head|--worker path_to_hf_home [additional_args...]" exit 1 fi DOCKER_IMAGE="$1" HEAD_NODE_ADDRESS="$2" NODE_TYPE="$3" # Should be --head or --worker PATH_TO_HF_HOME="$4" shift 4 ADDITIONAL_ARGS=("$@") if [ "${NODE_TYPE}" != "--head" ] && [ "${NODE_TYPE}" != "--worker" ]; then echo "Error: Node type must be --head or --worker" exit 1 fi cleanup() { docker stop node docker rm node } trap cleanup EXIT RAY_START_CMD="ray start --block" if [ "${NODE_TYPE}" == "--head" ]; then RAY_START_CMD+=" --head --port=6379" else RAY_START_CMD+=" --address=${HEAD_NODE_ADDRESS}:6379" fi docker run \ --entrypoint /bin/bash \ --network host \ --name node \ --shm-size 10.24g \ --gpus all \ -v "${PATH_TO_HF_HOME}:/root/.cache/huggingface" \ "${ADDITIONAL_ARGS[@]}" \ "${DOCKER_IMAGE}" -c "${RAY_START_CMD}" 对脚本做了修改： 1. 因为Head节点中6379端口号被占用了，所以我将将端口号修改成了3338 2. 在RAY_START_CMD+=" --address=${HEAD_NODE_ADDRESS}:6379"中硬编码指定了Worker的IP地址，增加命令 --node-ip-address=XXX.XXX.XXX.101 3. run命令中指定显卡 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7。两个节点加起来16张卡我将服务器100作为Head节点，服务器101作为Worker节点，总共16张80G A800，执行启动命令：服务器100： bash run_cluster.sh \ vllm/vllm-openai:latest \ XXX.XXX.XXX.100 \ --head \ /path/to/the/huggingface/home/in/this/node \ -e VLLM_HOST_IP=XXX.XXX.XXX.100 服务器101： bash run_cluster.sh \ vllm/vllm-openai:latest \ XXX.XXX.XXX.100 \ --worker \ /path/to/the/huggingface/home/in/this/node \ -e VLLM_HOST_IP=XXX.XXX.XXX.101 执行之后，进入Head节点的node容器中： docker exec -it node /bin/bash 查看Ray status 开始显示两个节点16张GPU，但是再执行一次就只能发现一个节点 😅: root@hxrj-X640-G40:/vllm-workspace# ray status ======== Autoscaler status: 2025-03-03 19:43:53.393690 ======== Node status --------------------------------------------------------------- Active: 1 node_941be23a6aa1c5ceddd78cde63be3c5edfcd3fa9b40c9c608589462e 1 node_b9506642f73df1a75b4ba7bb216e45a1026fbae4e06cece133db8236 Pending: (no pending nodes) Recent failures: (no failures) Resources --------------------------------------------------------------- Usage: 0.0/224.0 CPU 0.0/16.0 GPU 0B/1.94TiB memory 0B/19.46GiB object_store_memory Demands: (no resource demands) root@hxrj-X640-G40:/vllm-workspace# ray status ======== Autoscaler status: 2025-03-03 19:44:18.425174 ======== Node status --------------------------------------------------------------- Active: 1 node_941be23a6aa1c5ceddd78cde63be3c5edfcd3fa9b40c9c608589462e Pending: (no pending nodes) Recent failures: (no failures) Resources --------------------------------------------------------------- Usage: 0.0/112.0 CPU 0.0/8.0 GPU 0B/987.88GiB memory 0B/9.73GiB object_store_memory Demands: (no resource demands) 最后，我发现是101服务器中容器闪退的问题： (vllm0.7.2) zzz@hxrj:/Work/zzz/vllm$ docker logs vllm Local node IP: 192.168.141.239 [2025-03-03 19:43:40,951 W 1 1] global_state_accessor.cc:463: Retrying to get node with node ID b9506642f73df1a75b4ba7bb216e45a1026fbae4e06cece133db8236 -------------------- Ray runtime started. -------------------- To terminate the Ray runtime, run ray stop --block This command will now block forever until terminated by a signal. Running subprocesses are monitored and a message will be printed if any of them terminate unexpectedly. Subprocesses exit with SIGTERM will be treated as graceful, thus NOT reported. Some Ray subprocesses exited unexpectedly: raylet [exit code=1] Remaining processes will be killed. 我不知道为什么这样，希望大家解惑，给亲们磕头了！😭

bsakaguchi closed this issue

2026-10-26 21:21:03 +00:00