[GH-ISSUE #27] 请问如何用VLLM部署33B #20

New issue

Open

opened 2026-10-26 21:02:18 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-26 21:02:18 +00:00

Owner

Originally created by @laisun on GitHub (Nov 10, 2023).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/27

会报错啊，单机A100 ，torch 2.01， transformers 4.35
key = torch.repeat_interleave(key, self.num_queries_per_kv, dim=1)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Originally created by @laisun on GitHub (Nov 10, 2023). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/27 会报错啊，单机A100 ，torch 2.01， transformers 4.35 key = torch.repeat_interleave(key, self.num_queries_per_kv, dim=1) RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

bsakaguchi commented

2026-10-26 21:02:18 +00:00

Author

Owner

@soloice commented on GitHub (Nov 28, 2023):

单机 A100 是几张卡？打开 CUDA_LAUNCH_BLOCKING=1 试试呢，报错在哪里？

@soloice commented on GitHub (Nov 28, 2023): 单机 A100 是几张卡？打开 CUDA_LAUNCH_BLOCKING=1 试试呢，报错在哪里？

bsakaguchi commented

2026-10-26 21:02:19 +00:00

Author

Owner

@FrankWhh commented on GitHub (Nov 30, 2023):

我部署后输出是乱码，有人遇到过吗

@FrankWhh commented on GitHub (Nov 30, 2023): 我部署后输出是乱码，有人遇到过吗

bsakaguchi commented

2026-10-26 21:02:19 +00:00

Author

Owner

@txy6666yr commented on GitHub (Dec 10, 2023):

单机 A100 是几张卡？打开 CUDA_LAUNCH_BLOCKING=1 试试呢，报错在哪里？

请问有vllm部署的教程吗？或者文件分享下文件

@txy6666yr commented on GitHub (Dec 10, 2023): > 单机 A100 是几张卡？打开 CUDA_LAUNCH_BLOCKING=1 试试呢，报错在哪里？请问有vllm部署的教程吗？或者文件分享下文件

bsakaguchi commented

2026-10-26 21:02:19 +00:00

Author

Owner

@hyperbolic-c commented on GitHub (Apr 12, 2024):

请问vllm部署时如何使用多卡加载模型，使用CUDA_VISIBLE_DEVICES=0,1还是只有一张卡load了，很奇怪，谢谢

@hyperbolic-c commented on GitHub (Apr 12, 2024): 请问vllm部署时如何使用多卡加载模型，使用`CUDA_VISIBLE_DEVICES=0,1`还是只有一张卡load了，很奇怪，谢谢

bsakaguchi commented

2026-10-26 21:02:19 +00:00

Author

Owner

@mklf commented on GitHub (Apr 12, 2024):

请问vllm部署时如何使用多卡加载模型，使用CUDA_VISIBLE_DEVICES=0,1还是只有一张卡load了，很奇怪，谢谢

try add --tp=2 to launch argument

@mklf commented on GitHub (Apr 12, 2024): > 请问vllm部署时如何使用多卡加载模型，使用`CUDA_VISIBLE_DEVICES=0,1`还是只有一张卡load了，很奇怪，谢谢 try add `--tp=2` to launch argument

bsakaguchi commented

2026-10-26 21:02:19 +00:00

Author

Owner

@hyperbolic-c commented on GitHub (Apr 17, 2024):

请问vllm部署时如何使用多卡加载模型，使用CUDA_VISIBLE_DEVICES=0,1还是只有一张卡load了，很奇怪，谢谢

try add --tp=2 to launch argument

thanks, I have solved it by set --tensor-parallel-size >1.

@hyperbolic-c commented on GitHub (Apr 17, 2024): > > 请问vllm部署时如何使用多卡加载模型，使用`CUDA_VISIBLE_DEVICES=0,1`还是只有一张卡load了，很奇怪，谢谢 > > try add `--tp=2` to launch argument thanks, I have solved it by set `--tensor-parallel-size` >1.