mirror of
https://github.com/deepseek-ai/DeepSeek-V3.git
synced 2026-11-05 00:47:48 +00:00
[GH-ISSUE #11] Hello, how many GPU (A800 80G) to deploy DeepSeek-V3 model with vllm? #8
Labels
No labels
closed-as-stale
enhancement
question
spam
stale
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-V3#8
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @13416157913 on GitHub (Dec 27, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/11
Hello, how many GPU (A800 80G) to deploy DeepSeek-V3 model with vllm?
@lebronjamesking commented on GitHub (Dec 27, 2024):
I have same question. I see the documentation says the model is 685B. I'm afraid of the scale.
@stack-heap-overflow commented on GitHub (Dec 27, 2024):
A800 does not support FP8 computation, and we have not tested model inference in BF16 format.
I guessed that deploying a BF16 model would require 4*8 A800s.
@13416157913 commented on GitHub (Dec 27, 2024):
As your meaning, in the A800, I can canvert FP8 weights to BF16 and deploy it by 4*80GB A800s? Right?
@barius commented on GitHub (Dec 27, 2024):
4 * 8 * 80
@Philomath88 commented on GitHub (Dec 29, 2024):
Since you guys seem to have some server experience, could you recommend a home server to use this model (can also be a used one)?
@pizts commented on GitHub (Dec 30, 2024):
Are there any plans to release int4 quantization?
@earmer commented on GitHub (Jan 1, 2025):
Sorry, but I think as for a 300B+ model, deploying it @ home it not a good idea. Running a 300B+ parameter model at home isn't practical due to the massive hardware requirements. You'd need over 1TB of high-speed GPU memory, multiple high-end GPUs, substantial CPU and system RAM, plus enterprise-grade cooling systems.
The power consumption would be excessive for residential use, and the total cost would far exceed using API access. For home server projects, you'd be better off exploring smaller open-source models in the 7B-70B parameter range that can run on consumer hardware, but for large models like this, cloud APIs are the more practical and cost-effective solution--And it is true because Deepseek recently announced a discount on API.
@ciaoyizhen commented on GitHub (Jan 10, 2025):
你好，我看paper里面
prefill: The minimum deployment unit of the prefilling stage consists of 4 nodes with 32 GPUs.
decode: The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs.
所以看不懂了
到底要多少资源？
@Gloria2tt commented on GitHub (Jan 10, 2025):
1 节点=1台机器 = 8张卡
@Philomath88 commented on GitHub (Jan 10, 2025):
@earmer Thank you for your reply! It would be awesome if in the future, one could run a version of deepseek at home using NVIDIA's new project Digits https://www.nvidia.com/en-us/project-digits/
@ciaoyizhen commented on GitHub (Jan 13, 2025):
@Gloria2tt 你好 我的意思是说，这两个prefill和decode用的资源不一样 我不知道以那个为准 然后两个啥区别呀？
@github-actions[bot] commented on GitHub (Feb 13, 2025):
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
@github-actions[bot] commented on GitHub (Feb 27, 2025):
false
@zengqingfu1442 commented on GitHub (Mar 18, 2025):
16*80G=1280G，16个可以跑8bit量化版本的