deepseek-ai/DeepSeek-V2
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-V2.git synced 2026-11-05 00:47:23 +00:00
Code Issues 90 Projects Packages Wiki Activity

[GH-ISSUE #21] Reproduce inference benchmark mentioned in the paper #19

New issue
Open
opened 2026-10-26 21:06:50 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-26 21:06:50 +00:00
Owner
Copy link

Originally created by @zhouheyun on GitHub (May 11, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V2/issues/21

I have a few questions about the inference efficiency of deepseek v2
1.

In order to efficiently deploy DeepSeek-V2 for service, we first convert its parameters into the precision of FP8.

Are all the storage and computation performed in fp8 ? Does this harm the performance of the model?
2.

On a single node with 8 H800 GPUs, DeepSeek-V2 achieves a generation throughput
exceeding 50K tokens per second, which is 5.76 times the maximum generation throughput of
DeepSeek 67B. In addition, the prompt input throughput of DeepSeek-V2 exceeds 100K tokens
per second.

Is this throughput achieved using testing request of 128K context length? Can we reproduce it using https://github.com/vllm-project/vllm/pull/4650

Originally created by @zhouheyun on GitHub (May 11, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V2/issues/21 I have a few questions about the inference efficiency of deepseek v2 1. > In order to efficiently deploy DeepSeek-V2 for service, we first convert its parameters into the precision of FP8. Are all the storage and computation performed in fp8 ? Does this harm the performance of the model? 2. > On a single node with 8 H800 GPUs, DeepSeek-V2 achieves a generation throughput > exceeding 50K tokens per second, which is 5.76 times the maximum generation throughput of > DeepSeek 67B. In addition, the prompt input throughput of DeepSeek-V2 exceeds 100K tokens > per second. Is this throughput achieved using testing request of 128K context length? Can we reproduce it using https://github.com/vllm-project/vllm/pull/4650
bsakaguchi commented 2026-10-26 21:06:50 +00:00
Author
Owner
Copy link

@luofuli commented on GitHub (May 14, 2024):

Our open-source code (https://github.com/vllm-project/vllm/pull/4650) is not the inference code used in the API platform, so it cannot achieve the throughput speed mentioned in the paper. @zhouheyun

<!-- gh-comment-id:2109306013 --> @luofuli commented on GitHub (May 14, 2024): Our open-source code (https://github.com/vllm-project/vllm/pull/4650) is not the inference code used in the API platform, so it cannot achieve the throughput speed mentioned in the paper. @zhouheyun
bsakaguchi commented 2026-10-26 21:06:51 +00:00
Author
Owner
Copy link

@zhouheyun commented on GitHub (May 14, 2024):

Our open-source code (vllm-project/vllm#4650) is not the inference code used in the API platform, so it cannot achieve the throughput speed mentioned in the paper. @zhouheyun

What‘s the average inference context length to achieve the claimed throughput in the paper? @luofuli

<!-- gh-comment-id:2109367386 --> @zhouheyun commented on GitHub (May 14, 2024): > Our open-source code ([vllm-project/vllm#4650](https://github.com/vllm-project/vllm/pull/4650)) is not the inference code used in the API platform, so it cannot achieve the throughput speed mentioned in the paper. @zhouheyun What‘s the average inference context length to achieve the claimed throughput in the paper? @luofuli
bsakaguchi commented 2026-10-26 21:06:51 +00:00
Author
Owner
Copy link

@luofuli commented on GitHub (May 27, 2024):

32K context length @zhouheyun

<!-- gh-comment-id:2133244267 --> @luofuli commented on GitHub (May 27, 2024): 32K context length @zhouheyun
bsakaguchi commented 2026-10-26 21:06:51 +00:00
Author
Owner
Copy link

@ArtificialZeng commented on GitHub (Aug 15, 2024):

vllm-project/vllm#4650

How much tokens/s this open source version can achieve? on 8*H800 ?

<!-- gh-comment-id:2290520249 --> @ArtificialZeng commented on GitHub (Aug 15, 2024): > vllm-project/vllm#4650 How much tokens/s this open source version can achieve? on 8*H800 ?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-V2#19
No description provided.