deepseek-ai/DeepGEMM
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepGEMM.git synced 2026-11-05 00:47:26 +00:00
Code Issues 162 Projects Packages Wiki Activity

[GH-ISSUE #118] Low E2E throughput performance on SM100 #68

New issue
Closed
opened 2026-10-26 21:11:46 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-26 21:11:46 +00:00
Owner
Copy link

Originally created by @yewentao256 on GitHub (Jun 22, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/118

Hi DeepGemm team,

https://github.com/vllm-project/vllm/pull/19820

Here I integrate the new version of DeepGemm On SM100

The performance test on kernel looks good, but for the e2e throughput performance, it look quite slow compared to triton.

VLLM_USE_DEEP_GEMM=1 vllm bench throughput  --model Qwen/Qwen3-30B-A3B-FP8 --load-format dummy --input-len 1000 --output-len 100 --trust_remote_code --enforce-eager --enable-expert-parallel --quantization fp8
Throughput: 26.34 requests/s, 28916.28 total tokens/s, 2634.07 output tokens/s
vllm bench throughput  --model Qwen/Qwen3-30B-A3B-FP8 --load-format dummy --input-len 1000 --output-len 100 --trust_remote_code --enforce-eager --enable-expert-parallel --quantization fp8
Throughput: 36.65 requests/s, 40270.79 total tokens/s, 3665.06 output tokens/s


VLLM_USE_DEEP_GEMM=1 vllm bench throughput  --model deepseek-ai/DeepSeek-R1 --load-format dummy --input-len 32 --output-len 128 --trust_remote_code --enforce-eager -tp 8 --enable-expert-parallel  --no-enable-prefix-caching
Throughput: 23.89 requests/s, 3821.89 total tokens/s, 3058.29 output tokens/s
# NO deepgemm
Throughput: 42.59 requests/s, 6811.08 total tokens/s, 5451.01 output tokens/s

Do you know why this would happen and how could we solve it?

Originally created by @yewentao256 on GitHub (Jun 22, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/118 Hi DeepGemm team, https://github.com/vllm-project/vllm/pull/19820 Here I integrate the new version of DeepGemm On SM100 The performance test on kernel looks good, but for the e2e throughput performance, it look quite slow compared to triton. ```bash VLLM_USE_DEEP_GEMM=1 vllm bench throughput --model Qwen/Qwen3-30B-A3B-FP8 --load-format dummy --input-len 1000 --output-len 100 --trust_remote_code --enforce-eager --enable-expert-parallel --quantization fp8 Throughput: 26.34 requests/s, 28916.28 total tokens/s, 2634.07 output tokens/s vllm bench throughput --model Qwen/Qwen3-30B-A3B-FP8 --load-format dummy --input-len 1000 --output-len 100 --trust_remote_code --enforce-eager --enable-expert-parallel --quantization fp8 Throughput: 36.65 requests/s, 40270.79 total tokens/s, 3665.06 output tokens/s VLLM_USE_DEEP_GEMM=1 vllm bench throughput --model deepseek-ai/DeepSeek-R1 --load-format dummy --input-len 32 --output-len 128 --trust_remote_code --enforce-eager -tp 8 --enable-expert-parallel --no-enable-prefix-caching Throughput: 23.89 requests/s, 3821.89 total tokens/s, 3058.29 output tokens/s # NO deepgemm Throughput: 42.59 requests/s, 6811.08 total tokens/s, 5451.01 output tokens/s ``` Do you know why this would happen and how could we solve it?
bsakaguchi commented 2026-10-26 21:11:47 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Jun 23, 2025):

The main branch currently does not support archs other than SM90. For SM100, #112 supports SM100 but it is maintained by NVIDIA and still in active development/optimization.

<!-- gh-comment-id:2994639088 --> @LyricZhao commented on GitHub (Jun 23, 2025): The main branch currently does not support archs other than SM90. For SM100, #112 supports SM100 but it is maintained by NVIDIA and still in active development/optimization.
bsakaguchi commented 2026-10-26 21:11:48 +00:00
Author
Owner
Copy link

@yewentao256 commented on GitHub (Jun 23, 2025):

The main branch currently does not support archs other than SM90. For SM100, #112 supports SM100 but it is maintained by NVIDIA and still in active development/optimization.

Thanks for the reply, I actually used this branch for SM100. Let's keep this issue and see whether they can give us some feedback or if there are something we can do to optimize it further.

<!-- gh-comment-id:2996600742 --> @yewentao256 commented on GitHub (Jun 23, 2025): > The main branch currently does not support archs other than SM90. For SM100, [#112](https://github.com/deepseek-ai/DeepGEMM/pull/112) supports SM100 but it is maintained by NVIDIA and still in active development/optimization. Thanks for the reply, I actually used this branch for SM100. Let's keep this issue and see whether they can give us some feedback or if there are something we can do to optimize it further.
bsakaguchi commented 2026-10-26 21:11:48 +00:00
Author
Owner
Copy link

@yewentao256 commented on GitHub (Jun 23, 2025):

@LyricZhao Hi, Tested on H100 with older version of DeepGemm, seems even worse

# Deepgemm for Qwen3
Throughput: 71.48 requests/s, 11429.83 total tokens/s, 9149.81 output tokens/s
# No deepgemm
Throughput: 142.95 requests/s, 22854.46 total tokens/s, 18297.18 output tokens/s
<!-- gh-comment-id:2997290697 --> @yewentao256 commented on GitHub (Jun 23, 2025): @LyricZhao Hi, Tested on H100 with older version of DeepGemm, seems even worse ```bash # Deepgemm for Qwen3 Throughput: 71.48 requests/s, 11429.83 total tokens/s, 9149.81 output tokens/s # No deepgemm Throughput: 142.95 requests/s, 22854.46 total tokens/s, 18297.18 output tokens/s ```
bsakaguchi commented 2026-10-26 21:11:48 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Jun 24, 2025):

The older version does not support SM100.

<!-- gh-comment-id:2998442264 --> @LyricZhao commented on GitHub (Jun 24, 2025): The older version does not support SM100.
bsakaguchi commented 2026-10-26 21:11:49 +00:00
Author
Owner
Copy link

@yewentao256 commented on GitHub (Jun 24, 2025):

The older version does not support SM100.

Yes, I tested the older version on H100 (SM90).

I've found it is a known issue that a lot of people are working on. Like https://github.com/vllm-project/vllm/pull/17934 or https://github.com/neuralmagic/vllm/tree/varun/ht-optimizations, so it has nothing to do with my integration of new DeepGemm.

We temporally don't need to discuss the E2E performance here. I will have other issues or PRs for that if needed in the future

<!-- gh-comment-id:3000375282 --> @yewentao256 commented on GitHub (Jun 24, 2025): > The older version does not support SM100. Yes, I tested the older version on H100 (SM90). I've found it is a known issue that a lot of people are working on. Like https://github.com/vllm-project/vllm/pull/17934 or https://github.com/neuralmagic/vllm/tree/varun/ht-optimizations, so it has nothing to do with my integration of new DeepGemm. We temporally don't need to discuss the E2E performance here. I will have other issues or PRs for that if needed in the future
bsakaguchi commented 2026-10-26 21:11:49 +00:00
Author
Owner
Copy link

@yewentao256 commented on GitHub (Jul 17, 2025):

Close this issue as fixed

<!-- gh-comment-id:3084732903 --> @yewentao256 commented on GitHub (Jul 17, 2025): Close this issue as fixed
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
public-sync-20261014
0930-release
nv_dev
fix-mega-moe-fence
public-release-2609
public-release-2607
public-release-2606
mega-update
public-release-260416
ko3n1g/ci/build-wheels
nv_dev_f8e8fb5
nv_dev_950d31a
nv_dev_67fc648
nv_dev_c491439
nv_dev_4ff3f54
v2.1.1.post3
v2.1.1.post2
v2.1.1.post1
v2.1.1
v2.1.0
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepGEMM#68
No description provided.