[GH-ISSUE #5] GPU utils is low compared with dense model #6

New issue

Closed

opened 2026-10-26 21:01:32 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 21:01:32 +00:00

Owner

Originally created by @charliedream1 on GitHub (Jan 12, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-MoE/issues/5

training time is longer than 14B model. GPU utils is low, it always drop to 0 and raise up again to 100%.

Originally created by @charliedream1 on GitHub (Jan 12, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-MoE/issues/5 training time is longer than 14B model. GPU utils is low, it always drop to 0 and raise up again to 100%.

bsakaguchi closed this issue

2026-10-26 21:01:32 +00:00

bsakaguchi commented

2026-10-26 21:01:33 +00:00

Author

Owner

@zwd003 commented on GitHub (Jan 12, 2024):

currently，this code has not been extensively optimized for efficient training(e.g. fused moe kernel or expert parallelism). When resources are sufficient, it is recommended that you use zero1 to reduce communication overhead.

@zwd003 commented on GitHub (Jan 12, 2024): currently，this code has not been extensively optimized for efficient training(e.g. fused moe kernel or expert parallelism). When resources are sufficient, it is recommended that you use zero1 to reduce communication overhead.

bsakaguchi commented

2026-10-26 21:01:33 +00:00

Author

Owner

@charliedream1 commented on GitHub (Jan 12, 2024):

thanks for reply. Hopefully, these optimization will be added soon

@charliedream1 commented on GitHub (Jan 12, 2024): thanks for reply. Hopefully, these optimization will be added soon

bsakaguchi commented

2026-10-26 21:01:33 +00:00

Author

Owner

@charliedream1 commented on GitHub (Jan 14, 2024):

So what would be the reason of much slower by equally compared with 14B model without optimizations? Activation params is only 2.7B，however, training is much slower than 14B model, even slower than 20B model. Is that because too many experts calculated in a pipeline but not parallel?

---Original---
From: "wangding @.>
Date: Fri, Jan 12, 2024 15:08 PM
To: @.>;
Cc: "Optimus @.@.>;
Subject: Re: [deepseek-ai/DeepSeek-MoE] GPU utils is low compared with densemodel (Issue #5)

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

@charliedream1 commented on GitHub (Jan 14, 2024): So what would be the reason of much slower by equally compared with 14B model without optimizations? Activation params is only 2.7B，however, training is much slower than 14B model, even slower than 20B model. Is that because too many experts calculated in a pipeline but not parallel? ---Original--- From: "wangding ***@***.***> Date: Fri, Jan 12, 2024 15:08 PM To: ***@***.***>; Cc: "Optimus ***@***.******@***.***>; Subject: Re: [deepseek-ai/DeepSeek-MoE] GPU utils is low compared with densemodel (Issue #5) currently，this code has not been extensively optimized for efficient training(e.g. fused moe kernel or expert parallelism). When resources are sufficient, it is recommended that you use zero1 to reduce communication overhead. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

bsakaguchi commented

2026-10-26 21:01:33 +00:00

Author

Owner

@soloice commented on GitHub (Feb 4, 2024):

So what would be the reason of much slower by equally compared with 14B model without optimizations? Activation params is only 2.7B，however, training is much slower than 14B model, even slower than 20B model. Is that because too many experts calculated in a pipeline but not parallel?

too many fragmented matmul and network communication

@soloice commented on GitHub (Feb 4, 2024): > So what would be the reason of much slower by equally compared with 14B model without optimizations? Activation params is only 2.7B，however, training is much slower than 14B model, even slower than 20B model. Is that because too many experts calculated in a pipeline but not parallel? too many fragmented matmul and network communication

bsakaguchi referenced this issue

2026-10-26 21:02:00 +00:00

[GH-ISSUE #47] CUDA error: unspecified launch failure #42