mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #102] In low-latency mode, during the combine phase, does a SM have enough threads to read all token elements? #93
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#93
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @lichangW on GitHub (Mar 31, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/102
As shown in the code, during the combine phase, each Streaming Multiprocessor (SM) has 30 × 32 threads (960 threads total), but each token contains hidden_bf16_int4 =7168/2 = 3584 elements. It appears that:
The assertion check fails (likely due to insufficient threads to cover all elements).
The data received in the combine phase cannot be fully read.
internode_ll.cu line:530~542
`
// Reduce tokens with FP8 cast
EP_DEVICE_ASSERT(num_topk <= 32 and hidden_bf16_int4 <= num_threads);
EP_STATIC_ASSERT(kHidden % (32 * kNumElemsPerInt4) == 0, "Invalid vectorization");
if (thread_id < hidden_bf16_int4) {
for (int token_idx = sm_id; token_idx < num_combined_tokens; token_idx += num_sms) {
// Read top-k indices and weights
`
@LyricZhao commented on GitHub (Apr 1, 2025):
hidden_bf16_int4should be7168 * sizeof(nv_bfloat16) / sizeof(int4) = 7168 * 2 / 16 = 896.int4(16 bytes) means 4int(4 bytes).@lichangW commented on GitHub (Apr 1, 2025):
got it, tks!