mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #258] Why does modifying topk_idx affect the delay? #192
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#192
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @nannaer on GitHub (Jun 25, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/258
My configuration is: num_tokens, hidden, num_topk, num_experts = 160, 7168, 8, 256. I changed the generation strategy of topk_idx from random generation to forcibly assigning the same number of tokens to each RANK. However, I found that the dispatch + combine latency increased significantly after the modification.
Here is an example. There is a RANK with 4 experts. The RANK receives 32 tokens in total. The dispatch + combine delay when all 32 tokens are allocated to one expert is much higher than that when the 32 tokens are evenly distributed among the four experts. What's the reason for that?
Origin topk_idx
topk_idx before: tensor([[183, 40, 201, 82, 254, 69, 108, 91],
[213, 89, 134, 218, 93, 181, 99, 144],
[178, 3, 154, 72, 100, 220, 196, 188],
[ 6, 198, 101, 209, 146, 117, 144, 194],
[ 93, 255, 6, 117, 148, 187, 223, 145],
[208, 215, 201, 210, 9, 29, 79, 238],
[228, 76, 103, 188, 143, 7, 79, 204],
[161, 8, 54, 134, 224, 98, 60, 78],
[ 45, 152, 89, 161, 190, 94, 191, 230],
[202, 156, 230, 55, 254, 16, 199, 221],
[ 69, 124, 81, 185, 35, 158, 233, 150],
[121, 213, 222, 192, 2, 206, 176, 156],
[ 65, 169, 12, 0, 44, 197, 61, 142],
[193, 50, 23, 31, 178, 61, 5, 250],
[ 11, 88, 4, 178, 48, 121, 71, 249],
[205, 148, 212, 113, 49, 207, 174, 32],
[185, 195, 18, 29, 216, 26, 48, 249],
[139, 162, 221, 117, 213, 228, 10, 184],
[ 48, 55, 109, 94, 51, 234, 140, 114],
[ 11, 135, 154, 201, 98, 236, 234, 246],
[ 30, 158, 69, 231, 102, 235, 44, 121],
[125, 67, 212, 79, 10, 238, 150, 103],
[175, 110, 224, 38, 92, 132, 15, 173],
[254, 7, 217, 78, 98, 11, 201, 164],
[235, 169, 46, 241, 196, 106, 202, 182],
[254, 140, 145, 132, 171, 87, 88, 125],
[130, 205, 237, 168, 111, 43, 81, 193],
[ 0, 241, 71, 108, 84, 192, 43, 147],
[148, 193, 150, 120, 216, 119, 225, 229],
[201, 153, 61, 231, 150, 236, 196, 248],
[252, 193, 74, 203, 63, 205, 200, 12],
[ 1, 217, 244, 187, 174, 128, 117, 213],
[ 19, 102, 174, 36, 190, 60, 241, 216],
[241, 76, 61, 95, 163, 89, 47, 82],
[246, 36, 232, 58, 27, 191, 16, 108],
[212, 98, 79, 210, 52, 36, 105, 190],
[ 79, 21, 170, 161, 185, 215, 137, 15],
[ 59, 97, 114, 82, 60, 20, 226, 10],
[ 4, 48, 19, 145, 138, 149, 122, 207],
[240, 201, 51, 122, 216, 62, 123, 7],
[ 72, 77, 96, 215, 227, 47, 14, 189],
[127, 14, 28, 141, 198, 105, 129, 62],
[ 58, 187, 214, 21, 177, 86, 78, 66],
[ 95, 30, 194, 195, 18, 17, 239, 122],
[228, 149, 234, 60, 85, 30, 173, 38],
[167, 33, 28, 253, 218, 212, 143, 175],
[ 0, 192, 220, 173, 210, 138, 167, 55],
[203, 117, 63, 175, 150, 32, 246, 86],
[ 63, 254, 95, 84, 175, 131, 28, 13],
[ 33, 3, 137, 166, 53, 169, 226, 164],
[ 32, 195, 159, 246, 66, 220, 88, 229],
[134, 107, 43, 79, 199, 82, 50, 12],
[157, 129, 247, 207, 160, 228, 73, 60],
[100, 201, 163, 225, 63, 135, 82, 132],
[182, 115, 150, 76, 242, 70, 137, 207],
[255, 204, 74, 58, 123, 144, 106, 0],
[138, 107, 255, 116, 36, 168, 12, 230],
[ 35, 100, 151, 129, 186, 123, 125, 60],
[180, 157, 7, 151, 237, 244, 45, 141],
[185, 94, 11, 144, 59, 169, 99, 42],
[181, 235, 245, 155, 248, 72, 134, 57],
[178, 195, 123, 240, 197, 237, 133, 95],
[177, 77, 146, 55, 252, 172, 244, 182],
[157, 138, 142, 89, 6, 180, 79, 91],
[178, 130, 238, 48, 54, 53, 162, 65],
[243, 210, 74, 170, 236, 26, 219, 208],
[ 81, 255, 9, 225, 21, 252, 31, 171],
[108, 75, 41, 186, 217, 39, 23, 255],
[181, 252, 67, 83, 190, 211, 101, 136],
[ 46, 53, 134, 235, 125, 118, 100, 255],
[172, 56, 20, 202, 43, 158, 119, 38],
[ 70, 227, 208, 212, 166, 17, 192, 113],
[216, 88, 190, 95, 137, 4, 138, 53],
[162, 222, 90, 64, 36, 40, 213, 81],
[176, 34, 95, 31, 42, 225, 5, 227],
[108, 185, 50, 232, 101, 102, 4, 133],
[ 84, 146, 158, 45, 234, 189, 138, 112],
[214, 178, 113, 116, 208, 218, 151, 30],
[ 64, 129, 87, 109, 227, 217, 53, 251],
[ 1, 226, 218, 174, 63, 205, 212, 29],
[169, 48, 203, 215, 3, 192, 106, 70],
[219, 28, 217, 58, 198, 197, 46, 183],
[167, 219, 96, 37, 169, 202, 181, 184],
[133, 122, 67, 163, 13, 195, 237, 14],
[ 76, 64, 44, 215, 8, 42, 231, 238],
[ 50, 247, 92, 209, 101, 87, 85, 214],
[182, 32, 238, 235, 215, 195, 13, 59],
[ 2, 155, 67, 85, 166, 109, 35, 48],
[177, 183, 1, 192, 33, 243, 127, 145],
[202, 220, 197, 211, 145, 241, 69, 13],
[252, 39, 16, 235, 13, 208, 60, 34],
[135, 224, 117, 99, 190, 140, 107, 153],
[ 9, 68, 250, 24, 173, 60, 175, 145],
[232, 52, 6, 69, 215, 95, 98, 138],
[ 6, 74, 223, 181, 135, 189, 226, 191],
[103, 42, 249, 218, 226, 87, 190, 224],
[164, 117, 108, 125, 200, 53, 5, 71],
[ 36, 114, 41, 22, 31, 113, 138, 20],
[ 19, 49, 194, 186, 251, 178, 86, 123],
[106, 115, 132, 73, 80, 35, 4, 247],
[254, 144, 54, 138, 46, 215, 26, 203],
[216, 199, 67, 92, 4, 235, 203, 237],
[ 97, 16, 239, 246, 254, 231, 136, 101],
[235, 204, 209, 11, 98, 134, 247, 21],
[ 45, 70, 246, 238, 136, 142, 181, 206],
[170, 25, 93, 105, 188, 129, 231, 218],
[123, 186, 204, 18, 157, 28, 192, 90],
[108, 73, 209, 210, 178, 103, 220, 2],
[105, 29, 240, 112, 193, 44, 136, 236],
[208, 55, 103, 36, 41, 84, 138, 240],
[205, 140, 99, 235, 131, 152, 76, 187],
[201, 95, 190, 100, 15, 96, 178, 240],
[ 99, 182, 209, 5, 205, 169, 178, 243],
[107, 240, 157, 188, 20, 113, 51, 207],
[ 81, 100, 198, 12, 119, 89, 196, 174],
[152, 149, 66, 40, 185, 68, 187, 97],
[226, 56, 89, 141, 215, 129, 175, 254],
[148, 176, 74, 169, 214, 171, 69, 28],
[ 92, 177, 107, 192, 40, 110, 199, 190],
[110, 155, 131, 90, 20, 98, 61, 128],
[169, 253, 100, 36, 254, 193, 239, 198],
[233, 76, 17, 92, 138, 82, 74, 212],
[233, 151, 11, 74, 66, 150, 206, 164],
[ 16, 238, 23, 27, 250, 209, 206, 135],
[ 81, 132, 109, 79, 128, 105, 125, 71],
[ 68, 212, 38, 185, 147, 50, 125, 243],
[196, 72, 30, 247, 26, 234, 162, 55],
[207, 131, 250, 218, 44, 30, 45, 94],
[ 31, 143, 32, 37, 108, 208, 218, 249],
[ 96, 130, 19, 122, 25, 187, 189, 215],
[ 17, 201, 149, 147, 94, 253, 194, 8],
[ 76, 166, 211, 105, 48, 205, 245, 66],
[222, 105, 84, 47, 104, 71, 10, 196],
[245, 183, 161, 101, 112, 200, 192, 148],
[ 13, 168, 164, 57, 96, 58, 27, 155],
[147, 165, 238, 185, 245, 116, 136, 18],
[195, 224, 133, 3, 29, 205, 164, 81],
[ 8, 202, 71, 108, 25, 2, 115, 91],
[241, 163, 232, 196, 101, 233, 23, 133],
[127, 79, 141, 113, 92, 22, 17, 29],
[ 4, 66, 127, 13, 252, 98, 43, 168],
[154, 83, 49, 215, 151, 80, 164, 171],
[166, 146, 60, 79, 148, 36, 187, 32],
[179, 97, 147, 87, 146, 127, 200, 42],
[228, 48, 180, 66, 33, 249, 207, 44],
[228, 59, 48, 204, 255, 43, 62, 156],
[ 4, 188, 9, 202, 105, 165, 59, 85],
[172, 19, 26, 228, 95, 78, 51, 7],
[207, 64, 196, 80, 116, 41, 140, 219],
[135, 223, 154, 18, 56, 224, 45, 105],
[ 83, 218, 82, 115, 58, 132, 84, 213],
[ 82, 220, 67, 161, 43, 202, 239, 121],
[212, 216, 200, 84, 188, 105, 247, 240],
[152, 101, 16, 118, 91, 240, 62, 39],
[ 82, 97, 180, 149, 197, 89, 247, 65],
[ 87, 215, 29, 212, 138, 78, 193, 139],
[ 74, 176, 47, 242, 175, 207, 189, 72],
[223, 147, 218, 74, 117, 204, 106, 232],
[233, 177, 90, 103, 224, 114, 141, 97],
[190, 119, 152, 42, 239, 118, 210, 107]], device='cuda:0')
Mock topk_idx
topk_idx after: tensor([[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248],
[ 0, 8, 16, 24, 32, 40, 48, 56],
[ 64, 72, 80, 88, 96, 104, 112, 120],
[128, 136, 144, 152, 160, 168, 176, 184],
[192, 200, 208, 216, 224, 232, 240, 248]], device='cuda:0')
@LyricZhao commented on GitHub (Jul 2, 2025):
Each expert has a fixed number of warps to process, if more tokens received for one expert, more waves of warps are needed (but I never test the affect on latency). It is by design, as we expect evenly distributed tokens for experts. If this is unbalanced, we should somehow blame the model or load balancer.
@nannaer commented on GitHub (Jul 2, 2025):
Thanks!!!