[PR #731] [CLOSED] feat(low_latency): support top-16 routing and hidden-dim 3584 #736

New issue

Closed

opened 2026-10-27 21:05:45 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:45 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/731
Author: @samsja
Created: 8/18/2026
Status: ❌ Closed

Base: main ← Head: fix/k3-topk16-hidden3584

📝 Commits (1)

6b29c94 feat(low_latency): support top-16 routing and hidden-dim 3584

📊 Changes

2 files changed (+4 additions, -2 deletions)

View changed files

📝 csrc/kernels/legacy/internode_ll.cu (+2 -2)
📝 csrc/kernels/legacy/launch.cuh (+2 -0)

📄 Description

Problem

The low_latency internode dispatch/combine kernels hard-cap top-k at 11 and only
template-specialize on a fixed set of a2a hidden dims. Two MoE configs don't fit
(both hit by Kimi-K3 — 896 experts, top-16, routed-expert hidden 3584):

top-16 routing trips EP_HOST_ASSERT(num_topk <= kNumMaxTopK) (dispatch) and the
kNumMaxTopk equivalent (combine).
routed-expert hidden 3584 falls through SWITCH_HIDDEN's default → "Unsupported
hidden". (This is the a2a hidden dim, not the model hidden_size.)

Change

kNumMaxTopK / kNumMaxTopk 11 → 16 (dispatch + combine).
Add case 3584 to SWITCH_HIDDEN — 3584 satisfies the LL alignment invariants
(%128, %256 = 32·kNumElemsPerRead, %512 for the combine kNumSendUnrolls=2 path).

Follows #458 (hidden-dim 3072).

Validation

mini-K3 (2 nodes, EP16, --deepep-mode auto, cuda-graph ON): decode cuda-graph capture
passes SWITCH_HIDDEN and serves without crash; greedy A/B vs the naive-a2a baseline
agrees token-for-token into the low_latency decode region. Draft pending a full-model
multi-node coherence run.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/731 **Author:** [@samsja](https://github.com/samsja) **Created:** 8/18/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/k3-topk16-hidden3584` --- ### 📝 Commits (1) - [`6b29c94`](https://github.com/deepseek-ai/DeepEP/commit/6b29c949da9ff2e99e0540ce3729be92195cd211) feat(low_latency): support top-16 routing and hidden-dim 3584 ### 📊 Changes **2 files changed** (+4 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/legacy/internode_ll.cu` (+2 -2) 📝 `csrc/kernels/legacy/launch.cuh` (+2 -0) </details> ### 📄 Description ### Problem The low_latency internode dispatch/combine kernels hard-cap top-k at 11 and only template-specialize on a fixed set of a2a hidden dims. Two MoE configs don't fit (both hit by Kimi-K3 — 896 experts, **top-16**, routed-expert **hidden 3584**): - **top-16 routing** trips `EP_HOST_ASSERT(num_topk <= kNumMaxTopK)` (dispatch) and the `kNumMaxTopk` equivalent (combine). - **routed-expert hidden 3584** falls through `SWITCH_HIDDEN`'s default → "Unsupported hidden". (This is the a2a hidden dim, not the model `hidden_size`.) ### Change - `kNumMaxTopK` / `kNumMaxTopk` 11 → 16 (dispatch + combine). - Add `case 3584` to `SWITCH_HIDDEN` — 3584 satisfies the LL alignment invariants (%128, %256 = 32·kNumElemsPerRead, %512 for the combine kNumSendUnrolls=2 path). Follows #458 (hidden-dim 3072). ### Validation mini-K3 (2 nodes, EP16, `--deepep-mode auto`, cuda-graph ON): decode cuda-graph capture passes `SWITCH_HIDDEN` and serves without crash; greedy A/B vs the naive-a2a baseline agrees token-for-token into the low_latency decode region. Draft pending a full-model multi-node coherence run. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:05:45 +00:00