[PR #341] [MERGED] Support nvfp4 low latency mode dispatch #526

New issue

Closed

opened 2026-10-27 21:03:41 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:03:41 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/341
Author: @shifangx
Created: 7/30/2025
Status: ✅ Merged
Merged: 10/23/2025
Merged by: @jershi425

Base: hybrid-ep ← Head: shifang/ll_dispatch_nvfp4

📝 Commits (10+)

2bf764c support NVFP4 data format in low latency dispatch
d320aaa add support fp32_vec_to_e2m1 for CUDA_ARCH less than 1000
d88e77e change threshold for diff
3a28b71 add debug message
2add019 change physical layout to be (l, m/128, k/4, 32, 4, 4)
9d9e395 use global scale for entire dispatch instead of per token scale
ccf4eaf change test case
82147f2 change some names and dtype:
fc15ca6 support padding m
d89a25b calibrate nvfp4 scale layout with grouped gemm

📊 Changes

8 files changed (+745 additions, -45 deletions)

View changed files

📝 csrc/deep_ep.cpp (+36 -2)
📝 csrc/deep_ep.hpp (+2 -0)
📝 csrc/kernels/api.cuh (+2 -0)
📝 csrc/kernels/internode_ll.cu (+232 -14)
📝 deep_ep/buffer.py (+17 -3)
📝 tests/test_low_latency.py (+134 -21)
➕ tests/test_low_latency_nvfp4.py (+206 -0)
📝 tests/utils.py (+116 -5)

📄 Description

This MR support nvfp4 low latency mode dispatch.
In the dispatch kernel, we first convert input to nvfp4, and then send to other ranks.
So it's a fused (quantize + dispatch) kernel, and the global_scaling_factor for nvfp4 is a scalar across all experts.

test_log.md

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/341 **Author:** [@shifangx](https://github.com/shifangx) **Created:** 7/30/2025 **Status:** ✅ Merged **Merged:** 10/23/2025 **Merged by:** [@jershi425](https://github.com/jershi425) **Base:** `hybrid-ep` ← **Head:** `shifang/ll_dispatch_nvfp4` --- ### 📝 Commits (10+) - [`2bf764c`](https://github.com/deepseek-ai/DeepEP/commit/2bf764cfd442b2423a13d423bd72135a03187896) support NVFP4 data format in low latency dispatch - [`d320aaa`](https://github.com/deepseek-ai/DeepEP/commit/d320aaa09783090923837571be009388bfdcd1ab) add support fp32_vec_to_e2m1 for __CUDA_ARCH__ less than 1000 - [`d88e77e`](https://github.com/deepseek-ai/DeepEP/commit/d88e77e03fb345e6b27808bb63067e0711b821a6) change threshold for diff - [`3a28b71`](https://github.com/deepseek-ai/DeepEP/commit/3a28b71ad5a0e46542d33ad7e644029a97a34d4a) add debug message - [`2add019`](https://github.com/deepseek-ai/DeepEP/commit/2add0199bde30d24e77bd7ce1215fecf256ce876) change physical layout to be (l, m/128, k/4, 32, 4, 4) - [`9d9e395`](https://github.com/deepseek-ai/DeepEP/commit/9d9e395cafeb7eae20c444d29aa8f575c1fe72ba) use global scale for entire dispatch instead of per token scale - [`ccf4eaf`](https://github.com/deepseek-ai/DeepEP/commit/ccf4eaf24ff133678ee6292cb26b76d363d7ae26) change test case - [`82147f2`](https://github.com/deepseek-ai/DeepEP/commit/82147f21fe41e8ed0641a047f3eb8758f1fc4c11) change some names and dtype: - [`fc15ca6`](https://github.com/deepseek-ai/DeepEP/commit/fc15ca6717aae4391f69aed60eb86010857c8baa) support padding m - [`d89a25b`](https://github.com/deepseek-ai/DeepEP/commit/d89a25b25cc09cb7a557287cba16002aff851b7b) calibrate nvfp4 scale layout with grouped gemm ### 📊 Changes **8 files changed** (+745 additions, -45 deletions) <details> <summary>View changed files</summary> 📝 `csrc/deep_ep.cpp` (+36 -2) 📝 `csrc/deep_ep.hpp` (+2 -0) 📝 `csrc/kernels/api.cuh` (+2 -0) 📝 `csrc/kernels/internode_ll.cu` (+232 -14) 📝 `deep_ep/buffer.py` (+17 -3) 📝 `tests/test_low_latency.py` (+134 -21) ➕ `tests/test_low_latency_nvfp4.py` (+206 -0) 📝 `tests/utils.py` (+116 -5) </details> ### 📄 Description This MR support nvfp4 low latency mode dispatch. In the dispatch kernel, we first convert input to nvfp4, and then send to other ranks. So it's a fused (quantize + dispatch) kernel, and the global_scaling_factor for nvfp4 is a scalar across all experts. [test_log.md](https://github.com/user-attachments/files/23051880/test_log.md) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:03:41 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#526

No description provided.

Rows
Columns