[PR #453] Higher Performance with Lower SM Occupancy through Zero-Copy and TMA Offloading #576

New issue

Open

opened 2026-10-27 21:04:13 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:04:13 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/453
Author: @monethuang1
Created: 10/14/2025
Status: 🔄 Open

Base: main ← Head: trmt-zero-copy

📝 Commits (10+)

be77c85 Init zero-copy internode dispatch/combine
8ffa6f6 Use 1 SM per channel for cached_notify
1b38673 add combine tma alighment and fix rdma receivers num
6dd446b Reduce SMEM usage
79d6ef5 Support multiple buffers for zero-copy
6c37dfa Add assertion against too large recv token count
5e5bd79 Use separate sizes for Dispatch/Combine input buffers
875a5a6 Ensure NVLink fused buffers are properly aligned by reordering NVL buffer segments
9e52f9b Cleanup TMA-related code (1)
c4eef60 Cleanup TMA-related code (2)

📊 Changes

14 files changed (+2187 additions, -155 deletions)

View changed files

📝 csrc/config.hpp (+132 -8)
📝 csrc/deep_ep.cpp (+258 -49)
📝 csrc/deep_ep.hpp (+21 -4)
📝 csrc/kernels/CMakeLists.txt (+2 -1)
📝 csrc/kernels/api.cuh (+54 -5)
📝 csrc/kernels/buffer.cuh (+27 -27)
📝 csrc/kernels/configs.cuh (+16 -0)
📝 csrc/kernels/ibgda_device.cuh (+133 -0)
📝 csrc/kernels/internode.cu (+108 -17)
➕ csrc/kernels/internode_zcopy.cu (+1209 -0)
📝 csrc/kernels/utils.cuh (+53 -14)
📝 deep_ep/buffer.py (+79 -13)
📝 setup.py (+1 -1)
📝 tests/test_internode.py (+94 -16)

📄 Description

The original Internode Normal Kernel suffers from high GPU SM utilization and underutilized interconnect bandwidth, which constrains prefill performance.
In our optimized version, we apply buffer fusion and TMA offloading to enable true zero-copy communication and maximize NVLink bandwidth usage.

Evaluation on H20 clusters shows significant gains:

With EP=16, performance (dispatch(FP8) / dispatch(BF16) / combine) improved from 76.50 / 84.05 / 62.50 to 89.46 / 91.82 / 82.27.
With EP=32, performance increased from 59.95 / 61.33 / 61.24 to 62.53 / 63.24 / 62.55.

Additionally, SM occupancy was reduced by up to 66.7%. The optimized kernel uses only 12 SMs for EP=16 and 8 SMs for EP=32, compared to 24 SMs in the original version.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/453 **Author:** [@monethuang1](https://github.com/monethuang1) **Created:** 10/14/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `trmt-zero-copy` --- ### 📝 Commits (10+) - [`be77c85`](https://github.com/deepseek-ai/DeepEP/commit/be77c85e8b9112258936dd9cb50502ce33d947c8) Init zero-copy internode dispatch/combine - [`8ffa6f6`](https://github.com/deepseek-ai/DeepEP/commit/8ffa6f61db939a23f4337adca8ef3212339a6112) Use 1 SM per channel for cached_notify - [`1b38673`](https://github.com/deepseek-ai/DeepEP/commit/1b38673afdd24b0b5b1a7b3bdaebd9de08a241e4) add combine tma alighment and fix rdma receivers num - [`6dd446b`](https://github.com/deepseek-ai/DeepEP/commit/6dd446b0b195d67d2750f56729a3effffba67684) Reduce SMEM usage - [`79d6ef5`](https://github.com/deepseek-ai/DeepEP/commit/79d6ef5d4200773be1b33bf473d18d433d95b816) Support multiple buffers for zero-copy - [`6c37dfa`](https://github.com/deepseek-ai/DeepEP/commit/6c37dfa7259dd49b5b5a5b36bba7b693dec9cf4b) Add assertion against too large recv token count - [`5e5bd79`](https://github.com/deepseek-ai/DeepEP/commit/5e5bd79a917d7066c2bc430dd07a71403bd7071d) Use separate sizes for Dispatch/Combine input buffers - [`875a5a6`](https://github.com/deepseek-ai/DeepEP/commit/875a5a64ea9de77ab8c04814030816efb4d61dab) Ensure NVLink fused buffers are properly aligned by reordering NVL buffer segments - [`9e52f9b`](https://github.com/deepseek-ai/DeepEP/commit/9e52f9bdf242de2732812c310228ecd24ce50aef) Cleanup TMA-related code (1) - [`c4eef60`](https://github.com/deepseek-ai/DeepEP/commit/c4eef60e7cd71c2a6914c20a44f9ced0aec78a73) Cleanup TMA-related code (2) ### 📊 Changes **14 files changed** (+2187 additions, -155 deletions) <details> <summary>View changed files</summary> 📝 `csrc/config.hpp` (+132 -8) 📝 `csrc/deep_ep.cpp` (+258 -49) 📝 `csrc/deep_ep.hpp` (+21 -4) 📝 `csrc/kernels/CMakeLists.txt` (+2 -1) 📝 `csrc/kernels/api.cuh` (+54 -5) 📝 `csrc/kernels/buffer.cuh` (+27 -27) 📝 `csrc/kernels/configs.cuh` (+16 -0) 📝 `csrc/kernels/ibgda_device.cuh` (+133 -0) 📝 `csrc/kernels/internode.cu` (+108 -17) ➕ `csrc/kernels/internode_zcopy.cu` (+1209 -0) 📝 `csrc/kernels/utils.cuh` (+53 -14) 📝 `deep_ep/buffer.py` (+79 -13) 📝 `setup.py` (+1 -1) 📝 `tests/test_internode.py` (+94 -16) </details> ### 📄 Description The original Internode Normal Kernel suffers from high GPU SM utilization and underutilized interconnect bandwidth, which constrains prefill performance. In our optimized version, we apply buffer fusion and TMA offloading to enable true zero-copy communication and maximize NVLink bandwidth usage. Evaluation on H20 clusters shows significant gains: * With EP=16, performance (dispatch(FP8) / dispatch(BF16) / combine) improved from 76.50 / 84.05 / 62.50 to 89.46 / 91.82 / 82.27. * With EP=32, performance increased from 59.95 / 61.33 / 61.24 to 62.53 / 63.24 / 62.55. Additionally, SM occupancy was reduced by up to 66.7%. The optimized kernel uses only 12 SMs for EP=16 and 8 SMs for EP=32, compared to 24 SMs in the original version. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:04:13 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#576

No description provided.

Rows
Columns