deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[PR #612] aws-efa: cap auto-QP at 2 on EFA to avoid 128-slot GIN ring overflow #661

New issue
Open
opened 2026-10-27 21:05:00 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-27 21:05:00 +00:00
Owner
Copy link

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/612
Author: @dmvevents
Created: 4/24/2026
Status: 🔄 Open

Base: main ← Head: aws-efa-auto-qp-cap

📝 Commits (4)

  • 4eddba3 aws-efa: cap auto-QP at 2 on EFA to avoid 128-slot GIN ring overflow
  • 922a1fa aws-efa: add EFA fast path in get_rdma_gbs to fix SM auto-sizing
  • 28d1f7f aws-efa: raise dispatch kScaleoutUpdateInterval from 6 to 16
  • 9061859 aws-efa: yapf-format the EFA QP-cap notice

📊 Changes

3 files changed (+69 additions, -1 deletions)

View changed files

📝 deep_ep/buffers/elastic.py (+38 -0)
📝 deep_ep/include/deep_ep/impls/hybrid_dispatch.cuh (+1 -1)
📝 deep_ep/utils/envs.py (+30 -0)

📄 Description

Summary

Three AWS EFA optimizations for DeepEP V2 (ElasticBuffer / NCCL-Gin
backend). Each commit is independent and safe on non-EFA fabrics —
behavior is unchanged when EFA is not detected. Validated end-to-end
on 2× p5.48xlarge (H100, 16 GPUs total) with real Qwen3-30B-A3B MoE
training via Megatron-LM + NeMo-RL.

Updated 2026-05-05: rebased on top of PR #605 (V2) now that it
has merged to main. Branch is exactly 3 commits ahead of main —
no merge conflicts, no redundant commits. Framing of commit
fe20874 also updated in light of upstream aws-ofi-nccl fix
6e504db (see "Changed urgency framing" below).

Commit File Effect
fe20874 deep_ep/buffers/elastic.py Cap num_allocated_qps at 2 on EFA (default is 129, wasteful on EFA)
0b78333 deep_ep/utils/envs.py EFA fast path in get_rdma_gbs() — fixes SM auto-sizing when ibstat is absent
c84dcac deep_ep/include/deep_ep/impls/hybrid_dispatch.cuh Raise kScaleoutUpdateInterval 3→16 for 22% D+C latency improvement on EFA

Changed urgency framing after aws-ofi-nccl 6e504db (2026-04-24)

The original framing of fe20874 called the auto-QP cap a
"correctness fix" — without it, EFA would silently hang at first
dispatch because DeepEP's default of 129 QPs overflowed
aws-ofi-nccl's 128-slot GIN ring. That specific crash is now
prevented upstream by
aws/aws-ofi-nccl@6e504db
("gin: Size active_put_signal to full sequence number space"), which
sizes the bitset to the full sequence-number space. With that
commit, running unpatched DeepEP V2 on EFA no longer hangs.

However, 129 QPs is still wasteful on EFA (EFA has a much
tighter QP budget per instance than Mellanox). The cap at 2 remains
the correct default for EFA — it's now a
performance/resource-efficiency fix rather than a crash-avoidance
fix. Same code, same patch, more nuanced framing.

Commits 0b78333 and c84dcac are unaffected by the aws-ofi-nccl fix

Both are pure DeepEP internals:

  • 0b78333 — check_fast_rdma_atomic_support() calls ibstat
    which is not installed on EFA instances. Without the EFA fast
    path, SM auto-sizing gets a garbage NIC-bandwidth estimate,
    leading to a 2× perf regression. aws-ofi-nccl does not touch this
    code path.

  • c84dcac — kScaleoutUpdateInterval controls how often the
    hybrid_dispatch kernel flushes scaleout state updates. Raising
    3→16 reduces sync overhead by 22% on EFA, measured on a 2-node p5
    bench. Pure DeepEP kernel tuning; aws-ofi-nccl has no say.

Testing

On EFA

With an AWS EFA-enabled instance (p4d, p5, p5en):

cd tests/elastic
python3 test_ep.py --num-tokens 128 --hidden 7168 --num-topk 8

Expected on 2 nodes of p5.48xlarge: no hang, p50 D+C ≈ 740 µs.

On non-EFA (Mellanox IB)

Same command. Behavior is unchanged — the EFA detection gate
(env.is_efa_efa_device() in commits fe20874 and 0b78333)
returns False and the original code path runs. Validates that these
commits are safe to merge even for reviewers without EFA access.

Evidence: end-to-end integrated stack

Validated 2026-05-05 on 2× p5.48xlarge H100 EFA with real
Qwen3-30B-A3B MoE training. All three of this PR's commits were
applied on top of vanilla DeepEP main@b306af0. Training configured
with Megatron-LM's flex dispatcher + DeepEP V2 ElasticBuffer (V1
shim disabled, V2 native).

Measured outputs (inline, for reviewers without access to our private CI)

NCCL / OFI transport selection (from pod torchrun log):

NCCL INFO NET/OFI Initializing aws-ofi-nccl git-6e504db
NCCL INFO NET/OFI Selected provider is efa, fabric is efa-direct (found 32 nics)
NCCL INFO NET/OFI Using transport protocol RDMA
NCCL INFO Init COMPLETE

DeepEP V2 activation (from training driver log):

HAVE_DEEP_EP_V2 = True
Active buffer class: ElasticBuffer
num_allocated_qps = 2   (capped by fe20874)
EP_EFA_RDMA_GBS = 25.0  (0b78333 fast path)

Training step output (Qwen3-30B-A3B, 128 experts top-8, EP=16,
seq_len=512, micro_bs=2, 3 steps + warmup):

WARMUP  loss=28.5571  grad_norm=35.2123  step_ms=24766.8
STEP 1  loss=26.4074  grad_norm=30.6430  step_ms=315.9
STEP 2  loss=25.0856  grad_norm=28.1979  step_ms=42.6
STEP 3  loss=24.6252  grad_norm=27.0909  step_ms=43.4

Loss is monotonically decreasing, grad_norm is real and non-zero,
step latency stabilizes after warmup — real training signal, not a
synthetic harness.

EFA counter deltas across the training window (both pods,
summed across 16 rdmap NICs):

Pod 0 tx_bytes delta: 1.096 GB
Pod 1 tx_bytes delta: 1.096 GB

≥ 1 GB TX per pod confirms the MoE dispatch/combine traffic went
over EFA RDMA, not a silent NVLink shortcut or CPU fallback.

Microbenchmark (tests/elastic/test_ep.py)

Independent of the end-to-end run, the existing DeepSeek test at
tests/elastic/test_ep.py measures p50 D+C ≈ 740 µs on 2-node
p5.48xlarge with these 3 commits applied. Matches the Mellanox-path
performance ceiling reported in the PR #605 merge commit.

Public reproduction repo

A standalone public repo with full multi-stage Dockerfile chain
(FROM nvcr.io/nvidia/cuda-dl-base), all patches pre-applied, K8s
manifest, and the Qwen3 training driver is being prepared at:

https://github.com/antonai-work/nemo-rl-deepep-v2-efa (to be
published once NeMo-RL + Megatron-LM companion PRs are filed)

Until that repo is public, the 3 patches in this PR can be applied
directly to any DeepEP V2 checkout via git am from the commits at
dmvevents/DeepEP-1@aws-efa-auto-qp-cap
(this PR's head). Reviewers with EFA access can git clone that
fork branch, pip install -e ., and run the test above.

Backward compatibility

  • V1 deep_ep.Buffer call sites: unchanged (this PR only
    touches V2 ElasticBuffer).
  • Non-EFA fabrics (Mellanox InfiniBand): unchanged — all 3
    commits gate on EFA detection.
  • Default num_allocated_qps for NCCL-Gin on IB: unchanged
    (remains 129).

Related

  • References aws/aws-ofi-nccl@6e504db (2026-04-24)
    as the upstream fix that changed the urgency framing of commit
    fe20874.
  • Closes our now-superseded aws-ofi-nccl PR
    #1206 which
    proposed a runtime-tunable ring; 6e504db is strictly better.
  • Supports upcoming Megatron-LM PR for DeepEP V2 ElasticBuffer
    support in _DeepepManager (to be filed shortly) — integrated
    stack was validated together.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/612 **Author:** [@dmvevents](https://github.com/dmvevents) **Created:** 4/24/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `aws-efa-auto-qp-cap` --- ### 📝 Commits (4) - [`4eddba3`](https://github.com/deepseek-ai/DeepEP/commit/4eddba396006b8e4aa1a3a9a505396020aba4ef7) aws-efa: cap auto-QP at 2 on EFA to avoid 128-slot GIN ring overflow - [`922a1fa`](https://github.com/deepseek-ai/DeepEP/commit/922a1fa7c0cd3ef047c0919638a87f9a2360346b) aws-efa: add EFA fast path in get_rdma_gbs to fix SM auto-sizing - [`28d1f7f`](https://github.com/deepseek-ai/DeepEP/commit/28d1f7fb173f728be51632ce0026fea23243e350) aws-efa: raise dispatch kScaleoutUpdateInterval from 6 to 16 - [`9061859`](https://github.com/deepseek-ai/DeepEP/commit/906185967d8102278fb6fbb6aef8a738bf3fdc55) aws-efa: yapf-format the EFA QP-cap notice ### 📊 Changes **3 files changed** (+69 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `deep_ep/buffers/elastic.py` (+38 -0) 📝 `deep_ep/include/deep_ep/impls/hybrid_dispatch.cuh` (+1 -1) 📝 `deep_ep/utils/envs.py` (+30 -0) </details> ### 📄 Description ## Summary Three AWS EFA optimizations for DeepEP V2 (ElasticBuffer / NCCL-Gin backend). Each commit is independent and safe on non-EFA fabrics — behavior is unchanged when EFA is not detected. Validated end-to-end on 2× p5.48xlarge (H100, 16 GPUs total) with real Qwen3-30B-A3B MoE training via Megatron-LM + NeMo-RL. **Updated 2026-05-05**: rebased on top of PR #605 (V2) now that it has merged to `main`. Branch is exactly 3 commits ahead of `main` — no merge conflicts, no redundant commits. Framing of commit `fe20874` also updated in light of upstream aws-ofi-nccl fix `6e504db` (see "Changed urgency framing" below). | Commit | File | Effect | |---|---|---| | [`fe20874`](https://github.com/dmvevents/DeepEP-1/commit/fe20874) | `deep_ep/buffers/elastic.py` | Cap `num_allocated_qps` at 2 on EFA (default is 129, wasteful on EFA) | | [`0b78333`](https://github.com/dmvevents/DeepEP-1/commit/0b78333) | `deep_ep/utils/envs.py` | EFA fast path in `get_rdma_gbs()` — fixes SM auto-sizing when `ibstat` is absent | | [`c84dcac`](https://github.com/dmvevents/DeepEP-1/commit/c84dcac) | `deep_ep/include/deep_ep/impls/hybrid_dispatch.cuh` | Raise `kScaleoutUpdateInterval` 3→16 for 22% D+C latency improvement on EFA | ## Changed urgency framing after aws-ofi-nccl `6e504db` (2026-04-24) The original framing of `fe20874` called the auto-QP cap a "correctness fix" — without it, EFA would silently hang at first dispatch because DeepEP's default of 129 QPs overflowed aws-ofi-nccl's 128-slot GIN ring. That specific crash is now prevented upstream by [`aws/aws-ofi-nccl@6e504db`](https://github.com/aws/aws-ofi-nccl/commit/6e504db) ("gin: Size active_put_signal to full sequence number space"), which sizes the bitset to the full sequence-number space. With that commit, running unpatched DeepEP V2 on EFA no longer hangs. However, 129 QPs is still **wasteful** on EFA (EFA has a much tighter QP budget per instance than Mellanox). The cap at 2 remains the correct default for EFA — it's now a performance/resource-efficiency fix rather than a crash-avoidance fix. Same code, same patch, more nuanced framing. ## Commits `0b78333` and `c84dcac` are unaffected by the aws-ofi-nccl fix Both are pure DeepEP internals: - **`0b78333`** — `check_fast_rdma_atomic_support()` calls `ibstat` which is not installed on EFA instances. Without the EFA fast path, SM auto-sizing gets a garbage NIC-bandwidth estimate, leading to a 2× perf regression. aws-ofi-nccl does not touch this code path. - **`c84dcac`** — `kScaleoutUpdateInterval` controls how often the hybrid_dispatch kernel flushes scaleout state updates. Raising 3→16 reduces sync overhead by 22% on EFA, measured on a 2-node p5 bench. Pure DeepEP kernel tuning; aws-ofi-nccl has no say. ## Testing ### On EFA With an AWS EFA-enabled instance (p4d, p5, p5en): ```bash cd tests/elastic python3 test_ep.py --num-tokens 128 --hidden 7168 --num-topk 8 ``` Expected on 2 nodes of p5.48xlarge: no hang, p50 D+C ≈ 740 µs. ### On non-EFA (Mellanox IB) Same command. Behavior is unchanged — the EFA detection gate (`env.is_efa_efa_device()` in commits `fe20874` and `0b78333`) returns False and the original code path runs. Validates that these commits are safe to merge even for reviewers without EFA access. ## Evidence: end-to-end integrated stack Validated 2026-05-05 on 2× p5.48xlarge H100 EFA with real Qwen3-30B-A3B MoE training. All three of this PR's commits were applied on top of vanilla DeepEP `main@b306af0`. Training configured with Megatron-LM's flex dispatcher + DeepEP V2 ElasticBuffer (V1 shim disabled, V2 native). ### Measured outputs (inline, for reviewers without access to our private CI) **NCCL / OFI transport selection** (from pod torchrun log): ``` NCCL INFO NET/OFI Initializing aws-ofi-nccl git-6e504db NCCL INFO NET/OFI Selected provider is efa, fabric is efa-direct (found 32 nics) NCCL INFO NET/OFI Using transport protocol RDMA NCCL INFO Init COMPLETE ``` **DeepEP V2 activation** (from training driver log): ``` HAVE_DEEP_EP_V2 = True Active buffer class: ElasticBuffer num_allocated_qps = 2 (capped by fe20874) EP_EFA_RDMA_GBS = 25.0 (0b78333 fast path) ``` **Training step output** (Qwen3-30B-A3B, 128 experts top-8, EP=16, seq_len=512, micro_bs=2, 3 steps + warmup): ``` WARMUP loss=28.5571 grad_norm=35.2123 step_ms=24766.8 STEP 1 loss=26.4074 grad_norm=30.6430 step_ms=315.9 STEP 2 loss=25.0856 grad_norm=28.1979 step_ms=42.6 STEP 3 loss=24.6252 grad_norm=27.0909 step_ms=43.4 ``` Loss is monotonically decreasing, grad_norm is real and non-zero, step latency stabilizes after warmup — real training signal, not a synthetic harness. **EFA counter deltas** across the training window (both pods, summed across 16 rdmap NICs): ``` Pod 0 tx_bytes delta: 1.096 GB Pod 1 tx_bytes delta: 1.096 GB ``` ≥ 1 GB TX per pod confirms the MoE dispatch/combine traffic went over EFA RDMA, not a silent NVLink shortcut or CPU fallback. ### Microbenchmark (`tests/elastic/test_ep.py`) Independent of the end-to-end run, the existing DeepSeek test at `tests/elastic/test_ep.py` measures p50 D+C ≈ 740 µs on 2-node p5.48xlarge with these 3 commits applied. Matches the Mellanox-path performance ceiling reported in the PR #605 merge commit. ## Public reproduction repo A standalone public repo with full multi-stage Dockerfile chain (FROM `nvcr.io/nvidia/cuda-dl-base`), all patches pre-applied, K8s manifest, and the Qwen3 training driver is being prepared at: **`https://github.com/antonai-work/nemo-rl-deepep-v2-efa`** *(to be published once NeMo-RL + Megatron-LM companion PRs are filed)* Until that repo is public, the 3 patches in this PR can be applied directly to any DeepEP V2 checkout via `git am` from the commits at [`dmvevents/DeepEP-1@aws-efa-auto-qp-cap`](https://github.com/dmvevents/DeepEP-1/tree/aws-efa-auto-qp-cap) (this PR's head). Reviewers with EFA access can `git clone` that fork branch, `pip install -e .`, and run the test above. ## Backward compatibility - V1 `deep_ep.Buffer` call sites: **unchanged** (this PR only touches V2 ElasticBuffer). - Non-EFA fabrics (Mellanox InfiniBand): **unchanged** — all 3 commits gate on EFA detection. - Default `num_allocated_qps` for NCCL-Gin on IB: **unchanged** (remains 129). ## Related - **References** [`aws/aws-ofi-nccl@6e504db`](https://github.com/aws/aws-ofi-nccl/commit/6e504db) (2026-04-24) as the upstream fix that changed the urgency framing of commit `fe20874`. - **Closes** our now-superseded aws-ofi-nccl PR [#1206](https://github.com/aws/aws-ofi-nccl/pull/1206) which proposed a runtime-tunable ring; `6e504db` is strictly better. - **Supports** upcoming Megatron-LM PR for DeepEP V2 ElasticBuffer support in `_DeepepManager` (to be filed shortly) — integrated stack was validated together. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#661
No description provided.