deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[PR #780] Document and test ordinary sync-free receive tensor lifetimes #776

New issue

Open

opened 2026-10-27 21:06:15 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:06:15 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/780
Author: @yuweih205
Created: 10/15/2026
Status: 🔄 Open

Base: main ← Head: codex/syncfree-recv-lifetime-1015

📝 Commits (1)

97fa6d1 Document and test sync-free receive tensor lifetimes

📊 Changes

2 files changed (+168 additions, -0 deletions)

View changed files

📝 README.md (+2 -0)
➕ tests/ep/test_recv_lifetime.py (+166 -0)

📄 Description

Summary

Document and exercise the lifetime contract of ordinary independently allocated receive outputs from compact dispatch(do_cpu_sync=False).

Add a README paragraph explaining valid receive prefixes, output ownership versus stream readiness, the explicit ready-event/record_stream handoff, and why a prefix view retains the full capacity allocation.
Add a 166-line standalone native regression for saved-view backward after later dispatches and delayed cross-stream consumers under allocator pressure.

No production Python/C++/CUDA implementation, default, or API changes. The internal dispatch copy epilogue still runs. This is not borrowed communication-buffer storage or a zero-copy dispatch implementation.

Regression coverage

The new tests/ep/test_recv_lifetime.py uses the independent NCCL dispatch reference and requires bitwise equality of payloads, local expert indices, routing weights, source-token metadata, and the saved-activation weight gradient.

Each process run covers:

Deterministic dispatch off/on; output allocation on the compute/communication stream.
Uneven inputs, balanced routes, masked routes, and highly skewed routing with empty receivers.
Two selected experts on the same destination rank, distinguishing unique receive rows from token/expert assignments.
A prefix saved with ctx.save_for_backward, followed by three additional dispatches through the same buffer before backward.
A delayed reader on another stream, dropping the original receive allocation references and allocating/filling four same-capacity tensors on the allocation stream. The copy destination and allocator warm-up are prepared before the delay. Pressure writes must finish while a pre-read event is still pending; a pending post-copy event alone is insufficient.

Scheduling can miss the pressure window. All ranks agree on numerical correctness and window validity, then retry in collective lockstep with a longer delay (at most three attempts). Actual corruption fails immediately; an inconclusive window is never silently reported as coverage.

The consumer handoff uses a CUDA event captured after the first dispatch-event wait, including its deterministic-sort hook. It does not depend on repeated waits on the original EventOverlap; that is separate from this lifetime coverage.

Validation

Current polished revision

Head: 97fa6d18ff717df99acbdbb862325bda3d4a0d46, based on public main b5a33eefda39485f438dc764695b71ba7b9249b6.

The current 166-line test passed on H200 with PyTorch 2.13.0+cu132, CUDA 13.2 and NCCL 2.32.3. Each native lifetime policy was run once with eight ranks: 24 distributed saved-prefix/backward cases in total, with 24 valid delayed-consumer allocator-pressure windows. This is positive coverage of the current reorganized test; the historical negative and scheduling controls below were not rerun against this revision.

EP_AVOID_RECORD_STREAM=0 python tests/ep/test_recv_lifetime.py --num-processes 8 --num-tokens 64 --hidden 512
EP_AVOID_RECORD_STREAM=1 python tests/ep/test_recv_lifetime.py --num-processes 8 --num-tokens 64 --hidden 512

Historical validation of the earlier test version

Tested commit: 01c31d0d75f155ca00ba933a6b9ab7a01459169e, based on upstream 93eb6eb238127e96c6d7a4a625a6dad158348509. The results in this subsection belong to that earlier test version, not to a rerun of the current 166-line test.

One node, eight NVIDIA H200 GPUs in a single NVLink domain; PyTorch 2.13.0+cu132, CUDA runtime 13.2, loaded NCCL 2.32.3. The unchanged upstream native extension and the tested source archive/test-file hashes were verified. No borrowed-view implementation or event-wrapper patch was included.

Both native lifetime policies were run in fresh processes, three repeats each, using the same two command lines above.

Validation controls: EP_DISABLE_GIN=1, EP_REUSE_NCCL_COMM=0, EP_BUFFER_DEBUG=0, NCCL_NET_PLUGIN=none, CUBLAS_WORKSPACE_CONFIG=:4096:8. Seed 42; deterministic PyTorch algorithms enabled with uninitialized-memory filling disabled.

Native record-stream policy	Fresh repeats per arm	Positive rank/cases	Exact saved-view gradients	Deleted-record rejections	Forced-retry passes
`EP_AVOID_RECORD_STREAM=0`	3	288/288	288/288	96/96	96/96
`EP_AVOID_RECORD_STREAM=1`	3	288/288	288/288	96/96	96/96

Two additional controls were run outside the PR using that exact earlier public test source, without changing DeepEP's implementation:

Negative control: AST-remove exactly the consumer-side prefix.record_stream(reader) expression from cross_stream_pressure. For balanced nonempty receives, run both deterministic modes and both allocation streams. All 24 distributed cases were rejected by the specific CROSS_STREAM_RECEIVE_MISMATCH assertion; actual local data corruption was observed in all 192 rank/cases. Other assertions or an ineffective window do not count as detection.
Scheduling control: synchronize only rank 0 immediately before its first pending_before query. All ranks must retry attempt 0 together and pass on a later effective attempt. All 24 distributed cases / 192 rank/cases passed.

An earlier draft's post-copy-event stress missed the deleted-record mutation. That draft is not the sensitivity evidence above; the pre-read/completed-pressure ordering and destination preallocation were added before rerunning all historical arms.

All eighteen historical fresh launches exited successfully. Source/input hashes, exact per-rank coverage, control mutation identities and completion/exit sentinels were audited from downloaded logs. Archive SHA256: 834a31fa761e439678244ae49cdaf98491ead5fb23b4724500ba038366eac777.

Local checks for that earlier revision: Python compilation, Ruff 0.6.5, YAPF 0.40.2 (no diff), and git diff --check passed.

Scope

This establishes component lifetime/gradient correctness for ordinary compact BF16 receives in one physical NVLink domain. It does not establish full-model convergence, training throughput gains, borrowed-storage safety, FP8/expanded layouts, or multi-node hybrid routing. Capacity-retention assertions deliberately distinguish logical tensor bytes from retained storage bytes.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/780 **Author:** [@yuweih205](https://github.com/yuweih205) **Created:** 10/15/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `codex/syncfree-recv-lifetime-1015` --- ### 📝 Commits (1) - [`97fa6d1`](https://github.com/deepseek-ai/DeepEP/commit/97fa6d18ff717df99acbdbb862325bda3d4a0d46) Document and test sync-free receive tensor lifetimes ### 📊 Changes **2 files changed** (+168 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+2 -0) ➕ `tests/ep/test_recv_lifetime.py` (+166 -0) </details> ### 📄 Description ## Summary Document and exercise the lifetime contract of **ordinary independently allocated receive outputs** from compact `dispatch(do_cpu_sync=False)`. - Add a README paragraph explaining valid receive prefixes, output ownership versus stream readiness, the explicit ready-event/`record_stream` handoff, and why a prefix view retains the full capacity allocation. - Add a 166-line standalone native regression for saved-view backward after later dispatches and delayed cross-stream consumers under allocator pressure. No production Python/C++/CUDA implementation, default, or API changes. The internal dispatch copy epilogue still runs. This is not borrowed communication-buffer storage or a zero-copy dispatch implementation. ## Regression coverage The new `tests/ep/test_recv_lifetime.py` uses the independent NCCL dispatch reference and requires bitwise equality of payloads, local expert indices, routing weights, source-token metadata, and the saved-activation weight gradient. Each process run covers: - Deterministic dispatch off/on; output allocation on the compute/communication stream. - Uneven inputs, balanced routes, masked routes, and highly skewed routing with empty receivers. - Two selected experts on the same destination rank, distinguishing unique receive rows from token/expert assignments. - A prefix saved with `ctx.save_for_backward`, followed by three additional dispatches through the same buffer before backward. - A delayed reader on another stream, dropping the original receive allocation references and allocating/filling four same-capacity tensors on the allocation stream. The copy destination and allocator warm-up are prepared before the delay. Pressure writes must finish while a **pre-read** event is still pending; a pending post-copy event alone is insufficient. Scheduling can miss the pressure window. All ranks agree on numerical correctness and window validity, then retry in collective lockstep with a longer delay (at most three attempts). Actual corruption fails immediately; an inconclusive window is never silently reported as coverage. The consumer handoff uses a CUDA event captured **after the first dispatch-event wait**, including its deterministic-sort hook. It does not depend on repeated waits on the original `EventOverlap`; that is separate from this lifetime coverage. ## Validation ### Current polished revision Head: `97fa6d18ff717df99acbdbb862325bda3d4a0d46`, based on public main `b5a33eefda39485f438dc764695b71ba7b9249b6`. The current 166-line test passed on H200 with PyTorch `2.13.0+cu132`, CUDA 13.2 and NCCL 2.32.3. Each native lifetime policy was run once with eight ranks: 24 distributed saved-prefix/backward cases in total, with 24 valid delayed-consumer allocator-pressure windows. This is positive coverage of the current reorganized test; the historical negative and scheduling controls below were not rerun against this revision. ```bash EP_AVOID_RECORD_STREAM=0 python tests/ep/test_recv_lifetime.py --num-processes 8 --num-tokens 64 --hidden 512 EP_AVOID_RECORD_STREAM=1 python tests/ep/test_recv_lifetime.py --num-processes 8 --num-tokens 64 --hidden 512 ``` ### Historical validation of the earlier test version Tested commit: `01c31d0d75f155ca00ba933a6b9ab7a01459169e`, based on upstream `93eb6eb238127e96c6d7a4a625a6dad158348509`. The results in this subsection belong to that earlier test version, not to a rerun of the current 166-line test. One node, eight NVIDIA H200 GPUs in a single NVLink domain; PyTorch `2.13.0+cu132`, CUDA runtime 13.2, loaded NCCL 2.32.3. The unchanged upstream native extension and the tested source archive/test-file hashes were verified. No borrowed-view implementation or event-wrapper patch was included. Both native lifetime policies were run in fresh processes, three repeats each, using the same two command lines above. Validation controls: `EP_DISABLE_GIN=1`, `EP_REUSE_NCCL_COMM=0`, `EP_BUFFER_DEBUG=0`, `NCCL_NET_PLUGIN=none`, `CUBLAS_WORKSPACE_CONFIG=:4096:8`. Seed 42; deterministic PyTorch algorithms enabled with uninitialized-memory filling disabled. | Native record-stream policy | Fresh repeats per arm | Positive rank/cases | Exact saved-view gradients | Deleted-record rejections | Forced-retry passes | | --- | ---: | ---: | ---: | ---: | ---: | | `EP_AVOID_RECORD_STREAM=0` | 3 | 288/288 | 288/288 | 96/96 | 96/96 | | `EP_AVOID_RECORD_STREAM=1` | 3 | 288/288 | 288/288 | 96/96 | 96/96 | Two additional controls were run outside the PR using that exact earlier public test source, without changing DeepEP's implementation: - **Negative control:** AST-remove exactly the consumer-side `prefix.record_stream(reader)` expression from `cross_stream_pressure`. For balanced nonempty receives, run both deterministic modes and both allocation streams. All 24 distributed cases were rejected by the specific `CROSS_STREAM_RECEIVE_MISMATCH` assertion; actual local data corruption was observed in all 192 rank/cases. Other assertions or an ineffective window do not count as detection. - **Scheduling control:** synchronize only rank 0 immediately before its first `pending_before` query. All ranks must retry attempt 0 together and pass on a later effective attempt. All 24 distributed cases / 192 rank/cases passed. An earlier draft's post-copy-event stress missed the deleted-record mutation. That draft is not the sensitivity evidence above; the pre-read/completed-pressure ordering and destination preallocation were added before rerunning all historical arms. All eighteen historical fresh launches exited successfully. Source/input hashes, exact per-rank coverage, control mutation identities and completion/exit sentinels were audited from downloaded logs. Archive SHA256: `834a31fa761e439678244ae49cdaf98491ead5fb23b4724500ba038366eac777`. Local checks for that earlier revision: Python compilation, Ruff 0.6.5, YAPF 0.40.2 (no diff), and `git diff --check` passed. ## Scope This establishes component lifetime/gradient correctness for ordinary compact BF16 receives in one physical NVLink domain. It does not establish full-model convergence, training throughput gains, borrowed-storage safety, FP8/expanded layouts, or multi-node hybrid routing. Capacity-retention assertions deliberately distinguish logical tensor bytes from retained storage bytes. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:06:15 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#776

No description provided.

Rows
Columns