[PR #730] [Runtime] Preflight full intranode P2P matrix #737

New issue

Open

opened 2026-10-27 21:05:45 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:45 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/730
Author: @0z5a
Created: 8/17/2026
Status: 🔄 Open

Base: main ← Head: fix/p2p-preflight

📝 Commits (5)

56a639e [Runtime] Preflight full intranode P2P matrix
ac0b0c8 Run yapf and ruff
29d1afe [Runtime] Resolve P2P peers by physical GPU identity
356cc06 [Runtime] Synchronize P2P identity discovery failures
db8166d Merge main and resolve upstream conflicts

📊 Changes

6 files changed (+537 additions, -36 deletions)

View changed files

📝 README.md (+5 -1)
📝 deep_ep/buffers/ep.py (+3 -3)
📝 deep_ep/utils/envs.py (+126 -32)
➕ deep_ep/utils/p2p.py (+108 -0)
➕ tests/utils/test_envs_p2p.py (+152 -0)
➕ tests/utils/test_p2p.py (+143 -0)

📄 Description

Incremental review: Files changed against main.

Fixes #584.

Problem

DeepEP currently reaches CUDA peer setup before validating that every participating intranode GPU can access every other GPU. On partial-P2P hosts this fails on the first unsupported pair, after initialization has already progressed, and does not show the complete incompatible topology.

A rank-local physical-device or node identity discovery failure is also gathered before raising, so other ranks do not wait for a missing participant in the first or next collective.

Changes

Gather stable Linux boot IDs and physical GPU UUIDs before buffer construction; CUDA logical ordinals are never exchanged across processes.
Map UUIDs into each source process's visible CUDA ordinals, with an NVML READ/WRITE fallback for peers hidden by per-rank CUDA_VISIBLE_DEVICES masks.
Query every required directed intranode P2P edge and reject duplicate rank assignments to one physical GPU.
Aggregate and deterministically report all unsupported rank/device pairs on every rank.
Gather rank-local identity discovery errors before peer queries, with the same deterministic error on all ranks and no extra collective on the successful path.
Run the preflight before legacy _C.Buffer allocation and before elastic communicator/buffer setup.
Document the full directed-P2P requirement and the alternative-backend guidance.
Add host-only regression tests, including the 8-GPU four-pair topology from #584 (48/56 unsupported directed pairs).

Validation

python -m pytest -q tests/utils/test_p2p.py tests/utils/test_envs_p2p.py — 31 passed. The 20 additional cases exercise the actual preflight with PyTorch/MPI adapters, identity discovery failures, missing UUID properties, deterministic multi-rank errors, successful two-collective execution, and peer-query failures.
Repository format.sh, Python bytecode compilation, and git diff --check — passed.
Built and imported the SM120 CUDA extension with PyTorch 2.13.0+cu130 and CUDA 13.0. The image's existing CUDA/NCCL wheel include and library directories were supplied through build environment variables.
Real two-process Gloo differential test, with GPU execution explicitly disabled and a rank-1 UUID discovery failure injected: at 2161508, rank 0 timed out after 6.006 seconds while rank 1 raised locally. With this fix, both ranks reported the same discovery error in about 4.1 ms. Both legacy and elastic constructors were exercised. These are failure-path observations, not performance benchmarks.

Current GPU validation (08df5fb)

On an RTX PRO 4000 Blackwell with PyTorch 2.13.0+cu130, using the freshly built SM120 extension:

Real CUDA + two-rank Gloo baseline reproduces the missing-UUID mismatch: the healthy rank times out after 6.005 seconds.
Both legacy and elastic constructors report the same injected rank-local discovery error on every rank before a requested 1 GiB buffer allocation. Allocated, reserved, and device free-memory deltas are all zero on every rank.
Duplicate physical-GPU assignments are rejected consistently on both ranks.
A single-visible-GPU NCCL preflight passes. Legacy communication kernels do not support a one-rank buffer; this check covers preflight only.

These checks validate initialization and error propagation. They are not dispatch/combine or model-throughput E2E results. The earlier eight-GPU topology benchmark below was not rerun for this follow-up.

Earlier GPU validation (2161508)

RTX 5080 real-CUDA check with only one visible device — physical UUID/boot ID discovery and NCCL preflight passed.

Earlier 8-rank failure-path benchmark (2161508)

Environment: 8x NVIDIA GeForce RTX 5060 Ti, CUDA 13, one NCCL rank per GPU. This PCIe host exposes no off-diagonal CUDA P2P edges, so the real topology is 56/56 unsupported directed pairs. Results use 10 warmups and 100 measured iterations; each iteration reports the slowest rank.

Measurement	Result
Full preflight median	1.717 ms
Full preflight p95 / max	2.908 / 3.082 ms
Equivalent two-object-gather baseline median	1.670 ms
Median increment over gather baseline	0.047 ms
Error agreement	identical on all 8 ranks
Reported unsupported edges	56 entries, `56/56`
Failed 1 GiB legacy buffer probe	0-byte allocated/reserved/free-memory delta on every GPU

The benchmark host validates the real distributed all-unsupported failure path. The exact partial-P2P topology from #584 is covered by the 48/56 regression test.

Scope

This is an initialization-only check. Dispatch and combine hot paths are unchanged.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/730 **Author:** [@0z5a](https://github.com/0z5a) **Created:** 8/17/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/p2p-preflight` --- ### 📝 Commits (5) - [`56a639e`](https://github.com/deepseek-ai/DeepEP/commit/56a639eb023b4cfaffddb0adbaf618a0c5db58ce) [Runtime] Preflight full intranode P2P matrix - [`ac0b0c8`](https://github.com/deepseek-ai/DeepEP/commit/ac0b0c8782516645f0bb2f1114cf501bc07981b3) Run yapf and ruff - [`29d1afe`](https://github.com/deepseek-ai/DeepEP/commit/29d1afe6ede7a2a79f8ffa57914b422204b2c212) [Runtime] Resolve P2P peers by physical GPU identity - [`356cc06`](https://github.com/deepseek-ai/DeepEP/commit/356cc06d1f355cd3bc1dc88d4d5f02eaf501071d) [Runtime] Synchronize P2P identity discovery failures - [`db8166d`](https://github.com/deepseek-ai/DeepEP/commit/db8166d6892d1fe1a4072d2a4ff798d2b5bfc38c) Merge main and resolve upstream conflicts ### 📊 Changes **6 files changed** (+537 additions, -36 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+5 -1) 📝 `deep_ep/buffers/ep.py` (+3 -3) 📝 `deep_ep/utils/envs.py` (+126 -32) ➕ `deep_ep/utils/p2p.py` (+108 -0) ➕ `tests/utils/test_envs_p2p.py` (+152 -0) ➕ `tests/utils/test_p2p.py` (+143 -0) </details> ### 📄 Description **Incremental review:** [Files changed](https://github.com/deepseek-ai/DeepEP/pull/730/files) against `main`. Fixes #584. ## Problem DeepEP currently reaches CUDA peer setup before validating that every participating intranode GPU can access every other GPU. On partial-P2P hosts this fails on the first unsupported pair, after initialization has already progressed, and does not show the complete incompatible topology. A rank-local physical-device or node identity discovery failure is also gathered before raising, so other ranks do not wait for a missing participant in the first or next collective. ## Changes - Gather stable Linux boot IDs and physical GPU UUIDs before buffer construction; CUDA logical ordinals are never exchanged across processes. - Map UUIDs into each source process's visible CUDA ordinals, with an NVML READ/WRITE fallback for peers hidden by per-rank `CUDA_VISIBLE_DEVICES` masks. - Query every required directed intranode P2P edge and reject duplicate rank assignments to one physical GPU. - Aggregate and deterministically report all unsupported rank/device pairs on every rank. - Gather rank-local identity discovery errors before peer queries, with the same deterministic error on all ranks and no extra collective on the successful path. - Run the preflight before legacy `_C.Buffer` allocation and before elastic communicator/buffer setup. - Document the full directed-P2P requirement and the alternative-backend guidance. - Add host-only regression tests, including the 8-GPU four-pair topology from #584 (`48/56` unsupported directed pairs). ## Validation - `python -m pytest -q tests/utils/test_p2p.py tests/utils/test_envs_p2p.py` — 31 passed. The 20 additional cases exercise the actual preflight with PyTorch/MPI adapters, identity discovery failures, missing UUID properties, deterministic multi-rank errors, successful two-collective execution, and peer-query failures. - Repository `format.sh`, Python bytecode compilation, and `git diff --check` — passed. - Built and imported the SM120 CUDA extension with PyTorch 2.13.0+cu130 and CUDA 13.0. The image's existing CUDA/NCCL wheel include and library directories were supplied through build environment variables. - Real two-process Gloo differential test, with GPU execution explicitly disabled and a rank-1 UUID discovery failure injected: at `2161508`, rank 0 timed out after 6.006 seconds while rank 1 raised locally. With this fix, both ranks reported the same discovery error in about 4.1 ms. Both legacy and elastic constructors were exercised. These are failure-path observations, not performance benchmarks. ### Current GPU validation (08df5fb) On an RTX PRO 4000 Blackwell with PyTorch 2.13.0+cu130, using the freshly built SM120 extension: - Real CUDA + two-rank Gloo baseline reproduces the missing-UUID mismatch: the healthy rank times out after 6.005 seconds. - Both legacy and elastic constructors report the same injected rank-local discovery error on every rank before a requested 1 GiB buffer allocation. Allocated, reserved, and device free-memory deltas are all zero on every rank. - Duplicate physical-GPU assignments are rejected consistently on both ranks. - A single-visible-GPU NCCL preflight passes. Legacy communication kernels do not support a one-rank buffer; this check covers preflight only. These checks validate initialization and error propagation. They are not dispatch/combine or model-throughput E2E results. The earlier eight-GPU topology benchmark below was not rerun for this follow-up. ### Earlier GPU validation (2161508) - RTX 5080 real-CUDA check with only one visible device — physical UUID/boot ID discovery and NCCL preflight passed. ### Earlier 8-rank failure-path benchmark (2161508) Environment: 8x NVIDIA GeForce RTX 5060 Ti, CUDA 13, one NCCL rank per GPU. This PCIe host exposes no off-diagonal CUDA P2P edges, so the real topology is `56/56` unsupported directed pairs. Results use 10 warmups and 100 measured iterations; each iteration reports the slowest rank. | Measurement | Result | |---|---:| | Full preflight median | 1.717 ms | | Full preflight p95 / max | 2.908 / 3.082 ms | | Equivalent two-object-gather baseline median | 1.670 ms | | Median increment over gather baseline | 0.047 ms | | Error agreement | identical on all 8 ranks | | Reported unsupported edges | 56 entries, `56/56` | | Failed 1 GiB legacy buffer probe | 0-byte allocated/reserved/free-memory delta on every GPU | The benchmark host validates the real distributed all-unsupported failure path. The exact partial-P2P topology from #584 is covered by the `48/56` regression test. ## Scope This is an initialization-only check. Dispatch and combine hot paths are unchanged. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:05:45 +00:00

bsakaguchi referenced this issue

2026-10-27 21:05:48 +00:00

[PR #737] fix: st.bulk support for size operand with 32-bit length is introduce… #742