[PR #678] legacy: allow build-time override of watchdog timeout constants #699

New issue

Open

opened 2026-10-27 21:05:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:23 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/678
Author: @dmvevents
Created: 7/9/2026
Status: 🔄 Open

Base: main ← Head: legacy-timeout-build-override

📝 Commits (1)

0f6a826 legacy: allow build-time override of CPU/GPU watchdog timeout constants

📊 Changes

1 file changed (+8 additions, -0 deletions)

View changed files

📝 csrc/kernels/legacy/compiled.cuh (+8 -0)

📄 Description

Summary

Wraps LEGACY_NUM_CPU_TIMEOUT_SECS / LEGACY_NUM_TIMEOUT_CYCLES in #ifndef so integrators can raise the watchdog at build time (e.g. -DLEGACY_NUM_CPU_TIMEOUT_SECS=1800) without patching sources. Defaults unchanged.

Motivation

On first run, frameworks that JIT-compile their expert GEMMs hold some ranks inside dispatch while peers finish compiling. Concretely: SGLang's DeepGEMM warmup (10–20 min cold, per its own log banner) exceeds the hardcoded 100 s watchdog, which then kills the run with DeepEP error: timeout (dispatch CPU) although no communication fault exists.

Measured on 2× p5en.48xlarge (H200, 16 EFA NICs, EP16, Qwen3-30B-A3B-FP8, SGLang 0.5.5.post3): with the constant raised to 1800 s the identical workload completes and serves; write-up with numbers: https://dmvevents.github.io/deepep-efa-guidance/sections/08-v1-nvshmem-backend.html

Notes

Touches only the legacy (deep_ep.Buffer) path where the constant lives; happy to extend the same guard to the EPv2 tree if wanted.
An env-var runtime override would be nicer but the constants are compiled into device code; #ifndef is the minimal non-breaking step.
Ready for review — 8-line #ifndef wrap, zero behavior change when nothing is -D-defined. Note the two constants are coupled (200G cycles ~= 100 s device-side): overriding only LEGACY_NUM_CPU_TIMEOUT_SECS leaves the device timeout at its default, so integrators should raise both together.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/678 **Author:** [@dmvevents](https://github.com/dmvevents) **Created:** 7/9/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `legacy-timeout-build-override` --- ### 📝 Commits (1) - [`0f6a826`](https://github.com/deepseek-ai/DeepEP/commit/0f6a826034915d117e58f0398d2aa503099288a3) legacy: allow build-time override of CPU/GPU watchdog timeout constants ### 📊 Changes **1 file changed** (+8 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/legacy/compiled.cuh` (+8 -0) </details> ### 📄 Description ## Summary Wraps `LEGACY_NUM_CPU_TIMEOUT_SECS` / `LEGACY_NUM_TIMEOUT_CYCLES` in `#ifndef` so integrators can raise the watchdog at build time (e.g. `-DLEGACY_NUM_CPU_TIMEOUT_SECS=1800`) without patching sources. Defaults unchanged. ## Motivation On first run, frameworks that JIT-compile their expert GEMMs hold some ranks inside dispatch while peers finish compiling. Concretely: SGLang's DeepGEMM warmup (10–20 min cold, per its own log banner) exceeds the hardcoded 100 s watchdog, which then kills the run with `DeepEP error: timeout (dispatch CPU)` although no communication fault exists. Measured on 2× p5en.48xlarge (H200, 16 EFA NICs, EP16, Qwen3-30B-A3B-FP8, SGLang 0.5.5.post3): with the constant raised to 1800 s the identical workload completes and serves; write-up with numbers: https://dmvevents.github.io/deepep-efa-guidance/sections/08-v1-nvshmem-backend.html ## Notes - Touches only the legacy (`deep_ep.Buffer`) path where the constant lives; happy to extend the same guard to the EPv2 tree if wanted. - An env-var runtime override would be nicer but the constants are compiled into device code; `#ifndef` is the minimal non-breaking step. - Ready for review — 8-line `#ifndef` wrap, zero behavior change when nothing is `-D`-defined. Note the two constants are coupled (200G cycles ~= 100 s device-side): overriding only `LEGACY_NUM_CPU_TIMEOUT_SECS` leaves the device timeout at its default, so integrators should raise both together. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:05:23 +00:00

bsakaguchi referenced this issue

2026-10-27 21:05:31 +00:00

[PR #699] [MERGED] fix(build): inherit PyTorch C++ standard for hybrid EP #715