[PR #644] Warn if Linux automatic NUMA balancing is enabled at buffer init #684

New issue

Open

opened 2026-10-27 21:05:15 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:15 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/644
Author: @toffee-desuwa
Created: 5/26/2026
Status: 🔄 Open

Base: main ← Head: numa-balancing-warning

📝 Commits (3)

15bac18 Warn if Linux automatic NUMA balancing is enabled at buffer init
f26be9f Soften NUMA balancing warning wording based on counter-evidence
d389e22 Explain the actual NUMA balancing cost mechanism in warning and README

📊 Changes

2 files changed (+44 additions, -0 deletions)

View changed files

📝 README.md (+13 -0)
📝 deep_ep/__init__.py (+31 -0)

📄 Description

Summary

Adds a startup-time warning when /proc/sys/kernel/numa_balancing != 0, plus a new ## Performance gotchas section in the README documenting the mitigation. Fixes #624.

Background

@wenjianhn reported in #624 that Linux automatic NUMA balancing (kernel.numa_balancing != 0, default on most 5.x kernels) periodically rewrites PTEs at the user/kernel boundary via task_numa_work, which is directly visible on the deep_ep::Buffer::internode_dispatch hot path.

funclatency -d 60 -m -p <pid> task_numa_work over 60 s on an SGLang workload:

avg = 5 msecs, total: 789 msecs, count: 139
msecs distribution: 0-1: 72, 2-3: 8, 8-15: 57, 16-31: 2

139 invocations in 60 s, with 2 samples reaching the 16-31 ms bucket — added directly to the EP critical path.

Change

deep_ep/__init__.py: new check_numa_balancing() helper, called between check_nccl_so() and init_jit() in the existing init block. Mirrors the check_nccl_so pattern (env-var suppress, silent skip on non-Linux, single warning emit). kernel.numa_balancing is a bitmask, so the check is value != '0' (covers modes 1, 2, and OR-ed combinations, all of which run task_numa_work).
README.md: new ## Performance gotchas section documenting the issue and the sysctl kernel.numa_balancing=0 mitigation, plus an EP_SUPPRESS_NUMA_CHECK entry in the env var listing.

The check is opt-out via EP_SUPPRESS_NUMA_CHECK=1 for CI / containerized environments where the user cannot toggle the sysctl.

Validation

Warning fires when kernel.numa_balancing is non-zero:

$ python -c "import deep_ep"
RuntimeWarning: Automatic NUMA balancing is enabled (kernel.numa_balancing=1), which can add ...

Silent when kernel.numa_balancing=0:

$ sudo sysctl -w kernel.numa_balancing=0
$ python -c "import deep_ep"
(no warning)

Silent when EP_SUPPRESS_NUMA_CHECK=1:

$ EP_SUPPRESS_NUMA_CHECK=1 python -c "import deep_ep"
(no warning)

Silent on non-Linux (no /proc/sys/kernel/numa_balancing): OSError caught silently.
python -m py_compile deep_ep/__init__.py passes; git diff --check clean; file is excluded from the repository ruff target so no lint conflicts.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/644 **Author:** [@toffee-desuwa](https://github.com/toffee-desuwa) **Created:** 5/26/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `numa-balancing-warning` --- ### 📝 Commits (3) - [`15bac18`](https://github.com/deepseek-ai/DeepEP/commit/15bac1860c338fd60ea415d6ebbcfa66fa960052) Warn if Linux automatic NUMA balancing is enabled at buffer init - [`f26be9f`](https://github.com/deepseek-ai/DeepEP/commit/f26be9f1ee35cdc3ec576c858b605e713c3ec04a) Soften NUMA balancing warning wording based on counter-evidence - [`d389e22`](https://github.com/deepseek-ai/DeepEP/commit/d389e22166516dd4d68c7179e3bd497ea49ddd54) Explain the actual NUMA balancing cost mechanism in warning and README ### 📊 Changes **2 files changed** (+44 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+13 -0) 📝 `deep_ep/__init__.py` (+31 -0) </details> ### 📄 Description ## Summary Adds a startup-time warning when `/proc/sys/kernel/numa_balancing != 0`, plus a new `## Performance gotchas` section in the README documenting the mitigation. Fixes #624. ## Background `@wenjianhn` reported in #624 that Linux automatic NUMA balancing (`kernel.numa_balancing != 0`, default on most 5.x kernels) periodically rewrites PTEs at the user/kernel boundary via `task_numa_work`, which is directly visible on the `deep_ep::Buffer::internode_dispatch` hot path. `funclatency -d 60 -m -p <pid> task_numa_work` over 60 s on an SGLang workload: ``` avg = 5 msecs, total: 789 msecs, count: 139 msecs distribution: 0-1: 72, 2-3: 8, 8-15: 57, 16-31: 2 ``` 139 invocations in 60 s, with 2 samples reaching the 16-31 ms bucket — added directly to the EP critical path. ## Change - `deep_ep/__init__.py`: new `check_numa_balancing()` helper, called between `check_nccl_so()` and `init_jit()` in the existing init block. Mirrors the `check_nccl_so` pattern (env-var suppress, silent skip on non-Linux, single warning emit). `kernel.numa_balancing` is a bitmask, so the check is `value != '0'` (covers modes 1, 2, and OR-ed combinations, all of which run `task_numa_work`). - `README.md`: new `## Performance gotchas` section documenting the issue and the `sysctl kernel.numa_balancing=0` mitigation, plus an `EP_SUPPRESS_NUMA_CHECK` entry in the env var listing. The check is opt-out via `EP_SUPPRESS_NUMA_CHECK=1` for CI / containerized environments where the user cannot toggle the sysctl. ## Validation - Warning fires when `kernel.numa_balancing` is non-zero: ``` $ python -c "import deep_ep" RuntimeWarning: Automatic NUMA balancing is enabled (kernel.numa_balancing=1), which can add ... ``` - Silent when `kernel.numa_balancing=0`: ``` $ sudo sysctl -w kernel.numa_balancing=0 $ python -c "import deep_ep" (no warning) ``` - Silent when `EP_SUPPRESS_NUMA_CHECK=1`: ``` $ EP_SUPPRESS_NUMA_CHECK=1 python -c "import deep_ep" (no warning) ``` - Silent on non-Linux (no `/proc/sys/kernel/numa_balancing`): `OSError` caught silently. - `python -m py_compile deep_ep/__init__.py` passes; `git diff --check` clean; file is excluded from the repository ruff target so no lint conflicts. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:05:15 +00:00

bsakaguchi referenced this issue

2026-10-27 21:05:25 +00:00

[PR #684] Fix JIT cache crash for rename-race losers on shared filesystems #704