[GH-ISSUE #683] JIT cache: rename-race loser crashes with `EP_HOST_ASSERT(runtime != nullptr)` when the cache is on a shared filesystem (NFS) #402

New issue

Open

opened 2026-10-26 10:00:06 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 10:00:06 +00:00

Owner

Originally created by @Oseltamivir on GitHub (Jul 11, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/683

Summary

Compiler::build() handles concurrent builds of the same kernel by compiling into a private temp directory and atomically renaming it into the cache; a losing process deletes its temp directory and re-reads the winner's (csrc/jit/compiler.hpp:156-158):

const auto runtime = kernel_runtime_cache->get(dir_path);
EP_HOST_ASSERT(runtime != nullptr);

This assumes a failed rename implies the winner's directory is visible to the loser. On a local filesystem that is guaranteed; on NFS it is not, and that is the default configuration, since the cache falls back to $HOME/.deep_ep (compiler.hpp:52) and home directories are commonly NFS on multi-node clusters. The losing rank dies on the assert and kills the job. Observed on 2.0.0 (EPv2); code unchanged on current main (60d44037).

Mechanism

Cold start, all ranks race build() for the same kernel signature:

Every rank misses at compiler.hpp:116. that stat primes each node's NFS client negative lookup cache for dir_path ("does not exist"), valid until the parent directory's attribute cache expires (up to ~60 s on default mounts).
All ranks compile in parallel into private <cache>/tmp/<uuid> dirs; the first finisher's rename lands atomically at the server.
A loser on another node gets an authoritative rename failure, deletes its temp dir, and re-stats dir_path. Answered "does not exist" from its still-valid negative cache entry, without a server round-trip. check_validity → false, get() → nullptr, assert at compiler.hpp:158.

The existing fsync/atomic-rename hardening is writer-side (durability and ordering at the server); this failure is reader-side, in the loser's own client cache, which writer-side fsync cannot invalidate. A failed rename() does not invalidate the caller's cached lookup of the target either.

Reproduced on the first attempt with 16 ranks across two 8-GPU H200 nodes and a cold NFS cache: all 8 ranks of the losing node crashed at the assert while JIT-building the dispatch/combine kernels (whole-node failure as the stale cache belongs to the node's NFS client), and the winner node's ranks hung in the next collective until the job was killed.

GB200/GB300-class systems (4 GPUs per node) are more exposed: every EP ≥ 8 job spans multiple NFS clients, so a shared cache races from the smallest multi-node shape (a GB300 deployment we checked shares an NFSv4.2 filesystem with default caching options). The interconnect is irrelevant — the race is in the host-side compile path, identical on aarch64.

Proposed fix: #684

The loser does not need the winner's directory: it holds freshly compiled, equivalent artifacts (the cache key hashes kernel name, source, flags, and compiler signature). Load the runtime from its own temp directory, cache it in-memory under the canonical key, then delete the temp dir. This is safe because KernelRuntime's constructor consumes the files eagerly (cuobjdump + cudaLibraryLoadFromFile/cuModuleLoad) and keeps only GPU handles; warm starts still read the shared cache as before, and other rename failures degrade gracefully instead of asserting.

Validated on the reproducing setup: the crashing scenario passes 16/16 with the fix, with the losing branch exercised 73 times across 80 rank×kernel builds (verified via EP_JIT_DEBUG=1 load paths); a warm-start rerun serves all 80 loads from the canonical cache directories.

Notes

Workaround until fixed: point EP_JIT_CACHE_DIR at a node-local path (e.g. under /tmp).
A per-kernel flock() (cf. deepseek-ai/DeepGEMM#301) would reduce the N-way duplicate compilation but not fix this crash: acquiring a lock does not invalidate the client's cached lookup of an unrelated path, and flock is not cross-node on nolock NFSv3 mounts.
With EP_JIT_DUMP_* set, the losing rank's dumps are deleted with its temp dir under the fix; the winner's remain in the cache.

Originally created by @Oseltamivir on GitHub (Jul 11, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/683 ## Summary `Compiler::build()` handles concurrent builds of the same kernel by compiling into a private temp directory and atomically renaming it into the cache; a losing process deletes its temp directory and re-reads the winner's (`csrc/jit/compiler.hpp:156-158`): ```cpp const auto runtime = kernel_runtime_cache->get(dir_path); EP_HOST_ASSERT(runtime != nullptr); ``` This assumes a failed rename implies the winner's directory is visible to the loser. On a local filesystem that is guaranteed; on NFS it is not, and that is the default configuration, since the cache falls back to `$HOME/.deep_ep` (`compiler.hpp:52`) and home directories are commonly NFS on multi-node clusters. The losing rank dies on the assert and kills the job. Observed on 2.0.0 (EPv2); code unchanged on current `main` (60d44037). ## Mechanism Cold start, all ranks race `build()` for the same kernel signature: 1. Every rank misses at `compiler.hpp:116`. that stat primes each node's NFS client *negative* lookup cache for `dir_path` ("does not exist"), valid until the parent directory's attribute cache expires (up to ~60 s on default mounts). 2. All ranks compile in parallel into private `<cache>/tmp/<uuid>` dirs; the first finisher's rename lands atomically at the server. 3. A loser on another node gets an authoritative rename failure, deletes its temp dir, and re-stats `dir_path`. Answered "does not exist" from its still-valid negative cache entry, without a server round-trip. `check_validity` → false, `get()` → `nullptr`, assert at `compiler.hpp:158`. The existing fsync/atomic-rename hardening is writer-side (durability and ordering at the server); this failure is reader-side, in the loser's own client cache, which writer-side fsync cannot invalidate. A failed `rename()` does not invalidate the caller's cached lookup of the target either. Reproduced on the first attempt with 16 ranks across two 8-GPU H200 nodes and a cold NFS cache: all 8 ranks of the losing node crashed at the assert while JIT-building the dispatch/combine kernels (whole-node failure as the stale cache belongs to the node's NFS client), and the winner node's ranks hung in the next collective until the job was killed. GB200/GB300-class systems (4 GPUs per node) are more exposed: every EP ≥ 8 job spans multiple NFS clients, so a shared cache races from the smallest multi-node shape (a GB300 deployment we checked shares an NFSv4.2 filesystem with default caching options). The interconnect is irrelevant — the race is in the host-side compile path, identical on aarch64. ## Proposed fix: #684 The loser does not need the winner's directory: it holds freshly compiled, equivalent artifacts (the cache key hashes kernel name, source, flags, and compiler signature). Load the runtime from its own temp directory, cache it in-memory under the canonical key, then delete the temp dir. This is safe because `KernelRuntime`'s constructor consumes the files eagerly (`cuobjdump` + `cudaLibraryLoadFromFile`/`cuModuleLoad`) and keeps only GPU handles; warm starts still read the shared cache as before, and other rename failures degrade gracefully instead of asserting. Validated on the reproducing setup: the crashing scenario passes 16/16 with the fix, with the losing branch exercised 73 times across 80 rank×kernel builds (verified via `EP_JIT_DEBUG=1` load paths); a warm-start rerun serves all 80 loads from the canonical cache directories. ## Notes - Workaround until fixed: point `EP_JIT_CACHE_DIR` at a node-local path (e.g. under `/tmp`). - A per-kernel `flock()` (cf. deepseek-ai/DeepGEMM#301) would reduce the N-way duplicate compilation but not fix this crash: acquiring a lock does not invalidate the client's cached lookup of an unrelated path, and `flock` is not cross-node on `nolock` NFSv3 mounts. - With `EP_JIT_DUMP_*` set, the losing rank's dumps are deleted with its temp dir under the fix; the winner's remain in the cache.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#402

No description provided.

Rows
Columns

[GH-ISSUE #683] JIT cache: rename-race loser crashes with EP_HOST_ASSERT(runtime != nullptr) when the cache is on a shared filesystem (NFS) #402

Summary

Mechanism

Proposed fix: #684

Notes

[GH-ISSUE #683] JIT cache: rename-race loser crashes with `EP_HOST_ASSERT(runtime != nullptr)` when the cache is on a shared filesystem (NFS) #402