[PR #684] Fix JIT cache crash for rename-race losers on shared filesystems #704

New issue

Open

opened 2026-10-27 21:05:25 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:25 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/684
Author: @Oseltamivir
Created: 7/11/2026
Status: 🔄 Open

Base: main ← Head: fix-jit-cache-nfs-rename-race

📝 Commits (2)

1ed3fbb Fix JIT cache crash for rename-race losers on shared filesystems
ef2ae56 Address review: share validity diagnostics in put(), document winner-path invariant, debug-log the fallback branch

📊 Changes

2 files changed (+27 additions, -1 deletions)

View changed files

📝 csrc/jit/cache.hpp (+12 -0)
📝 csrc/jit/compiler.hpp (+15 -1)

📄 Description

Fixes #683.

Problem

On a shared filesystem (the default $HOME/.deep_ep is typically NFS on multi-node clusters), the losing side of the compile/rename race in Compiler::build() crashes at EP_HOST_ASSERT(runtime != nullptr): its rename fails authoritatively at the server, but the winner's directory may not be visible to its NFS client yet, the initial cache miss primed a negative lookup cache entry that can outlive the failed rename. Full mechanism in #683.

Change

Make the losing path self-sufficient instead of requiring cross-client visibility:

csrc/jit/compiler.hpp: on rename failure, load the runtime from this process's own just-compiled temp directory (equivalent artifacts. The cache key hashes kernel name, source, flags, and compiler signature), cache it under the canonical dir_path key, then delete the temp directory.
csrc/jit/cache.hpp: add KernelRuntimeCache::put(key_path, load_path) to construct from one path and cache under another.

This is safe because the KernelRuntime constructor consumes the files eagerly (cuobjdump -symbols, then cudaLibraryLoadFromFile/cuModuleLoad) and keeps only GPU handles. Warm starts still read the shared cache exactly as before; other rename failures (e.g. permissions) now degrade gracefully instead of asserting (scope: the rename failure itself — a subsequent failure to load the loser's own artifacts remains fatal, with the same check_validity diagnostics as any other cache entry).

Testing

Two 8-GPU H200 nodes (16 ranks), JIT cache on shared NFS, cold start, ElasticBuffer construction plus a dispatch/combine round trip per rank (EP_JIT_DEBUG=1):

Unpatched: reproduced: all 8 ranks of the losing node crash at compiler.hpp:158 while JIT-building the dispatch/combine kernels; the other node hangs in the next collective.
Patched, same scenario: 16/16 pass; the new branch executed 73 times across 80 rank×kernel builds (Loading CUBIN: <cache>/tmp/<uuid>/kernel.cubin), and the shared tmp/ is empty afterwards.
Warm start: 16/16 pass with all 80 loads served from the canonical cache directories; the new branch is never taken.
The full extension builds cleanly with the patch (torch 2.10, CUDA 13.0, NCCL 2.30.4, sm90).
Review commit ef2ae56 re-validated on the same setup: the new EP_JIT_DEBUG fallback log fired 75/75 times, each paired 1:1 with a tmp-dir CUBIN load, 16/16 ranks green.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/684 **Author:** [@Oseltamivir](https://github.com/Oseltamivir) **Created:** 7/11/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix-jit-cache-nfs-rename-race` --- ### 📝 Commits (2) - [`1ed3fbb`](https://github.com/deepseek-ai/DeepEP/commit/1ed3fbb53a6539f4c5f2ad4c8123a7edab7d444a) Fix JIT cache crash for rename-race losers on shared filesystems - [`ef2ae56`](https://github.com/deepseek-ai/DeepEP/commit/ef2ae56ef54583ce60c1c327abab750ba8a43395) Address review: share validity diagnostics in put(), document winner-path invariant, debug-log the fallback branch ### 📊 Changes **2 files changed** (+27 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `csrc/jit/cache.hpp` (+12 -0) 📝 `csrc/jit/compiler.hpp` (+15 -1) </details> ### 📄 Description Fixes #683. ## Problem On a shared filesystem (the default `$HOME/.deep_ep` is typically NFS on multi-node clusters), the losing side of the compile/rename race in `Compiler::build()` crashes at `EP_HOST_ASSERT(runtime != nullptr)`: its rename fails authoritatively at the server, but the winner's directory may not be visible to its NFS client yet, the initial cache miss primed a negative lookup cache entry that can outlive the failed rename. Full mechanism in #683. ## Change Make the losing path self-sufficient instead of requiring cross-client visibility: - `csrc/jit/compiler.hpp`: on rename failure, load the runtime from this process's own just-compiled temp directory (equivalent artifacts. The cache key hashes kernel name, source, flags, and compiler signature), cache it under the canonical `dir_path` key, then delete the temp directory. - `csrc/jit/cache.hpp`: add `KernelRuntimeCache::put(key_path, load_path)` to construct from one path and cache under another. This is safe because the `KernelRuntime` constructor consumes the files eagerly (`cuobjdump -symbols`, then `cudaLibraryLoadFromFile`/`cuModuleLoad`) and keeps only GPU handles. Warm starts still read the shared cache exactly as before; other rename failures (e.g. permissions) now degrade gracefully instead of asserting (scope: the rename failure itself — a subsequent failure to load the loser's own artifacts remains fatal, with the same `check_validity` diagnostics as any other cache entry). ## Testing Two 8-GPU H200 nodes (16 ranks), JIT cache on shared NFS, cold start, `ElasticBuffer` construction plus a dispatch/combine round trip per rank (`EP_JIT_DEBUG=1`): - **Unpatched**: reproduced: all 8 ranks of the losing node crash at `compiler.hpp:158` while JIT-building the dispatch/combine kernels; the other node hangs in the next collective. - **Patched, same scenario**: 16/16 pass; the new branch executed 73 times across 80 rank×kernel builds (`Loading CUBIN: <cache>/tmp/<uuid>/kernel.cubin`), and the shared `tmp/` is empty afterwards. - **Warm start**: 16/16 pass with all 80 loads served from the canonical cache directories; the new branch is never taken. - The full extension builds cleanly with the patch (torch 2.10, CUDA 13.0, NCCL 2.30.4, sm90). - Review commit ef2ae56 re-validated on the same setup: the new `EP_JIT_DEBUG` fallback log fired 75/75 times, each paired 1:1 with a tmp-dir CUBIN load, 16/16 ranks green. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:05:25 +00:00

bsakaguchi referenced this issue

2026-10-27 21:05:33 +00:00

[PR #704] Make the low-latency timeout trap self-attributing in the log #719

bsakaguchi referenced this issue

2026-10-27 21:05:37 +00:00

[PR #706] Make the elastic GIN signal reads backend-agnostic (fix latent PROXY layout mismatch) #721

bsakaguchi referenced this issue

2026-10-27 21:05:37 +00:00