kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3349] [Bug]: graceful_shutdown_test can hang after fork in USE_ETCD CI builds #851

New issue

Closed

opened 2026-10-29 21:20:11 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:20:11 +00:00

Owner

Originally created by @Icedcoco on GitHub (Aug 10, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3349

Bug Report

Summary

GracefulShutdownTest.SigtermTriggersCleanExit intermittently hangs in the Linux CI build and is killed by the test after a five-second timeout.

This failure is not specific to the PR on which it appears. The same signature has occurred on main and on multiple unrelated PRs.

The current evidence points to a bug in the test process model:

With USE_ETCD=ON, graceful_shutdown_test loads the Go c-shared etcd wrapper, whose runtime starts background threads before the tests run.
The test calls fork() from this already-multithreaded process.
The child then constructs TransferEngine, takes C++ locks, allocates memory, initializes logging/runtime state, and creates a std::thread.
These operations are not safe between fork() and exec() in the child of a multithreaded process.

The fixed 100 ms sleep used by the parent is a separate synchronization bug that makes the failure more likely under CI load.

Observed failure

Example job:

https://github.com/kvcache-ai/Mooncake/actions/runs/31331251297/job/93289807744?pr=3344

[ RUN      ] GracefulShutdownTest.SigtermTriggersCleanExit
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0809 19:41:01.889427 17248 transfer_engine_impl.cpp:990] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
/home/runner/work/Mooncake/Mooncake/mooncake-transfer-engine/tests/graceful_shutdown_test.cpp:37: Failure
Failed
child did not exit before timeout

/home/runner/work/Mooncake/Mooncake/mooncake-transfer-engine/tests/graceful_shutdown_test.cpp:58: Failure
Value of: (((status) & 0x7f) == 0)
  Actual: false
Expected: true
Child did not exit normally (signaled: true)

[  FAILED  ] GracefulShutdownTest.SigtermTriggersCleanExit (5111 ms)

The WIFEXITED(status) failure is secondary. After waiting for five seconds, waitChildWithTimeout() sends SIGKILL to the child, so the resulting status is necessarily signaled rather than normally exited.

Expected behavior

The test child should:

Finish constructing the requested TransferEngine state.
Install the graceful shutdown handlers.
Report that it is ready.
Receive SIGTERM or SIGINT.
Exit normally with status 128 + signo.

The test must also terminate within a bounded time when child setup fails.

Affected CI runs

I found the same failure signature in at least six independent workflow runs:

These runs include both main and unrelated PRs. One occurrence is in a Python 3.10 job, so this does not appear to be specific to Python 3.12.

The example PR, #3344, only removes unused declarations from:

mooncake-store/include/master_metric_manager.h

It does not modify Transfer Engine, graceful shutdown, signal handling, or this test.

Relevant CI configuration

The main Linux CI config enables the Go etcd wrapper and ASan:

cmake -G Ninja .. \
  -DUSE_HTTP=ON \
  -DUSE_CXL=ON \
  -DUSE_UB=ON \
  -DUSE_ETCD=ON \
  -DSTORE_USE_ETCD=ON \
  -DENABLE_ASAN=ON \
  -DENABLE_SCCACHE=ON \
  -DCMAKE_BUILD_TYPE=Debug \
  -DENABLE_DEBUG_SYMBOLS=OFF

The test is then run as part of:

MC_METADATA_SERVER=http://127.0.0.1:8080/metadata \
DEFAULT_KV_LEASE_TTL=500 \
ctest -j --output-on-failure

Local reproduction

Using a build matching the important CI options, especially USE_ETCD=ON and ENABLE_ASAN=ON, I stress-ran the failing test concurrently:

seq 1 500 | xargs -P32 -I{} \
  timeout 10s \
  ./build/mooncake-transfer-engine/tests/graceful_shutdown_test \
  --gtest_filter=GracefulShutdownTest.SigtermTriggersCleanExit

In one captured run, 8 out of 500 invocations reproduced the same approximately 5.1-second timeout.

With a minimal build using USE_ETCD=OFF, 1,000 concurrent invocations completed without this timeout.

The exact rate is scheduling-dependent, but enabling the Go c-shared etcd wrapper makes the original CI failure reproducible.

Root-cause analysis

1. The test forks from a multithreaded process

The test currently does the following:

pid_t pid = fork();

if (pid == 0) {
    auto engine = std::make_unique<TransferEngine>(false);
    engine->enableGracefulShutdown();
    pause();
    _exit(99);
}

With USE_ETCD=ON, transfer_engine links:

mooncake-common/etcd/libetcd_wrapper.so

This library is built with:

go build -buildmode=c-shared

Tracing the CI-equivalent test binary shows that the Go runtime has created roughly 60 threads before GTest reaches fork().

After fork(), only the calling thread exists in the child. The child nevertheless performs substantial non-async-signal-safe work, including:

constructing TransferEngineImpl;
allocating shared_ptr and shutdown token state;
acquiring g_registry_mutex and g_install_mutex;
interacting with logging and sanitizer runtime state;
creating and detaching the signal watcher std::thread.

POSIX requires the child of a multithreaded process to execute only async-signal-safe operations until a successful exec():

https://pubs.opengroup.org/onlinepubs/9799919799/functions/fork.html

The test therefore relies on undefined/unsupported post-fork behavior.

The Go wrapper is the trigger that makes this easy to reproduce, but this should not be treated as an etcd or Go runtime bug. Any background thread holding runtime, allocator, logging, or library state at the time of fork() can expose the same test defect.

The exact runtime-internal lock or loop where every failed child stalls has not been reduced to one symbol. However, tracing confirms that failed children remain running or stuck before completing their ready state, and the test already violates the required process contract independently of that internal detail.

2. A fixed sleep is used as readiness synchronization

The parent currently does:

usleep(100000);
kill(pid, SIGTERM);

The 100 ms delay does not prove that the child has completed either:

std::make_unique<TransferEngine>(false);

or:

engine->enableGracefulShutdown();

Under ASan, Go runtime initialization, and concurrent CI load, the parent can send SIGTERM while the child is still initializing.

In a simpler single-threaded build, this race can appear as an immediate signal termination before the Mooncake handler is installed. In the USE_ETCD=ON build, it can interact with the invalid post-fork runtime state and produce the observed hang.

Relationship to existing reports and fixes

#3271 and #3278

Issue #3271 and PR #3278 addressed a different production bug:

the shutdown watcher thread could receive SIGTERM/SIGINT itself;
the signal handler would write to the pipe and then pause on the only reader thread;
no thread remained to consume the pipe byte and perform cleanup.

PR #3278 fixed that by blocking SIGTERM/SIGINT while creating the watcher so the watcher inherits the blocked mask:

https://github.com/kvcache-ai/Mooncake/pull/3278

The failing CI commits already contain that fix. The current failure therefore does not indicate that #3278 is missing or ineffective for its intended case.

#3274

PR #3274 replaces the fixed 100 ms sleep with a readiness pipe:

https://github.com/kvcache-ai/Mooncake/pull/3274

That addresses the readiness race, but it does not address the multithreaded fork() problem. The child still constructs TransferEngine and starts the watcher before exec().

I applied the #3274 patch to the CI-equivalent build and stress-ran it. In one 500-run batch, six children hung before sending the readiness byte.

The current #3274 parent also uses an unbounded blocking read(). If the child never reaches the readiness notification, the parent can wait indefinitely instead of failing after five seconds.

For these reasons, #3274 is not sufficient by itself.

Proposed fix

Run the signal target in a fresh process image instead of constructing TransferEngine directly in the post-fork child.

A minimal implementation can keep everything in the existing test source:

Give graceful_shutdown_test a custom main() instead of linking gtest_main.

Add an internal helper mode, for example:

graceful_shutdown_test --shutdown-child single <ready-fd>
graceful_shutdown_test --shutdown-child destroyed <ready-fd>
graceful_shutdown_test --shutdown-child multiple <ready-fd>

From the GTest parent, create a readiness pipe and use posix_spawn() to execute /proc/self/exe in helper mode.
In helper mode, after the new process image has initialized normally:
- construct the requested engine state;
- call enableGracefulShutdown();
- write one readiness byte;
- wait for SIGTERM or SIGINT.
In the parent, use poll() with a finite timeout to wait for readiness.
After readiness, send the signal and wait for the expected exit status.
On every error or timeout path, close the pipe, kill the child if needed, and reap it with waitpid().

The process flow becomes:

multithreaded GTest parent
        |
        | posix_spawn("/proc/self/exe", helper mode)
        v
fresh process image and runtime initialization
        |
        | construct TransferEngine
        | enableGracefulShutdown()
        | write readiness byte
        v
receive SIGTERM/SIGINT
        |
        v
cleanup and _Exit(128 + signo)

The following existing cases can share this helper:

SigtermTriggersCleanExit
SigintTriggersCleanExit
EngineDestroyedBeforeSignal
MultipleEngines

IdempotentEnable can remain an in-process test.

ForkAfterInstallDoesNotHangChildSignal should remain a direct fork() test because it specifically verifies inherited handler behavior. In that test, the engine and handler are installed before fork(), and the child only needs to call async-signal-safe operations such as pause() and _exit(). It does not need a fixed sleep because the handler state already exists when fork() returns.

The expected implementation scope is limited to:

mooncake-transfer-engine/tests/graceful_shutdown_test.cpp
mooncake-transfer-engine/tests/CMakeLists.txt

No production graceful shutdown change should be required.

Approaches that would only hide the issue

The following changes may reduce the failure rate but do not fix the invalid process model:

increasing the 100 ms sleep;
increasing the five-second timeout;
marking the test RUN_SERIAL;
disabling ASan for this test;
disabling USE_ETCD for this test;
adding only an unbounded readiness pipe;
attempting to reset all affected library state with pthread_atfork().

In particular, pthread_atfork() cannot generally repair inherited Go, sanitizer, allocator, logging, and arbitrary C++ library state.

Acceptance criteria

The test passes with the normal Linux CI configuration, including USE_ETCD=ON and ENABLE_ASAN=ON.
At least 2,000 invocations under 32-way concurrency complete without a timeout.
The same test passes with USE_ETCD=OFF.
SIGTERM produces normal exit status 143.
SIGINT produces normal exit status 130.
Failure before readiness is reported within a bounded timeout.
Failure after signal delivery is reported within a bounded timeout.
Every failure path reaps the child and leaves no orphaned test process.
Existing coverage for idempotent enable, destroyed engines, multiple engines, and fork-after-install behavior is retained.

Impact

The confirmed impact is flaky CI and unrelated PRs being blocked by a Transfer Engine test failure.

This investigation does not show a failure in the normal graceful shutdown path of an application that installs the handlers without first entering an unsafe post-fork child state.

Applications that fork from a multithreaded process and then construct or reinitialize Mooncake components in the child without exec() would have the same general process-safety problem, but that is separate from the normal supported shutdown flow.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @Icedcoco on GitHub (Aug 10, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3349 ### Bug Report ## Summary `GracefulShutdownTest.SigtermTriggersCleanExit` intermittently hangs in the Linux CI build and is killed by the test after a five-second timeout. This failure is not specific to the PR on which it appears. The same signature has occurred on `main` and on multiple unrelated PRs. The current evidence points to a bug in the test process model: 1. With `USE_ETCD=ON`, `graceful_shutdown_test` loads the Go c-shared etcd wrapper, whose runtime starts background threads before the tests run. 2. The test calls `fork()` from this already-multithreaded process. 3. The child then constructs `TransferEngine`, takes C++ locks, allocates memory, initializes logging/runtime state, and creates a `std::thread`. 4. These operations are not safe between `fork()` and `exec()` in the child of a multithreaded process. The fixed 100 ms sleep used by the parent is a separate synchronization bug that makes the failure more likely under CI load. ## Observed failure Example job: https://github.com/kvcache-ai/Mooncake/actions/runs/31331251297/job/93289807744?pr=3344 ```text [ RUN ] GracefulShutdownTest.SigtermTriggersCleanExit WARNING: Logging before InitGoogleLogging() is written to STDERR I0809 19:41:01.889427 17248 transfer_engine_impl.cpp:990] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) /home/runner/work/Mooncake/Mooncake/mooncake-transfer-engine/tests/graceful_shutdown_test.cpp:37: Failure Failed child did not exit before timeout /home/runner/work/Mooncake/Mooncake/mooncake-transfer-engine/tests/graceful_shutdown_test.cpp:58: Failure Value of: (((status) & 0x7f) == 0) Actual: false Expected: true Child did not exit normally (signaled: true) [ FAILED ] GracefulShutdownTest.SigtermTriggersCleanExit (5111 ms) ``` The `WIFEXITED(status)` failure is secondary. After waiting for five seconds, `waitChildWithTimeout()` sends `SIGKILL` to the child, so the resulting status is necessarily signaled rather than normally exited. ## Expected behavior The test child should: 1. Finish constructing the requested `TransferEngine` state. 2. Install the graceful shutdown handlers. 3. Report that it is ready. 4. Receive SIGTERM or SIGINT. 5. Exit normally with status `128 + signo`. The test must also terminate within a bounded time when child setup fails. ## Affected CI runs I found the same failure signature in at least six independent workflow runs: - https://github.com/kvcache-ai/Mooncake/actions/runs/31331251297 - https://github.com/kvcache-ai/Mooncake/actions/runs/31263153332 - https://github.com/kvcache-ai/Mooncake/actions/runs/31169844813 - https://github.com/kvcache-ai/Mooncake/actions/runs/31159834899 - https://github.com/kvcache-ai/Mooncake/actions/runs/31086084818 - https://github.com/kvcache-ai/Mooncake/actions/runs/31082176523 These runs include both `main` and unrelated PRs. One occurrence is in a Python 3.10 job, so this does not appear to be specific to Python 3.12. The example PR, #3344, only removes unused declarations from: ```text mooncake-store/include/master_metric_manager.h ``` It does not modify Transfer Engine, graceful shutdown, signal handling, or this test. ## Relevant CI configuration The main Linux CI config enables the Go etcd wrapper and ASan: ```bash cmake -G Ninja .. \ -DUSE_HTTP=ON \ -DUSE_CXL=ON \ -DUSE_UB=ON \ -DUSE_ETCD=ON \ -DSTORE_USE_ETCD=ON \ -DENABLE_ASAN=ON \ -DENABLE_SCCACHE=ON \ -DCMAKE_BUILD_TYPE=Debug \ -DENABLE_DEBUG_SYMBOLS=OFF ``` The test is then run as part of: ```bash MC_METADATA_SERVER=http://127.0.0.1:8080/metadata \ DEFAULT_KV_LEASE_TTL=500 \ ctest -j --output-on-failure ``` ## Local reproduction Using a build matching the important CI options, especially `USE_ETCD=ON` and `ENABLE_ASAN=ON`, I stress-ran the failing test concurrently: ```bash seq 1 500 | xargs -P32 -I{} \ timeout 10s \ ./build/mooncake-transfer-engine/tests/graceful_shutdown_test \ --gtest_filter=GracefulShutdownTest.SigtermTriggersCleanExit ``` In one captured run, 8 out of 500 invocations reproduced the same approximately 5.1-second timeout. With a minimal build using `USE_ETCD=OFF`, 1,000 concurrent invocations completed without this timeout. The exact rate is scheduling-dependent, but enabling the Go c-shared etcd wrapper makes the original CI failure reproducible. ## Root-cause analysis ### 1. The test forks from a multithreaded process The test currently does the following: ```cpp pid_t pid = fork(); if (pid == 0) { auto engine = std::make_unique<TransferEngine>(false); engine->enableGracefulShutdown(); pause(); _exit(99); } ``` With `USE_ETCD=ON`, `transfer_engine` links: ```text mooncake-common/etcd/libetcd_wrapper.so ``` This library is built with: ```text go build -buildmode=c-shared ``` Tracing the CI-equivalent test binary shows that the Go runtime has created roughly 60 threads before GTest reaches `fork()`. After `fork()`, only the calling thread exists in the child. The child nevertheless performs substantial non-async-signal-safe work, including: - constructing `TransferEngineImpl`; - allocating `shared_ptr` and shutdown token state; - acquiring `g_registry_mutex` and `g_install_mutex`; - interacting with logging and sanitizer runtime state; - creating and detaching the signal watcher `std::thread`. POSIX requires the child of a multithreaded process to execute only async-signal-safe operations until a successful `exec()`: https://pubs.opengroup.org/onlinepubs/9799919799/functions/fork.html The test therefore relies on undefined/unsupported post-fork behavior. The Go wrapper is the trigger that makes this easy to reproduce, but this should not be treated as an etcd or Go runtime bug. Any background thread holding runtime, allocator, logging, or library state at the time of `fork()` can expose the same test defect. The exact runtime-internal lock or loop where every failed child stalls has not been reduced to one symbol. However, tracing confirms that failed children remain running or stuck before completing their ready state, and the test already violates the required process contract independently of that internal detail. ### 2. A fixed sleep is used as readiness synchronization The parent currently does: ```cpp usleep(100000); kill(pid, SIGTERM); ``` The 100 ms delay does not prove that the child has completed either: ```cpp std::make_unique<TransferEngine>(false); ``` or: ```cpp engine->enableGracefulShutdown(); ``` Under ASan, Go runtime initialization, and concurrent CI load, the parent can send SIGTERM while the child is still initializing. In a simpler single-threaded build, this race can appear as an immediate signal termination before the Mooncake handler is installed. In the `USE_ETCD=ON` build, it can interact with the invalid post-fork runtime state and produce the observed hang. ## Relationship to existing reports and fixes ### #3271 and #3278 Issue #3271 and PR #3278 addressed a different production bug: - the shutdown watcher thread could receive SIGTERM/SIGINT itself; - the signal handler would write to the pipe and then pause on the only reader thread; - no thread remained to consume the pipe byte and perform cleanup. PR #3278 fixed that by blocking SIGTERM/SIGINT while creating the watcher so the watcher inherits the blocked mask: https://github.com/kvcache-ai/Mooncake/pull/3278 The failing CI commits already contain that fix. The current failure therefore does not indicate that #3278 is missing or ineffective for its intended case. ### #3274 PR #3274 replaces the fixed 100 ms sleep with a readiness pipe: https://github.com/kvcache-ai/Mooncake/pull/3274 That addresses the readiness race, but it does not address the multithreaded `fork()` problem. The child still constructs `TransferEngine` and starts the watcher before `exec()`. I applied the #3274 patch to the CI-equivalent build and stress-ran it. In one 500-run batch, six children hung before sending the readiness byte. The current #3274 parent also uses an unbounded blocking `read()`. If the child never reaches the readiness notification, the parent can wait indefinitely instead of failing after five seconds. For these reasons, #3274 is not sufficient by itself. ## Proposed fix Run the signal target in a fresh process image instead of constructing `TransferEngine` directly in the post-fork child. A minimal implementation can keep everything in the existing test source: 1. Give `graceful_shutdown_test` a custom `main()` instead of linking `gtest_main`. 2. Add an internal helper mode, for example: ```text graceful_shutdown_test --shutdown-child single <ready-fd> graceful_shutdown_test --shutdown-child destroyed <ready-fd> graceful_shutdown_test --shutdown-child multiple <ready-fd> ``` 3. From the GTest parent, create a readiness pipe and use `posix_spawn()` to execute `/proc/self/exe` in helper mode. 4. In helper mode, after the new process image has initialized normally: - construct the requested engine state; - call `enableGracefulShutdown()`; - write one readiness byte; - wait for SIGTERM or SIGINT. 5. In the parent, use `poll()` with a finite timeout to wait for readiness. 6. After readiness, send the signal and wait for the expected exit status. 7. On every error or timeout path, close the pipe, kill the child if needed, and reap it with `waitpid()`. The process flow becomes: ```text multithreaded GTest parent | | posix_spawn("/proc/self/exe", helper mode) v fresh process image and runtime initialization | | construct TransferEngine | enableGracefulShutdown() | write readiness byte v receive SIGTERM/SIGINT | v cleanup and _Exit(128 + signo) ``` The following existing cases can share this helper: - `SigtermTriggersCleanExit` - `SigintTriggersCleanExit` - `EngineDestroyedBeforeSignal` - `MultipleEngines` `IdempotentEnable` can remain an in-process test. `ForkAfterInstallDoesNotHangChildSignal` should remain a direct `fork()` test because it specifically verifies inherited handler behavior. In that test, the engine and handler are installed before `fork()`, and the child only needs to call async-signal-safe operations such as `pause()` and `_exit()`. It does not need a fixed sleep because the handler state already exists when `fork()` returns. The expected implementation scope is limited to: ```text mooncake-transfer-engine/tests/graceful_shutdown_test.cpp mooncake-transfer-engine/tests/CMakeLists.txt ``` No production graceful shutdown change should be required. ## Approaches that would only hide the issue The following changes may reduce the failure rate but do not fix the invalid process model: - increasing the 100 ms sleep; - increasing the five-second timeout; - marking the test `RUN_SERIAL`; - disabling ASan for this test; - disabling `USE_ETCD` for this test; - adding only an unbounded readiness pipe; - attempting to reset all affected library state with `pthread_atfork()`. In particular, `pthread_atfork()` cannot generally repair inherited Go, sanitizer, allocator, logging, and arbitrary C++ library state. ## Acceptance criteria - The test passes with the normal Linux CI configuration, including `USE_ETCD=ON` and `ENABLE_ASAN=ON`. - At least 2,000 invocations under 32-way concurrency complete without a timeout. - The same test passes with `USE_ETCD=OFF`. - SIGTERM produces normal exit status 143. - SIGINT produces normal exit status 130. - Failure before readiness is reported within a bounded timeout. - Failure after signal delivery is reported within a bounded timeout. - Every failure path reaps the child and leaves no orphaned test process. - Existing coverage for idempotent enable, destroyed engines, multiple engines, and fork-after-install behavior is retained. ## Impact The confirmed impact is flaky CI and unrelated PRs being blocked by a Transfer Engine test failure. This investigation does not show a failure in the normal graceful shutdown path of an application that installs the handlers without first entering an unsafe post-fork child state. Applications that fork from a multithreaded process and then construct or reinitialize Mooncake components in the child without `exec()` would have the same general process-safety problem, but that is separate from the normal supported shutdown flow. ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:20:11 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:20:12 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 10, 2026):

Thanks for opening this issue, @Icedcoco!

Field	Value
Issue	#3349
GitHub user ID	`102317026`
Reporter	@Icedcoco

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 10, 2026): Thanks for opening this issue, @Icedcoco! | Field | Value | |-------|-------| | **Issue** | #3349 | | **GitHub user ID** | `102317026` | | **Reporter** | @Icedcoco | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.