[GH-ISSUE #4382] [Bug][TENT] Terminal destructors discard failed RDMA teardown and release retained owners #1099

New issue

Closed

opened 2026-10-29 21:22:53 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:22:53 +00:00

Owner

Originally created by @feichai0017 on GitHub (Sep 29, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4382

Originally assigned to: @TTThanos on GitHub.

Bug report

Confirmed on current main, 1ad008bc41c6de7c74e02fb1d48713309b880c0d, with injected verbs failures through TENT's existing per-context verbs table. This is a terminal-destruction ownership gap, distinct from normal endpoint retirement/CQ draining.

RdmaEndPoint::deconstructUnlocked() correctly keeps its notification QP, MRs and backing vectors when ibv_destroy_qp() or ibv_dereg_mr() fails. RdmaContext::disable() likewise returns an error and preserves the context resources when the endpoint store cannot clear. Those paths allow explicit callers to retry.

However, both destructors discard that error:

RdmaEndPoint::~RdmaEndPoint() { deconstruct(); }
RdmaContext::~RdmaContext() { disable(); }

After the endpoint destructor body returns, the notification vectors are destroyed even if a QP/MR is still live. After the context destructor returns, its store and context object disappear even if the store retained failed endpoints; an externally held endpoint can retain a pointer to a destroyed context. Logging that resources are preserved for retry does not preserve their C++ owners past terminal destruction.

Source at the tested revision: endpoint destructor, context destructor.

Reproduction

CPU-only Linux x86_64, Clang 20, USE_TENT=ON, no RDMA hardware required. A regression uses the existing friend test access and per-context verbs table:

Give an endpoint a notification QP, two notification MRs and owned backing vectors.
Have ibv_destroy_qp() return EIO. Explicit deconstruct() returns failure and retains both vectors without calling MR deregistration.
Allow QP destruction but have ibv_dereg_mr() fail. Explicit teardown again retains the vectors.
Reset the last endpoint shared_ptr while either failure persists: the current destructor returns normally.
Put the failed endpoint in the real FIFO store, then destroy its context: the current context destructor also returns normally despite the store's failed clear.

On unmodified production code, both explicit retry tests pass and all three terminal-destructor death tests fail with “failed to die.” A fix is prepared that retains explicit retry behavior but fails the process at terminal destruction if native teardown still fails. It does not pretend that a timeout, failed verb, or destructor entry proves DMA quiescence.

#2664 established TENT's endpoint lifecycle/retention behavior.
#4062 preserves the notification QP mapping and waits for notification completions during retirement.
#4263 exercises context acquisition failures using injected verbs.
#4061 tracks notification channel fault tolerance.

The current-main versions were inspected. I also searched open/closed destructor, teardown, deconstruct, and QP-destruction issues/PRs; no matching terminal-error check was found.

The proposed scope is deliberately limited to endpoint-owned notification resources and context failure propagated from endpoint-store teardown. This is CPU fault-injection evidence, not RNIC/provider fault-injection qualification.

Prepared with Codex assistance; baseline and corrected native tests were executed.

Originally created by @feichai0017 on GitHub (Sep 29, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4382 Originally assigned to: @TTThanos on GitHub. ## Bug report Confirmed on current `main`, `1ad008bc41c6de7c74e02fb1d48713309b880c0d`, with injected verbs failures through TENT's existing per-context verbs table. This is a terminal-destruction ownership gap, distinct from normal endpoint retirement/CQ draining. `RdmaEndPoint::deconstructUnlocked()` correctly keeps its notification QP, MRs and backing vectors when `ibv_destroy_qp()` or `ibv_dereg_mr()` fails. `RdmaContext::disable()` likewise returns an error and preserves the context resources when the endpoint store cannot clear. Those paths allow explicit callers to retry. However, both destructors discard that error: ```cpp RdmaEndPoint::~RdmaEndPoint() { deconstruct(); } RdmaContext::~RdmaContext() { disable(); } ``` After the endpoint destructor body returns, the notification vectors are destroyed even if a QP/MR is still live. After the context destructor returns, its store and context object disappear even if the store retained failed endpoints; an externally held endpoint can retain a pointer to a destroyed context. Logging that resources are preserved for retry does not preserve their C++ owners past terminal destruction. Source at the tested revision: [endpoint destructor](https://github.com/kvcache-ai/Mooncake/blob/1ad008bc41c6de7c74e02fb1d48713309b880c0d/mooncake-transfer-engine/tent/src/transport/rdma/endpoint.cpp#L89), [context destructor](https://github.com/kvcache-ai/Mooncake/blob/1ad008bc41c6de7c74e02fb1d48713309b880c0d/mooncake-transfer-engine/tent/src/transport/rdma/context.cpp#L310). ## Reproduction CPU-only Linux x86_64, Clang 20, `USE_TENT=ON`, no RDMA hardware required. A regression uses the existing friend test access and per-context verbs table: 1. Give an endpoint a notification QP, two notification MRs and owned backing vectors. 2. Have `ibv_destroy_qp()` return `EIO`. Explicit `deconstruct()` returns failure and retains both vectors without calling MR deregistration. 3. Allow QP destruction but have `ibv_dereg_mr()` fail. Explicit teardown again retains the vectors. 4. Reset the last endpoint `shared_ptr` while either failure persists: the current destructor returns normally. 5. Put the failed endpoint in the real FIFO store, then destroy its context: the current context destructor also returns normally despite the store's failed clear. On unmodified production code, both explicit retry tests pass and all three terminal-destructor death tests fail with “failed to die.” A fix is prepared that retains explicit retry behavior but fails the process at terminal destruction if native teardown still fails. It does not pretend that a timeout, failed verb, or destructor entry proves DMA quiescence. ## Related work checked - #2664 established TENT's endpoint lifecycle/retention behavior. - #4062 preserves the notification QP mapping and waits for notification completions during retirement. - #4263 exercises context acquisition failures using injected verbs. - #4061 tracks notification channel fault tolerance. The current-main versions were inspected. I also searched open/closed destructor, teardown, `deconstruct`, and QP-destruction issues/PRs; no matching terminal-error check was found. The proposed scope is deliberately limited to endpoint-owned notification resources and context failure propagated from endpoint-store teardown. This is CPU fault-injection evidence, not RNIC/provider fault-injection qualification. Prepared with Codex assistance; baseline and corrected native tests were executed.

bsakaguchi closed this issue

2026-10-29 21:22:53 +00:00

bsakaguchi commented

2026-10-29 21:22:54 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 29, 2026):

Thanks for opening this issue, @feichai0017!

Field	Value
Issue	#4382
GitHub user ID	`132118805`
Reporter	@feichai0017

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Sep 29, 2026): Thanks for opening this issue, @feichai0017! | Field | Value | |-------|-------| | **Issue** | #4382 | | **GitHub user ID** | `132118805` | | **Reporter** | @feichai0017 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:06 +00:00

[PR #1099] [MERGED] Add a CI test for Mooncake EP Backend (CPU only) #2011