kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4768] [Bug]: Stale NoF controllers and qpairs remain on the target after connection failures #1217

New issue

Open

opened 2026-10-29 21:24:04 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:24:04 +00:00

Owner

Originally created by @cage-goat on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4768

Bug Report

Description

Following a network failure, the client may detect the connection failure and start recovery while the target has not yet detected that the old connections are no longer valid. The old controller and qpairs can remain on the target and continue consuming resources.

PR #4656 implements client-side controller recovery and qpair reconnection, but timely cleanup of stale target-side connections still needs to be addressed.

Mooncake currently sets keep_alive_timeout_ms to 0 for NoF clients, disabling NVMe Keep Alive. The target therefore cannot use Keep Alive expiration to detect an unresponsive client and reclaim its connections.

Steps to Reproduce

Start an NVMe-oF target and create a namespace.
Connect a Mooncake NoF client and verify that reads and writes succeed.
Temporarily interrupt NVMe-oF network communication between the client and target, keeping the target process running.
Restore the network and wait for the client to complete controller recovery and qpair reconnection.
Inspect the subsystem's controllers and qpairs on the target to check whether the old connections have been reclaimed.

Actual Behavior

After the client reconnects, the old controller and qpairs may remain on the target and continue consuming resources.

Expected Behavior

Support configurable KATO for NoF clients and continuously process the admin queue so that:

Healthy clients send Keep Alive commands and maintain their connections even when no data I/O is submitted.
The target reclaims old controllers and qpairs after Keep Alive expires for an unreachable client.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @cage-goat on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4768 ### Bug Report ## Description Following a network failure, the client may detect the connection failure and start recovery while the target has not yet detected that the old connections are no longer valid. The old controller and qpairs can remain on the target and continue consuming resources. [PR #4656](https://github.com/kvcache-ai/Mooncake/pull/4656) implements client-side controller recovery and qpair reconnection, but timely cleanup of stale target-side connections still needs to be addressed. Mooncake currently sets `keep_alive_timeout_ms` to `0` for NoF clients, disabling NVMe Keep Alive. The target therefore cannot use Keep Alive expiration to detect an unresponsive client and reclaim its connections. ## Steps to Reproduce 1. Start an NVMe-oF target and create a namespace. 2. Connect a Mooncake NoF client and verify that reads and writes succeed. 3. Temporarily interrupt NVMe-oF network communication between the client and target, keeping the target process running. 4. Restore the network and wait for the client to complete controller recovery and qpair reconnection. 5. Inspect the subsystem's controllers and qpairs on the target to check whether the old connections have been reclaimed. ## Actual Behavior After the client reconnects, the old controller and qpairs may remain on the target and continue consuming resources. ## Expected Behavior Support configurable KATO for NoF clients and continuously process the admin queue so that: - Healthy clients send Keep Alive commands and maintain their connections even when no data I/O is submitted. - The target reclaims old controllers and qpairs after Keep Alive expires for an unreachable client. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:24:04 +00:00

bsakaguchi commented

2026-10-29 21:24:05 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 19, 2026):

Thanks for opening this issue, @cage-goat!

Field	Value
Issue	#4768
GitHub user ID	`318081269`
Reporter	@cage-goat

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 19, 2026): Thanks for opening this issue, @cage-goat! | Field | Value | |-------|-------| | **Issue** | #4768 | | **GitHub user ID** | `318081269` | | **Reporter** | @cage-goat | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:24:05 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 19, 2026):

The spot you identified is exactly where this lives: ApplyCtrlrOptsFromEnv (mooncake-store/src/spdk/spdk_wrapper.cpp) hardcodes keep_alive_timeout_ms = 0 right before the other env-driven options are applied. The config plumbing precedent is already in place — SpdkControllerConfig::FromEnvironment() carries num_io_queues, io_queue_size, transport_ack_timeout, admin_queue_size, fabrics_connect_timeout_us and the digest flags — so a MC_STORE_SPDK_KEEP_ALIVE_TIMEOUT_MS (or similar) field would slot in the same way, default 0 to keep today's behavior for existing deployments.

One design question worth nailing down before a PR: Keep Alive only works if the client's admin queue is actually being polled while idle — SPDK delivers keep-alive completion through admin-queue processing, and if Mooncake's NoF path only touches the admin queue at connect/disconnect time, setting a nonzero KATO on the client still leaves the target unable to reclaim. Whoever picks this up should check whether the transport has a background poll of the admin queue (or needs one) alongside the config plumbing. That check needs real NVMe-oF hardware to validate, so this is best authored by someone with a target to test against — if you have one and want to take it, a PR adding the config field + opts application + the admin-queue answer would be very welcome, and I'm happy to review.

(Related: #4656 covered client-side recovery; this target-side reclamation is its complement.)

@he-yufeng commented on GitHub (Oct 19, 2026): The spot you identified is exactly where this lives: `ApplyCtrlrOptsFromEnv` (mooncake-store/src/spdk/spdk_wrapper.cpp) hardcodes `keep_alive_timeout_ms = 0` right before the other env-driven options are applied. The config plumbing precedent is already in place — `SpdkControllerConfig::FromEnvironment()` carries `num_io_queues`, `io_queue_size`, `transport_ack_timeout`, `admin_queue_size`, `fabrics_connect_timeout_us` and the digest flags — so a `MC_STORE_SPDK_KEEP_ALIVE_TIMEOUT_MS` (or similar) field would slot in the same way, default 0 to keep today's behavior for existing deployments. One design question worth nailing down before a PR: Keep Alive only works if the client's admin queue is actually being polled while idle — SPDK delivers keep-alive completion through admin-queue processing, and if Mooncake's NoF path only touches the admin queue at connect/disconnect time, setting a nonzero KATO on the client still leaves the target unable to reclaim. Whoever picks this up should check whether the transport has a background poll of the admin queue (or needs one) alongside the config plumbing. That check needs real NVMe-oF hardware to validate, so this is best authored by someone with a target to test against — if you have one and want to take it, a PR adding the config field + opts application + the admin-queue answer would be very welcome, and I'm happy to review. (Related: #4656 covered client-side recovery; this target-side reclamation is its complement.)

bsakaguchi commented

2026-10-29 21:24:05 +00:00

Author

Owner

@cage-goat commented on GitHub (Oct 19, 2026):

Thanks @he-yufeng! I've already opened #4769 to address this. It adds configurable KATO and builds on the controller thread introduced in #4656, which continues polling admin queues even when there is no data I/O.
I've also validated it against a standalone SPDK NVMe/TCP target backed by a malloc bdev:

With a 10-second KATO, connections stayed healthy through 25 seconds without data I/O.
After pausing the client with SIGSTOP, the target reclaimed the old controller and all qpairs in approximately 15 seconds, confirmed through target RPCs and Keep Alive timeout logs.
After SIGCONT, the same client recovered using the original pool and segment handles, and both existing-data reads and new write/read verification passed.
With KATO disabled, the target retained the connections during the observation window, as expected.
These results are included in the PR description. I'd appreciate your review!

@cage-goat commented on GitHub (Oct 19, 2026): Thanks @he-yufeng! I've already opened [#4769](https://github.com/kvcache-ai/Mooncake/pull/4769) to address this. It adds configurable KATO and builds on the controller thread introduced in #4656, which continues polling admin queues even when there is no data I/O. I've also validated it against a standalone SPDK NVMe/TCP target backed by a malloc bdev: - With a 10-second KATO, connections stayed healthy through 25 seconds without data I/O. - After pausing the client with SIGSTOP, the target reclaimed the old controller and all qpairs in approximately 15 seconds, confirmed through target RPCs and Keep Alive timeout logs. - After SIGCONT, the same client recovered using the original pool and segment handles, and both existing-data reads and new write/read verification passed. - With KATO disabled, the target retained the connections during the observation window, as expected. These results are included in the PR description. I'd appreciate your review!

bsakaguchi referenced this issue

2026-10-29 21:32:39 +00:00

[PR #1217] [MERGED] refactor tensor api and add tests #2081