[GH-ISSUE #4807] [Bug]: Master NoF heartbeat connections lack KATO, leaving stale connections on the target #1221

New issue

Open

opened 2026-10-29 21:24:07 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:07 +00:00

Owner

Originally created by @cage-goat on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4807

Bug Report

Description

Client connections retaining stale controllers and qpairs on the target after connection failures have been reported in #4768. Master uses separate NoF connections for heartbeat probing. These connections also have KATO disabled and face a similar resource cleanup risk.

PR #4806 addresses qpair cleanup, controller reconnection, and probe recovery on Master. During a network failure, disconnect notifications may not reach the target. Master can therefore recover its connections while old controllers and qpairs remain allocated on the target.

Expected Behavior

Master probe connections should support configurable KATO:

Use the existing heartbeat thread to process the admin queue and drive Keep Alive.
Allow the target to reclaim stale controllers and qpairs after Keep Alive expires.
Preserve read-based health checks and the existing segment removal policy. Only successful probe reads should refresh the heartbeat liveness timestamp.

Client KATO support is being introduced in PR #4769, which has not yet been merged. The Master implementation should align with its configuration design to avoid introducing duplicate KATO settings.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @cage-goat on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4807 ### Bug Report ## Description Client connections retaining stale controllers and qpairs on the target after connection failures have been reported in [#4768](https://github.com/kvcache-ai/Mooncake/issues/4768). Master uses separate NoF connections for heartbeat probing. These connections also have KATO disabled and face a similar resource cleanup risk. [PR #4806](https://github.com/kvcache-ai/Mooncake/pull/4806) addresses qpair cleanup, controller reconnection, and probe recovery on Master. During a network failure, disconnect notifications may not reach the target. Master can therefore recover its connections while old controllers and qpairs remain allocated on the target. ## Expected Behavior Master probe connections should support configurable KATO: - Use the existing heartbeat thread to process the admin queue and drive Keep Alive. - Allow the target to reclaim stale controllers and qpairs after Keep Alive expires. - Preserve read-based health checks and the existing segment removal policy. Only successful probe reads should refresh the heartbeat liveness timestamp. ## Related Work Client KATO support is being introduced in [PR #4769](https://github.com/kvcache-ai/Mooncake/pull/4769), which has not yet been merged. The Master implementation should align with its configuration design to avoid introducing duplicate KATO settings. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:24:07 +00:00

bsakaguchi commented

2026-10-29 21:24:08 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 20, 2026):

Thanks for opening this issue, @cage-goat!

Field	Value
Issue	#4807
GitHub user ID	`318081269`
Reporter	@cage-goat

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 20, 2026): Thanks for opening this issue, @cage-goat! | Field | Value | |-------|-------| | **Issue** | #4807 | | **GitHub user ID** | `318081269` | | **Reporter** | @cage-goat | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.