[GH-ISSUE #4805] [Bug]: Master NoF heartbeat cannot recover after the target becomes available again #1219

New issue

Open

opened 2026-10-29 21:24:07 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:07 +00:00

Owner

Originally created by @cage-goat on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4805

Bug Report

Description

Master uses separate NoF connections for heartbeat probes. After a connection failure, subsequent probes may keep reusing the failed connection, causing heartbeats to fail even after the target recovers. This can eventually remove an available segment and clean up its associated replica metadata.

Steps to reproduce

Start Master and a NoF target, register a namespace, and confirm successful heartbeat probes.
Pause or stop the target until a probe times out or the connection fails.
Restore the target before the segment is removed, keeping the Master process and original registration unchanged, then observe subsequent probes.

Expected behavior

Subsequent heartbeats should reestablish the connection and resume successful probes while preserving the existing segment removal policy.

PR #4656 addresses client-side connection recovery but does not cover Master’s separate heartbeat connections.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @cage-goat on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4805 ### Bug Report ## Description Master uses separate NoF connections for heartbeat probes. After a connection failure, subsequent probes may keep reusing the failed connection, causing heartbeats to fail even after the target recovers. This can eventually remove an available segment and clean up its associated replica metadata. ## Steps to reproduce 1. Start Master and a NoF target, register a namespace, and confirm successful heartbeat probes. 2. Pause or stop the target until a probe times out or the connection fails. 3. Restore the target before the segment is removed, keeping the Master process and original registration unchanged, then observe subsequent probes. ## Expected behavior Subsequent heartbeats should reestablish the connection and resume successful probes while preserving the existing segment removal policy. ## Related work [PR #4656](https://github.com/kvcache-ai/Mooncake/pull/4656) addresses client-side connection recovery but does not cover Master’s separate heartbeat connections. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:24:07 +00:00

bsakaguchi commented

2026-10-29 21:24:08 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 20, 2026):

Thanks for opening this issue, @cage-goat!

Field	Value
Issue	#4805
GitHub user ID	`318081269`
Reporter	@cage-goat

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 20, 2026): Thanks for opening this issue, @cage-goat! | Field | Value | |-------|-------| | **Issue** | #4805 | | **GitHub user ID** | `318081269` | | **Reporter** | @cage-goat | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:39 +00:00

[PR #1219] [CLOSED] [Store] Delete useless meta from http metadata server #2083