[PR #1363] [CLOSED] [Store] Add HTTP metadata cleanup on client timeout #2194

New issue

Closed

opened 2026-10-29 21:33:29 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:33:29 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1363
Author: @chenkaiyue
Created: 1/12/2026
Status: ❌ Closed

Base: main ← Head: metadata_cleanup

📝 Commits (1)

694d48d feat(master): Add HTTP metadata cleanup on client timeout

📊 Changes

8 files changed (+141 additions, -4 deletions)

View changed files

📝 mooncake-store/include/http_metadata_server.h (+9 -0)
📝 mooncake-store/include/master_config.h (+4 -0)
📝 mooncake-store/include/master_service.h (+19 -0)
📝 mooncake-store/include/rpc_service.h (+6 -1)
📝 mooncake-store/src/http_metadata_server.cpp (+21 -0)
📝 mooncake-store/src/master.cpp (+38 -1)
📝 mooncake-store/src/master_service.cpp (+38 -0)
📝 mooncake-store/src/rpc_service.cpp (+6 -2)

📄 Description

Description

This PR adds a new feature that allows the Master Service to automatically clean up HTTP Metadata Server entries when a client heartbeat times out. This cleanup functionality is controlled by a new configuration flag --enable_metadata_cleanup_on_timeout and only works when the HTTP Metadata Server is deployed alongside the Master Service (--enable_http_metadata_server=true).

Problem

In Mooncake's distributed KV cache architecture, there are two services that store client metadata:

Master Service - Stores segment information, handles object storage coordination, and maintains client heartbeats via gRPC.
HTTP Metadata Server - Stores RDMA connection metadata (mooncake/ram/{segment_name}) and RPC routing information (mooncake/rpc_meta/{segment_name}).

When a client (e.g., SGLang inference server) terminates normally, it performs cleanup operations to both services:

Calls UnmountSegment() to the Master Service
Calls removeSegmentDesc() and removeRpcMetaEntry() to the HTTP Metadata Server

However, when a client crashes or is forcefully terminated (e.g., kill -9, OOM killed, node failure), the cleanup operations may fail or never execute. The Master Service can detect this through heartbeat timeout (client_ttl) and clean up its internal segment resources, but the HTTP Metadata Server has no heartbeat mechanism and cannot detect client failures.

This leads to stale metadata residue on the HTTP Metadata Server:

mooncake/ram/{segment_name} - Contains outdated RDMA buffer information
mooncake/rpc_meta/{segment_name} - Contains stale RPC connection details

These residual entries can cause issues when:

New nodes attempt to connect using stale metadata
The same node restarts and registers with different RDMA parameters
Other nodes query and cache outdated peer information

Solution

Leverage the Master Service's existing heartbeat mechanism to clean up HTTP Metadata Server entries when client timeout is detected:

Client (ping) ──────► Master Service ──────► HTTP Metadata Server
                            │                        │
                      Detect client timeout          │
                            │                        ▼
                            ▼                  Delete mooncake/ram/*
                      Unmount segment ────────► Delete mooncake/rpc_meta/*

Design Decisions

Opt-in Feature: Disabled by default (--enable_metadata_cleanup_on_timeout=false) to ensure backward compatibility.
Coupled Deployment Only: This feature only works when the HTTP Metadata Server is co-located with the Master Service. If enable_http_metadata_server=false but enable_metadata_cleanup_on_timeout=true, a warning is logged and the cleanup feature is automatically disabled.
Direct Memory Access: Since both services run in the same process when co-deployed, the Master Service directly calls HttpMetadataServer::removeKey() without network overhead.
Key Prefix Support: Respects the MC_METADATA_CLUSTER_ID environment variable for custom key prefixes.

Changes

Configuration (`mooncake-store/include/master_config.h`)

Added enable_metadata_cleanup_on_timeout boolean field to MasterConfig struct

Main Entry (`mooncake-store/src/master.cpp`)

Added --enable_metadata_cleanup_on_timeout command-line flag
Added configuration parsing for the new flag
Added validation logic: warns and disables cleanup if HTTP server is not enabled
Pass HttpMetadataServer pointer to WrappedMasterService when cleanup is enabled

HTTP Metadata Server (`mooncake-store/include/http_metadata_server.h`, `mooncake-store/src/http_metadata_server.cpp`)

Added removeKey(const std::string& key) method for single key deletion
Added removeKeys(const std::vector<std::string>& keys) method for batch deletion
Both methods are thread-safe using store_mutex_

Master Service (`mooncake-store/include/master_service.h`, `mooncake-store/src/master_service.cpp`)

Added HttpMetadataServer* http_metadata_server_ member
Added setHttpMetadataServer(HttpMetadataServer* server) method
Added cleanupHttpMetadata(const std::string& segment_name) method
Integrated cleanup call in ClientMonitorThreadMain() when processing expired clients

RPC Service (`mooncake-store/include/rpc_service.h`, `mooncake-store/src/rpc_service.cpp`)

Modified WrappedMasterService constructor to accept optional HttpMetadataServer* parameter
Pass the pointer to underlying MasterService when provided

Usage

To enable this feature:

mooncake_master \
    --enable_http_metadata_server=true \
    --enable_metadata_cleanup_on_timeout=true \
    --client_ttl=20 \
    --v=1

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1363 **Author:** [@chenkaiyue](https://github.com/chenkaiyue) **Created:** 1/12/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `metadata_cleanup` --- ### 📝 Commits (1) - [`694d48d`](https://github.com/kvcache-ai/Mooncake/commit/694d48d9d7abebbc7ad85ade14518948c24bc56e) feat(master): Add HTTP metadata cleanup on client timeout ### 📊 Changes **8 files changed** (+141 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-store/include/http_metadata_server.h` (+9 -0) 📝 `mooncake-store/include/master_config.h` (+4 -0) 📝 `mooncake-store/include/master_service.h` (+19 -0) 📝 `mooncake-store/include/rpc_service.h` (+6 -1) 📝 `mooncake-store/src/http_metadata_server.cpp` (+21 -0) 📝 `mooncake-store/src/master.cpp` (+38 -1) 📝 `mooncake-store/src/master_service.cpp` (+38 -0) 📝 `mooncake-store/src/rpc_service.cpp` (+6 -2) </details> ### 📄 Description ## Description This PR adds a new feature that allows the Master Service to automatically clean up HTTP Metadata Server entries when a client heartbeat times out. This cleanup functionality is controlled by a new configuration flag `--enable_metadata_cleanup_on_timeout` and only works when the HTTP Metadata Server is deployed alongside the Master Service (`--enable_http_metadata_server=true`). ## Problem In Mooncake's distributed KV cache architecture, there are two services that store client metadata: 1. **Master Service** - Stores segment information, handles object storage coordination, and maintains client heartbeats via gRPC. 2. **HTTP Metadata Server** - Stores RDMA connection metadata (`mooncake/ram/{segment_name}`) and RPC routing information (`mooncake/rpc_meta/{segment_name}`). When a client (e.g., SGLang inference server) terminates normally, it performs cleanup operations to both services: - Calls `UnmountSegment()` to the Master Service - Calls `removeSegmentDesc()` and `removeRpcMetaEntry()` to the HTTP Metadata Server However, when a client **crashes or is forcefully terminated** (e.g., `kill -9`, OOM killed, node failure), the cleanup operations may fail or never execute. The Master Service can detect this through heartbeat timeout (`client_ttl`) and clean up its internal segment resources, but the HTTP Metadata Server has **no heartbeat mechanism** and cannot detect client failures. This leads to **stale metadata residue** on the HTTP Metadata Server: - `mooncake/ram/{segment_name}` - Contains outdated RDMA buffer information - `mooncake/rpc_meta/{segment_name}` - Contains stale RPC connection details These residual entries can cause issues when: - New nodes attempt to connect using stale metadata - The same node restarts and registers with different RDMA parameters - Other nodes query and cache outdated peer information ## Solution Leverage the Master Service's existing heartbeat mechanism to clean up HTTP Metadata Server entries when client timeout is detected: ``` Client (ping) ──────► Master Service ──────► HTTP Metadata Server │ │ Detect client timeout │ │ ▼ ▼ Delete mooncake/ram/* Unmount segment ────────► Delete mooncake/rpc_meta/* ``` ### Design Decisions 1. **Opt-in Feature**: Disabled by default (`--enable_metadata_cleanup_on_timeout=false`) to ensure backward compatibility. 2. **Coupled Deployment Only**: This feature only works when the HTTP Metadata Server is co-located with the Master Service. If `enable_http_metadata_server=false` but `enable_metadata_cleanup_on_timeout=true`, a warning is logged and the cleanup feature is automatically disabled. 3. **Direct Memory Access**: Since both services run in the same process when co-deployed, the Master Service directly calls `HttpMetadataServer::removeKey()` without network overhead. 4. **Key Prefix Support**: Respects the `MC_METADATA_CLUSTER_ID` environment variable for custom key prefixes. ## Changes ### Configuration (`mooncake-store/include/master_config.h`) - Added `enable_metadata_cleanup_on_timeout` boolean field to `MasterConfig` struct ### Main Entry (`mooncake-store/src/master.cpp`) - Added `--enable_metadata_cleanup_on_timeout` command-line flag - Added configuration parsing for the new flag - Added validation logic: warns and disables cleanup if HTTP server is not enabled - Pass `HttpMetadataServer` pointer to `WrappedMasterService` when cleanup is enabled ### HTTP Metadata Server (`mooncake-store/include/http_metadata_server.h`, `mooncake-store/src/http_metadata_server.cpp`) - Added `removeKey(const std::string& key)` method for single key deletion - Added `removeKeys(const std::vector<std::string>& keys)` method for batch deletion - Both methods are thread-safe using `store_mutex_` ### Master Service (`mooncake-store/include/master_service.h`, `mooncake-store/src/master_service.cpp`) - Added `HttpMetadataServer* http_metadata_server_` member - Added `setHttpMetadataServer(HttpMetadataServer* server)` method - Added `cleanupHttpMetadata(const std::string& segment_name)` method - Integrated cleanup call in `ClientMonitorThreadMain()` when processing expired clients ### RPC Service (`mooncake-store/include/rpc_service.h`, `mooncake-store/src/rpc_service.cpp`) - Modified `WrappedMasterService` constructor to accept optional `HttpMetadataServer*` parameter - Pass the pointer to underlying `MasterService` when provided ## Usage To enable this feature: ```bash mooncake_master \ --enable_http_metadata_server=true \ --enable_metadata_cleanup_on_timeout=true \ --client_ttl=20 \ --v=1 ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:33:29 +00:00