[GH-ISSUE #1605] [RFC]: Embedded HTTP Server for Mooncake Store Client #476

New issue

Closed

opened 2026-10-29 21:15:11 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:15:11 +00:00

Owner

Originally created by @duhaode520 on GitHub (Mar 4, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1605

Originally assigned to: @duhaode520 on GitHub.

1. Background & Motivation

With the introduction of DummyClient, the Mooncake Store client architecture is evolving toward a standalone deployment model. Previously, the Store client was tightly coupled with the inference framework process (e.g., vLLM, SGLang). Now, RealClient can run as an independent long-lived process, while one or more DummyClient instances connect to it from inference workers via RPC.

This shift makes RealClient a first-class infrastructure component — similar to a sidecar or daemon — that needs proper observability and lifecycle management, just like the Master service. However, while the Master already has an embedded HTTP server exposing /metrics and management endpoints, the Client side lacks equivalent capability:

Gap	Impact
No HTTP health-check endpoint	K8s liveness/readiness probes cannot directly monitor Client status
No HTTP metrics exposure	`ClientMetric` already collects rich transfer/RPC metrics, but they are only accessible via logs — Prometheus cannot pull them

The current PR (feat/mooncake-health-check) adds a /health endpoint to RealClient. This RFC proposes extending that HTTP server to also expose metrics, establishing a unified observability interface for the Client.

2. Existing Metrics Infrastructure

The metrics collection layer is already in place — what's missing is only the HTTP exposure:

ClientMetric (always compiled, no feature flag needed) tracks:
- Transfer layer: mooncake_transfer_{read,write}_bytes, mooncake_transfer_{put,get,batch_put,batch_get}_latency
- RPC layer: mooncake_client_rpc_count, mooncake_client_rpc_latency (dynamic rpc_name label)
- Supports cluster_id static label via MC_STORE_CLUSTER_ID env var
ClientMetric::serialize() already outputs Prometheus text format
ClientMetric::summary_metrics() already outputs a human-readable summary

3. Design

3.1 Overview

Extend the existing coro_http_server in RealClient (introduced by the health-check PR) to serve metrics alongside health status:

graph TB
    subgraph "RealClient Process"
        RC[RealClient]
        HTTP["coro_http_server"]
        CM["ClientMetric\n(Transfer + RPC metrics)"]
        HC[health_check]

        RC --> CM
        RC --> HC
        HTTP -->|"GET /health"| HC
        HTTP -->|"GET /metrics"| CM
        HTTP -->|"GET /metrics/summary"| CM
    end

    Prom["Prometheus"] -->|"pull /metrics"| HTTP
    K8s["K8s Probe"] -->|"GET /health"| HTTP
    Ops["Operator"] -->|"GET /metrics/summary"| HTTP

3.2 HTTP Endpoints

Endpoint	Method	Content-Type	Description
`/health`	GET	`application/json`	Health check (already implemented)
`/metrics`	GET	`text/plain`	Prometheus text format for scraping
`/metrics/summary`	GET	`text/plain`	Human-readable summary for debugging

The HTTP server is disabled by default and opt-in via configuration, maintaining backward compatibility with existing deployments.

3.3 DummyClient Considerations [Optional]

DummyClient runs inside the inference framework process and should not open its own HTTP port. Instead:

Via RealClient: Since RealClient is the hub for all connected DummyClient instances, it can aggregate and expose metrics on their behalf.
Via Python binding: Expose get_metrics() in the Python layer so inference frameworks (vLLM, SGLang) can integrate Client metrics into their own observability stack.

4. Open Questions

DummyClient aggregation: Should RealClient aggregate metrics from all connected DummyClient instances, or report only its own?

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @duhaode520 on GitHub (Mar 4, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1605 Originally assigned to: @duhaode520 on GitHub. ## 1. Background & Motivation With the introduction of `DummyClient`, the Mooncake Store client architecture is evolving toward a **standalone deployment model**. Previously, the Store client was tightly coupled with the inference framework process (e.g., vLLM, SGLang). Now, `RealClient` can run as an independent long-lived process, while one or more `DummyClient` instances connect to it from inference workers via RPC. This shift makes `RealClient` a first-class infrastructure component — similar to a sidecar or daemon — that needs proper observability and lifecycle management, just like the Master service. However, while the Master already has an embedded HTTP server exposing `/metrics` and management endpoints, the Client side lacks equivalent capability: | Gap | Impact | |-----|--------| | No HTTP health-check endpoint | K8s liveness/readiness probes cannot directly monitor Client status | | No HTTP metrics exposure | `ClientMetric` already collects rich transfer/RPC metrics, but they are only accessible via logs — Prometheus cannot pull them | The current PR (`feat/mooncake-health-check`) adds a `/health` endpoint to `RealClient`. This RFC proposes extending that HTTP server to also expose metrics, establishing a unified observability interface for the Client. ## 2. Existing Metrics Infrastructure The metrics collection layer is already in place — what's missing is only the HTTP exposure: - **`ClientMetric`** (always compiled, no feature flag needed) tracks: - Transfer layer: `mooncake_transfer_{read,write}_bytes`, `mooncake_transfer_{put,get,batch_put,batch_get}_latency` - RPC layer: `mooncake_client_rpc_count`, `mooncake_client_rpc_latency` (dynamic `rpc_name` label) - Supports `cluster_id` static label via `MC_STORE_CLUSTER_ID` env var - **`ClientMetric::serialize()`** already outputs Prometheus text format - **`ClientMetric::summary_metrics()`** already outputs a human-readable summary ## 3. Design ### 3.1 Overview Extend the existing `coro_http_server` in `RealClient` (introduced by the health-check PR) to serve metrics alongside health status: ```mermaid graph TB subgraph "RealClient Process" RC[RealClient] HTTP["coro_http_server"] CM["ClientMetric\n(Transfer + RPC metrics)"] HC[health_check] RC --> CM RC --> HC HTTP -->|"GET /health"| HC HTTP -->|"GET /metrics"| CM HTTP -->|"GET /metrics/summary"| CM end Prom["Prometheus"] -->|"pull /metrics"| HTTP K8s["K8s Probe"] -->|"GET /health"| HTTP Ops["Operator"] -->|"GET /metrics/summary"| HTTP ``` ### 3.2 HTTP Endpoints | Endpoint | Method | Content-Type | Description | |----------|--------|-------------|-------------| | `/health` | GET | `application/json` | Health check (already implemented) | | `/metrics` | GET | `text/plain` | Prometheus text format for scraping | | `/metrics/summary` | GET | `text/plain` | Human-readable summary for debugging | The HTTP server is **disabled by default** and opt-in via configuration, maintaining backward compatibility with existing deployments. ### 3.3 DummyClient Considerations [Optional] `DummyClient` runs inside the inference framework process and should **not** open its own HTTP port. Instead: - **Via RealClient**: Since `RealClient` is the hub for all connected `DummyClient` instances, it can aggregate and expose metrics on their behalf. - **Via Python binding**: Expose `get_metrics()` in the Python layer so inference frameworks (vLLM, SGLang) can integrate Client metrics into their own observability stack. ## 4. Open Questions 1. **DummyClient aggregation**: Should `RealClient` aggregate metrics from all connected `DummyClient` instances, or report only its own? ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:15:11 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:15:12 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 9, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jun 9, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:15:12 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 16, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jun 16, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.