[GH-ISSUE #2318] [Monitoring]: Grafana dashboard has only 1 panel despite dozens of exported metrics #632

New issue

Closed

opened 2026-10-29 21:17:26 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:17:26 +00:00

Owner

Originally created by @jfeng18 on GitHub (Jun 3, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2318

Originally assigned to: @stmatengss on GitHub.

Description

The Grafana dashboard at monitoring/grafana/dashboards/mooncake.json contains only a single panel (rate(mooncake_master_rpc_requests_total[5m])), despite MasterMetricManager exporting 100+ Prometheus metrics covering:

Memory/SSD/NoF storage capacity and utilization
Put/Get/Remove request rates and failure rates
Cache hit counters and cached object counts
Eviction rates and evicted sizes
L2→L1 promotion pipeline (in-flight, admitted, completed, rejected)
Value size distribution histogram
Snapshot success/failure
NoF heartbeat health
Segment mount/unmount lifecycle
PutStart discard/release staging

None of these are visualized.

Impact

Operators deploying Mooncake Store have no out-of-the-box visibility into storage utilization, request health, eviction pressure, or promotion pipeline status. They must manually craft Prometheus queries.

Expected behavior

The dashboard should provide a comprehensive overview of the most important operational metrics, organized into logical sections.

Originally created by @jfeng18 on GitHub (Jun 3, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2318 Originally assigned to: @stmatengss on GitHub. ### Description The Grafana dashboard at `monitoring/grafana/dashboards/mooncake.json` contains only a single panel (`rate(mooncake_master_rpc_requests_total[5m])`), despite `MasterMetricManager` exporting 100+ Prometheus metrics covering: - Memory/SSD/NoF storage capacity and utilization - Put/Get/Remove request rates and failure rates - Cache hit counters and cached object counts - Eviction rates and evicted sizes - L2→L1 promotion pipeline (in-flight, admitted, completed, rejected) - Value size distribution histogram - Snapshot success/failure - NoF heartbeat health - Segment mount/unmount lifecycle - PutStart discard/release staging None of these are visualized. ### Impact Operators deploying Mooncake Store have no out-of-the-box visibility into storage utilization, request health, eviction pressure, or promotion pipeline status. They must manually craft Prometheus queries. ### Expected behavior The dashboard should provide a comprehensive overview of the most important operational metrics, organized into logical sections.

bsakaguchi closed this issue

2026-10-29 21:17:27 +00:00

bsakaguchi commented

2026-10-29 21:17:27 +00:00

Author

Owner

@jfeng18 commented on GitHub (Aug 21, 2026):

Resolved by #2944, which expanded the dashboard to 51 panels covering the metric families listed here (storage utilization, request/failure rates, cache, eviction, promotion pipeline, value-size, snapshots, NoF heartbeat, segment lifecycle, staging). Closing.

Remaining nit tracked in the #2319 closing note: segment mount/unmount/remount failure counters still have no panel.

@jfeng18 commented on GitHub (Aug 21, 2026): Resolved by #2944, which expanded the dashboard to 51 panels covering the metric families listed here (storage utilization, request/failure rates, cache, eviction, promotion pipeline, value-size, snapshots, NoF heartbeat, segment lifecycle, staging). Closing. Remaining nit tracked in the #2319 closing note: segment mount/unmount/remount *failure* counters still have no panel.