[PR #1068] [MERGED] [Store] MasterMetricManager Returns Zero-Value Variables #1990

New issue

Closed

opened 2026-10-29 21:31:57 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:31:57 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1068
Author: @ykwd
Created: 11/17/2025
Status: ✅ Merged
Merged: 11/20/2025
Merged by: @stmatengss

Base: main ← Head: dev/metrics-report-zero

📝 Commits (3)

ec70d63 Remove ha check in master metrics
1cba935 Adjust initialization ordering to resolve the following warnings: warning: ‘mooncake::MasterMetricManager::total_get_nums_’ will be initialized after [-Wreorder] ylt::metric::counter_t total_get_nums_;
7998377 master metrics output zero value variable

📊 Changes

4 files changed (+124 additions, -41 deletions)

View changed files

📝 mooncake-store/include/master_metric_manager.h (+3 -7)
📝 mooncake-store/src/master_metric_manager.cpp (+115 -32)
📝 mooncake-store/src/rpc_service.cpp (+0 -2)
📝 mooncake-store/tests/master_metrics_test.cpp (+6 -0)

📄 Description

Description

This PR includes the following changes:

MasterMetricManager now also returns zero-value variables to Prometheus.
For the client- and ping-related metrics in MasterMetricManager, it is no longer necessary to check whether the system is in HA mode.
Updated some code that previously caused compilation warnings.

After updation, the returned string is:

# HELP master_allocated_bytes Total memory bytes currently allocated across all segments
# TYPE master_allocated_bytes gauge
master_allocated_bytes 0
# HELP master_total_capacity_bytes Total memory capacity across all mounted segments
# TYPE master_total_capacity_bytes gauge
master_total_capacity_bytes 16777216
# HELP segment_allocated_bytes Total memory bytes currently allocated of the segment
# TYPE segment_allocated_bytes gauge
segment_allocated_bytes{segment="test_segment"} 0
# HELP segment_total_capacity_bytes Total memory capacity of the mounted segment
# TYPE segment_total_capacity_bytes gauge
segment_total_capacity_bytes{segment="test_segment"} 16777216
# HELP master_allocated_file_size_bytes Total bytes currently allocated for file storage in 3fs/nfs
# TYPE master_allocated_file_size_bytes gauge
master_allocated_file_size_bytes 0
# HELP master_total_file_capacity_bytes Total capacity for file storage in 3fs/nfs
# TYPE master_total_file_capacity_bytes gauge
master_total_file_capacity_bytes 0
# HELP master_key_count Total number of keys managed by the master
# TYPE master_key_count gauge
master_key_count 0
# HELP master_soft_pin_key_count Total number of soft-pinned keys managed by the master
# TYPE master_soft_pin_key_count gauge
master_soft_pin_key_count 0
# HELP master_active_clients Total number of active clients
# TYPE master_active_clients gauge
master_active_clients 0
# HELP master_value_size_bytes Distribution of object value sizes
# TYPE master_value_size_bytes histogram
master_value_size_bytes_bucket{le="4096.000000"} 6
master_value_size_bytes_bucket{le="65536.000000"} 6
master_value_size_bytes_bucket{le="262144.000000"} 6
master_value_size_bytes_bucket{le="1048576.000000"} 6
master_value_size_bytes_bucket{le="4194304.000000"} 6
master_value_size_bytes_bucket{le="16777216.000000"} 6
master_value_size_bytes_bucket{le="67108864.000000"} 6
master_value_size_bytes_bucket{le="+Inf"} 6
master_value_size_bytes_sum 5120
master_value_size_bytes_count 6
# HELP master_exist_key_requests_total Total number of ExistKey requests received
# TYPE master_exist_key_requests_total counter
master_exist_key_requests_total 1
# HELP master_exist_key_failures_total Total number of failed ExistKey requests
# TYPE master_exist_key_failures_total counter
master_exist_key_failures_total 0
# HELP master_put_start_requests_total Total number of PutStart requests received
# TYPE master_put_start_requests_total counter
master_put_start_requests_total 5
# HELP master_put_start_failures_total Total number of failed PutStart requests
# TYPE master_put_start_failures_total counter
master_put_start_failures_total 0
# HELP master_put_end_requests_total Total number of PutEnd requests received
# TYPE master_put_end_requests_total counter
master_put_end_requests_total 4
# HELP master_put_end_failures_total Total number of failed PutEnd requests
# TYPE master_put_end_failures_total counter
master_put_end_failures_total 0
# HELP master_put_revoke_requests_total Total number of PutRevoke requests received
# TYPE master_put_revoke_requests_total counter
master_put_revoke_requests_total 1
# HELP master_put_revoke_failures_total Total number of failed PutRevoke requests
# TYPE master_put_revoke_failures_total counter
master_put_revoke_failures_total 0
# HELP master_get_replica_list_requests_total Total number of GetReplicaList requests received
# TYPE master_get_replica_list_requests_total counter
master_get_replica_list_requests_total 2
# HELP master_get_replica_list_failures_total Total number of failed GetReplicaList requests
# TYPE master_get_replica_list_failures_total counter
master_get_replica_list_failures_total 0
# HELP master_get_replica_list_by_regex_requests_total Total number of GetReplicaListByRegex requests received
# TYPE master_get_replica_list_by_regex_requests_total counter
master_get_replica_list_by_regex_requests_total 0
# HELP master_get_replica_list_by_regex_failures_total Total number of failed GetReplicaListByRegex requests
# TYPE master_get_replica_list_by_regex_failures_total counter
master_get_replica_list_by_regex_failures_total 0
# HELP master_remove_requests_total Total number of Remove requests received
# TYPE master_remove_requests_total counter
master_remove_requests_total 2
# HELP master_remove_failures_total Total number of failed Remove requests
# TYPE master_remove_failures_total counter
master_remove_failures_total 1
# HELP master_remove_by_regex_requests_total Total number of RemoveByRegex requests received
# TYPE master_remove_by_regex_requests_total counter
master_remove_by_regex_requests_total 0
# HELP master_remove_by_regex_failures_total Total number of failed RemoveByRegex requests
# TYPE master_remove_by_regex_failures_total counter
master_remove_by_regex_failures_total 0
# HELP master_remove_all_requests_total Total number of Remove all requests received
# TYPE master_remove_all_requests_total counter
master_remove_all_requests_total 1
# HELP master_remove_all_failures_total Total number of failed Remove all requests
# TYPE master_remove_all_failures_total counter
master_remove_all_failures_total 0
# HELP master_mount_segment_requests_total Total number of MountSegment requests received
# TYPE master_mount_segment_requests_total counter
master_mount_segment_requests_total 3
# HELP master_mount_segment_failures_total Total number of failed MountSegment requests
# TYPE master_mount_segment_failures_total counter
master_mount_segment_failures_total 0
# HELP master_unmount_segment_requests_total Total number of UnmountSegment requests received
# TYPE master_unmount_segment_requests_total counter
master_unmount_segment_requests_total 1
# HELP master_unmount_segment_failures_total Total number of failed UnmountSegment requests
# TYPE master_unmount_segment_failures_total counter
master_unmount_segment_failures_total 0
# HELP master_remount_segment_requests_total Total number of RemountSegment requests received
# TYPE master_remount_segment_requests_total counter
master_remount_segment_requests_total 0
# HELP master_remount_segment_failures_total Total number of failed RemountSegment requests
# TYPE master_remount_segment_failures_total counter
master_remount_segment_failures_total 0
# HELP master_ping_requests_total Total number of ping requests received
# TYPE master_ping_requests_total counter
master_ping_requests_total 0
# HELP master_ping_failures_total Total number of failed ping requests
# TYPE master_ping_failures_total counter
master_ping_failures_total 0
# HELP master_batch_exist_key_requests_total Total number of BatchExistKey requests received
# TYPE master_batch_exist_key_requests_total counter
master_batch_exist_key_requests_total 0
# HELP master_batch_exist_key_failures_total Total number of failed BatchExistKey requests
# TYPE master_batch_exist_key_failures_total counter
master_batch_exist_key_failures_total 0
# HELP master_batch_get_replica_list_requests_total Total number of BatchGetReplicaList requests received
# TYPE master_batch_get_replica_list_requests_total counter
master_batch_get_replica_list_requests_total 0
# HELP master_batch_get_replica_list_failures_total Total number of failed BatchGetReplicaList requests
# TYPE master_batch_get_replica_list_failures_total counter
master_batch_get_replica_list_failures_total 0
# HELP master_batch_put_start_requests_total Total number of BatchPutStart requests received
# TYPE master_batch_put_start_requests_total counter
master_batch_put_start_requests_total 0
# HELP master_batch_put_start_failures_total Total number of failed BatchPutStart requests
# TYPE master_batch_put_start_failures_total counter
master_batch_put_start_failures_total 0
# HELP master_batch_put_end_requests_total Total number of BatchPutEnd requests received
# TYPE master_batch_put_end_requests_total counter
master_batch_put_end_requests_total 0
# HELP master_batch_put_end_failures_total Total number of failed BatchPutEnd requests
# TYPE master_batch_put_end_failures_total counter
master_batch_put_end_failures_total 0
# HELP master_batch_put_revoke_requests_total Total number of BatchPutRevoke requests received
# TYPE master_batch_put_revoke_requests_total counter
master_batch_put_revoke_requests_total 0
# HELP master_batch_put_revoke_failures_total Total number of failed BatchPutRevoke requests
# TYPE master_batch_put_revoke_failures_total counter
master_batch_put_revoke_failures_total 0
# HELP master_successful_evictions_total Total number of successful eviction operations
# TYPE master_successful_evictions_total counter
master_successful_evictions_total 0
# HELP master_attempted_evictions_total Total number of attempted eviction operations
# TYPE master_attempted_evictions_total counter
master_attempted_evictions_total 0
# HELP master_evicted_key_count Total number of keys evicted
# TYPE master_evicted_key_count counter
master_evicted_key_count 0
# HELP master_evicted_size_bytes Total bytes of evicted objects
# TYPE master_evicted_size_bytes counter
master_evicted_size_bytes 0
# HELP master_put_start_discard_cnt Total number of discarded PutStart operations
# TYPE master_put_start_discard_cnt counter
master_put_start_discard_cnt 0
# HELP master_put_start_release_cnt Total number of released PutStart operations
# TYPE master_put_start_release_cnt counter
master_put_start_release_cnt 0
# HELP master_put_start_discarded_staging_size Total size of memory replicas in discarded but not yet released PutStart operations
# TYPE master_put_start_discarded_staging_size gauge
master_put_start_discarded_staging_size 0

Type of Change

Types
- Bug fix
- New feature
  - Transfer Engine
  - Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- Documentation update
- Other

How Has This Been Tested?

Checklist

I have performed a self-review of my own code.
I have updated the documentation.
I have added tests to prove my changes are effective.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1068 **Author:** [@ykwd](https://github.com/ykwd) **Created:** 11/17/2025 **Status:** ✅ Merged **Merged:** 11/20/2025 **Merged by:** [@stmatengss](https://github.com/stmatengss) **Base:** `main` ← **Head:** `dev/metrics-report-zero` --- ### 📝 Commits (3) - [`ec70d63`](https://github.com/kvcache-ai/Mooncake/commit/ec70d63fdc57a4d62be08fcf60149d9f6a3e6075) Remove ha check in master metrics - [`1cba935`](https://github.com/kvcache-ai/Mooncake/commit/1cba9350da6c169c6c938c3c8f780ca9f1317e47) Adjust initialization ordering to resolve the following warnings: warning: ‘mooncake::MasterMetricManager::total_get_nums_’ will be initialized after [-Wreorder] ylt::metric::counter_t total_get_nums_; - [`7998377`](https://github.com/kvcache-ai/Mooncake/commit/7998377028faa53154072a2df99bd0c2b16ce352) master metrics output zero value variable ### 📊 Changes **4 files changed** (+124 additions, -41 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-store/include/master_metric_manager.h` (+3 -7) 📝 `mooncake-store/src/master_metric_manager.cpp` (+115 -32) 📝 `mooncake-store/src/rpc_service.cpp` (+0 -2) 📝 `mooncake-store/tests/master_metrics_test.cpp` (+6 -0) </details> ### 📄 Description ## Description  This PR includes the following changes: 1. **MasterMetricManager** now also returns zero-value variables to Prometheus. 2. For the client- and ping-related metrics in **MasterMetricManager**, it is no longer necessary to check whether the system is in HA mode. 3. Updated some code that previously caused compilation warnings. After updation, the returned string is: ``` # HELP master_allocated_bytes Total memory bytes currently allocated across all segments # TYPE master_allocated_bytes gauge master_allocated_bytes 0 # HELP master_total_capacity_bytes Total memory capacity across all mounted segments # TYPE master_total_capacity_bytes gauge master_total_capacity_bytes 16777216 # HELP segment_allocated_bytes Total memory bytes currently allocated of the segment # TYPE segment_allocated_bytes gauge segment_allocated_bytes{segment="test_segment"} 0 # HELP segment_total_capacity_bytes Total memory capacity of the mounted segment # TYPE segment_total_capacity_bytes gauge segment_total_capacity_bytes{segment="test_segment"} 16777216 # HELP master_allocated_file_size_bytes Total bytes currently allocated for file storage in 3fs/nfs # TYPE master_allocated_file_size_bytes gauge master_allocated_file_size_bytes 0 # HELP master_total_file_capacity_bytes Total capacity for file storage in 3fs/nfs # TYPE master_total_file_capacity_bytes gauge master_total_file_capacity_bytes 0 # HELP master_key_count Total number of keys managed by the master # TYPE master_key_count gauge master_key_count 0 # HELP master_soft_pin_key_count Total number of soft-pinned keys managed by the master # TYPE master_soft_pin_key_count gauge master_soft_pin_key_count 0 # HELP master_active_clients Total number of active clients # TYPE master_active_clients gauge master_active_clients 0 # HELP master_value_size_bytes Distribution of object value sizes # TYPE master_value_size_bytes histogram master_value_size_bytes_bucket{le="4096.000000"} 6 master_value_size_bytes_bucket{le="65536.000000"} 6 master_value_size_bytes_bucket{le="262144.000000"} 6 master_value_size_bytes_bucket{le="1048576.000000"} 6 master_value_size_bytes_bucket{le="4194304.000000"} 6 master_value_size_bytes_bucket{le="16777216.000000"} 6 master_value_size_bytes_bucket{le="67108864.000000"} 6 master_value_size_bytes_bucket{le="+Inf"} 6 master_value_size_bytes_sum 5120 master_value_size_bytes_count 6 # HELP master_exist_key_requests_total Total number of ExistKey requests received # TYPE master_exist_key_requests_total counter master_exist_key_requests_total 1 # HELP master_exist_key_failures_total Total number of failed ExistKey requests # TYPE master_exist_key_failures_total counter master_exist_key_failures_total 0 # HELP master_put_start_requests_total Total number of PutStart requests received # TYPE master_put_start_requests_total counter master_put_start_requests_total 5 # HELP master_put_start_failures_total Total number of failed PutStart requests # TYPE master_put_start_failures_total counter master_put_start_failures_total 0 # HELP master_put_end_requests_total Total number of PutEnd requests received # TYPE master_put_end_requests_total counter master_put_end_requests_total 4 # HELP master_put_end_failures_total Total number of failed PutEnd requests # TYPE master_put_end_failures_total counter master_put_end_failures_total 0 # HELP master_put_revoke_requests_total Total number of PutRevoke requests received # TYPE master_put_revoke_requests_total counter master_put_revoke_requests_total 1 # HELP master_put_revoke_failures_total Total number of failed PutRevoke requests # TYPE master_put_revoke_failures_total counter master_put_revoke_failures_total 0 # HELP master_get_replica_list_requests_total Total number of GetReplicaList requests received # TYPE master_get_replica_list_requests_total counter master_get_replica_list_requests_total 2 # HELP master_get_replica_list_failures_total Total number of failed GetReplicaList requests # TYPE master_get_replica_list_failures_total counter master_get_replica_list_failures_total 0 # HELP master_get_replica_list_by_regex_requests_total Total number of GetReplicaListByRegex requests received # TYPE master_get_replica_list_by_regex_requests_total counter master_get_replica_list_by_regex_requests_total 0 # HELP master_get_replica_list_by_regex_failures_total Total number of failed GetReplicaListByRegex requests # TYPE master_get_replica_list_by_regex_failures_total counter master_get_replica_list_by_regex_failures_total 0 # HELP master_remove_requests_total Total number of Remove requests received # TYPE master_remove_requests_total counter master_remove_requests_total 2 # HELP master_remove_failures_total Total number of failed Remove requests # TYPE master_remove_failures_total counter master_remove_failures_total 1 # HELP master_remove_by_regex_requests_total Total number of RemoveByRegex requests received # TYPE master_remove_by_regex_requests_total counter master_remove_by_regex_requests_total 0 # HELP master_remove_by_regex_failures_total Total number of failed RemoveByRegex requests # TYPE master_remove_by_regex_failures_total counter master_remove_by_regex_failures_total 0 # HELP master_remove_all_requests_total Total number of Remove all requests received # TYPE master_remove_all_requests_total counter master_remove_all_requests_total 1 # HELP master_remove_all_failures_total Total number of failed Remove all requests # TYPE master_remove_all_failures_total counter master_remove_all_failures_total 0 # HELP master_mount_segment_requests_total Total number of MountSegment requests received # TYPE master_mount_segment_requests_total counter master_mount_segment_requests_total 3 # HELP master_mount_segment_failures_total Total number of failed MountSegment requests # TYPE master_mount_segment_failures_total counter master_mount_segment_failures_total 0 # HELP master_unmount_segment_requests_total Total number of UnmountSegment requests received # TYPE master_unmount_segment_requests_total counter master_unmount_segment_requests_total 1 # HELP master_unmount_segment_failures_total Total number of failed UnmountSegment requests # TYPE master_unmount_segment_failures_total counter master_unmount_segment_failures_total 0 # HELP master_remount_segment_requests_total Total number of RemountSegment requests received # TYPE master_remount_segment_requests_total counter master_remount_segment_requests_total 0 # HELP master_remount_segment_failures_total Total number of failed RemountSegment requests # TYPE master_remount_segment_failures_total counter master_remount_segment_failures_total 0 # HELP master_ping_requests_total Total number of ping requests received # TYPE master_ping_requests_total counter master_ping_requests_total 0 # HELP master_ping_failures_total Total number of failed ping requests # TYPE master_ping_failures_total counter master_ping_failures_total 0 # HELP master_batch_exist_key_requests_total Total number of BatchExistKey requests received # TYPE master_batch_exist_key_requests_total counter master_batch_exist_key_requests_total 0 # HELP master_batch_exist_key_failures_total Total number of failed BatchExistKey requests # TYPE master_batch_exist_key_failures_total counter master_batch_exist_key_failures_total 0 # HELP master_batch_get_replica_list_requests_total Total number of BatchGetReplicaList requests received # TYPE master_batch_get_replica_list_requests_total counter master_batch_get_replica_list_requests_total 0 # HELP master_batch_get_replica_list_failures_total Total number of failed BatchGetReplicaList requests # TYPE master_batch_get_replica_list_failures_total counter master_batch_get_replica_list_failures_total 0 # HELP master_batch_put_start_requests_total Total number of BatchPutStart requests received # TYPE master_batch_put_start_requests_total counter master_batch_put_start_requests_total 0 # HELP master_batch_put_start_failures_total Total number of failed BatchPutStart requests # TYPE master_batch_put_start_failures_total counter master_batch_put_start_failures_total 0 # HELP master_batch_put_end_requests_total Total number of BatchPutEnd requests received # TYPE master_batch_put_end_requests_total counter master_batch_put_end_requests_total 0 # HELP master_batch_put_end_failures_total Total number of failed BatchPutEnd requests # TYPE master_batch_put_end_failures_total counter master_batch_put_end_failures_total 0 # HELP master_batch_put_revoke_requests_total Total number of BatchPutRevoke requests received # TYPE master_batch_put_revoke_requests_total counter master_batch_put_revoke_requests_total 0 # HELP master_batch_put_revoke_failures_total Total number of failed BatchPutRevoke requests # TYPE master_batch_put_revoke_failures_total counter master_batch_put_revoke_failures_total 0 # HELP master_successful_evictions_total Total number of successful eviction operations # TYPE master_successful_evictions_total counter master_successful_evictions_total 0 # HELP master_attempted_evictions_total Total number of attempted eviction operations # TYPE master_attempted_evictions_total counter master_attempted_evictions_total 0 # HELP master_evicted_key_count Total number of keys evicted # TYPE master_evicted_key_count counter master_evicted_key_count 0 # HELP master_evicted_size_bytes Total bytes of evicted objects # TYPE master_evicted_size_bytes counter master_evicted_size_bytes 0 # HELP master_put_start_discard_cnt Total number of discarded PutStart operations # TYPE master_put_start_discard_cnt counter master_put_start_discard_cnt 0 # HELP master_put_start_release_cnt Total number of released PutStart operations # TYPE master_put_start_release_cnt counter master_put_start_release_cnt 0 # HELP master_put_start_discarded_staging_size Total size of memory replicas in discarded but not yet released PutStart operations # TYPE master_put_start_discarded_staging_size gauge master_put_start_discarded_staging_size 0 ``` ## Type of Change * Types - [ ] Bug fix - [ ] New feature - [ ] Transfer Engine - [x] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [ ] CI/CD - [ ] Documentation update - [ ] Other ## How Has This Been Tested?  ## Checklist - [x] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:31:57 +00:00