[PR #1130] [MERGED] feat(metrics): add task completion latency tracking and reporting #2031

New issue

Closed

opened 2026-10-29 21:32:15 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:32:15 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1130
Author: @staryxchen
Created: 11/27/2025
Status: ✅ Merged
Merged: 12/17/2025
Merged by: @stmatengss

Base: main ← Head: feat/add-task-latency-metrics

📝 Commits (3)

2f4050b feat(metrics): add task completion latency tracking and reporting
013237a refactor(metrics): improve latency distribution logging logic
a648ee3 Merge branch 'kvcache-ai:main' into feat/add-task-latency-metrics

📊 Changes

2 files changed (+215 additions, -12 deletions)

View changed files

📝 mooncake-transfer-engine/include/transfer_engine.h (+80 -3)
📝 mooncake-transfer-engine/src/transfer_engine.cpp (+135 -9)

📄 Description

Add Transfer Task Latency Distribution Metrics

Summary

This PR adds histogram-based latency distribution tracking for transfer tasks in the Transfer Engine metrics system.

Motivation

Understanding task completion latency distribution is crucial for:

Performance tuning and bottleneck identification
SLA monitoring and analysis
Detecting outliers and tail latencies

Previously, only throughput metrics were available. This enhancement provides detailed visibility into task-level performance.

Changes

Core Implementation

Modified Files:

mooncake-transfer-engine/include/transfer_engine.h
mooncake-transfer-engine/src/transfer_engine.cpp

Key Components:

Latency Tracking
- Record task start time on first getTransferStatus() call
- Calculate and record completion latency when task completes
- Use ylt::metric::histogram_t for efficient distribution tracking
Histogram Buckets
- Fine-grained buckets (10μs - 10s) covering sub-millisecond to multi-second latencies
- 17 boundary values creating 18 buckets optimized for high-performance scenarios
Interval-based Reporting
- Snapshot mechanism for computing per-interval statistics
- Avoids cumulative drift, shows actual distribution within each reporting period
- Aligns with Prometheus-style metrics design (monotonic counters + rate calculation)

Output Format

[Metrics] Transfer Engine Stats (over last 5s): 
Throughput: 22480.90 MB/s | 
Latency Distribution (count=56215): 0-10μs:0.2%, 10-20μs:1.6%, 20-50μs:96.5%, 50-100μs:0.9%, 100-200μs:0.8%

Features:

Only shows buckets with ≥0.1% to reduce noise
Displays task count and percentage distribution
Single-line format for easy parsing and monitoring

Usage

# Enable metrics (required)
export MC_TE_METRIC=1

# Optional: customize interval (default: 5s)
export MC_TE_METRIC_INTERVAL_SECONDS=10

Design Decisions

Histogram over avg/max: Distribution provides complete picture; avg can be skewed by tail latencies, max is unstable
Snapshot mechanism: histogram_t doesn't support reset; snapshot-based interval calculation is standard practice (Prometheus)
Bucket granularity: Focused on sub-ms to catch high-performance variations, with coarser bins for tail

Performance Impact

Minimal overhead: only enabled with WITH_METRICS compile flag
Lock-free histogram updates
Automatic cleanup of completed task timestamps

Testing Instructions:

# Build with metrics
cmake -DWITH_METRICS=ON ..
make

# Run and observe output
export MC_TE_METRIC=1
./your_test_binary

Expected output will show latency distribution every 5 seconds alongside throughput metrics.

Type of Change

Types
- Bug fix
- New feature
  - Transfer Engine
  - Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- Documentation update
- Other

How Has This Been Tested?

✅ Compiles with WITH_METRICS enabled/disabled
✅ No memory leaks (completed tasks are cleaned up)
✅ Thread-safe concurrent task tracking
✅ Accurate distribution reporting under load

Checklist

I have performed a self-review of my own code.
I have updated the documentation.
I have added tests to prove my changes are effective.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1130 **Author:** [@staryxchen](https://github.com/staryxchen) **Created:** 11/27/2025 **Status:** ✅ Merged **Merged:** 12/17/2025 **Merged by:** [@stmatengss](https://github.com/stmatengss) **Base:** `main` ← **Head:** `feat/add-task-latency-metrics` --- ### 📝 Commits (3) - [`2f4050b`](https://github.com/kvcache-ai/Mooncake/commit/2f4050b8061f09054ca4d4eb5217642251b59297) feat(metrics): add task completion latency tracking and reporting - [`013237a`](https://github.com/kvcache-ai/Mooncake/commit/013237aca07aab556d8aa7b3272cd79775a75364) refactor(metrics): improve latency distribution logging logic - [`a648ee3`](https://github.com/kvcache-ai/Mooncake/commit/a648ee37113dfc15625ff595ce6bb038401f214a) Merge branch 'kvcache-ai:main' into feat/add-task-latency-metrics ### 📊 Changes **2 files changed** (+215 additions, -12 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-transfer-engine/include/transfer_engine.h` (+80 -3) 📝 `mooncake-transfer-engine/src/transfer_engine.cpp` (+135 -9) </details> ### 📄 Description # Add Transfer Task Latency Distribution Metrics ## Summary This PR adds histogram-based latency distribution tracking for transfer tasks in the Transfer Engine metrics system. ## Motivation Understanding task completion latency distribution is crucial for: - Performance tuning and bottleneck identification - SLA monitoring and analysis - Detecting outliers and tail latencies Previously, only throughput metrics were available. This enhancement provides detailed visibility into task-level performance. ## Changes ### Core Implementation **Modified Files:** - `mooncake-transfer-engine/include/transfer_engine.h` - `mooncake-transfer-engine/src/transfer_engine.cpp` **Key Components:** 1. **Latency Tracking** - Record task start time on first `getTransferStatus()` call - Calculate and record completion latency when task completes - Use `ylt::metric::histogram_t` for efficient distribution tracking 2. **Histogram Buckets** - Fine-grained buckets (10μs - 10s) covering sub-millisecond to multi-second latencies - 17 boundary values creating 18 buckets optimized for high-performance scenarios 3. **Interval-based Reporting** - Snapshot mechanism for computing per-interval statistics - Avoids cumulative drift, shows actual distribution within each reporting period - Aligns with Prometheus-style metrics design (monotonic counters + rate calculation) ### Output Format ``` [Metrics] Transfer Engine Stats (over last 5s): Throughput: 22480.90 MB/s | Latency Distribution (count=56215): 0-10μs:0.2%, 10-20μs:1.6%, 20-50μs:96.5%, 50-100μs:0.9%, 100-200μs:0.8% ``` **Features:** - Only shows buckets with ≥0.1% to reduce noise - Displays task count and percentage distribution - Single-line format for easy parsing and monitoring ## Usage ```bash # Enable metrics (required) export MC_TE_METRIC=1 # Optional: customize interval (default: 5s) export MC_TE_METRIC_INTERVAL_SECONDS=10 ``` ## Design Decisions 1. **Histogram over avg/max**: Distribution provides complete picture; avg can be skewed by tail latencies, max is unstable 2. **Snapshot mechanism**: `histogram_t` doesn't support reset; snapshot-based interval calculation is standard practice (Prometheus) 3. **Bucket granularity**: Focused on sub-ms to catch high-performance variations, with coarser bins for tail ## Performance Impact - Minimal overhead: only enabled with `WITH_METRICS` compile flag - Lock-free histogram updates - Automatic cleanup of completed task timestamps --- **Testing Instructions:** ```bash # Build with metrics cmake -DWITH_METRICS=ON .. make # Run and observe output export MC_TE_METRIC=1 ./your_test_binary ``` Expected output will show latency distribution every 5 seconds alongside throughput metrics.  ## Type of Change * Types - [ ] Bug fix - [x] New feature - [x] Transfer Engine - [ ] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [ ] CI/CD - [ ] Documentation update - [ ] Other ## How Has This Been Tested? - ✅ Compiles with `WITH_METRICS` enabled/disabled - ✅ No memory leaks (completed tasks are cleaned up) - ✅ Thread-safe concurrent task tracking - ✅ Accurate distribution reporting under load  ## Checklist - [x] I have performed a self-review of my own code. - [x] I have updated the documentation. - [x] I have added tests to prove my changes are effective. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:32:15 +00:00