[GH-ISSUE #4940] [RFC]: [TENT] End-to-end production observability contract #1255

New issue

Open

opened 2026-10-29 21:24:29 +00:00 by bsakaguchi · 11 comments

bsakaguchi commented 2026-10-29 21:24:29 +00:00

Owner

Copy link

Originally created by @zxpdemonio on GitHub (Oct 26, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4940

Changes proposed

Summary

This RFC defines a production-oriented observability contract for TENT across
four connected surfaces:

1. bounded Prometheus metrics for aggregation and alerting;
2. structured failure/slow-path logs with stable correlation identifiers;
3. compatible liveness plus meaningful readiness and redacted diagnostics;
4. sampled, process-local trace spans linked from OpenMetrics exemplars.

The complete implementation is staged as 24 linear commits in draft integration
PR #4939, rebased on community main at b46254de. Stage 0 is independently
reviewable in #4938. Later review PRs will be extracted from the integration
stack rather than merging #4939 monolithically.

The operating objective is concrete: during an incident, an operator should be
able to start from an alert or symptom, identify the failing phase and
transport, retrieve a bounded diagnostic snapshot, and pivot from a metric to
the correlated request/attempt logs or retained trace without adding
high-cardinality metric labels.

Status

• All planned implementation stages are present in #4939.
• The original metrics-enabled full build and 20-test cross-stage regression
  set pass. The four review-driven follow-up stages also pass their focused
  metrics-on and metrics-off build/test matrices.
• Dashboard JSON and alert YAML parse successfully; the final dashboard has 20
  panels and the alert bundle has 18 rules.
• Production-topology performance acceptance and alert-threshold tuning remain
  work for the extracted review PRs.

Motivation

TENT already exposed request, byte, latency, attempt, and stage metrics, but an
operator could not reliably answer several basic questions:

• Is the metrics pipeline compiled in, enabled, bound, and serializing?
• Did a logical request fail, or did a physical attempt fail and recover through
  failover?
• Was the fault in metadata, admission, selection, connection, registration,
  submission, queueing, posting, polling, completion, retry, notification,
  cleanup, or shutdown?
• Is an idle worker healthy, or is it stalled while work remains outstanding?
• Which bounded subsystem signal should lead to the detailed request evidence?
• Can a latency sample link to a retained trace without putting request IDs in
  Prometheus labels?
• Is a retry a TENT software retry, a kernel TCP retransmission, or terminal
  RDMA hardware retry exhaustion?
• Did HP-TCP reject a malformed frame, mismatch a response request ID, or
  observe a committed-length inconsistency?

Without this contract, incidents require source inspection and ad-hoc logging
before the failing subsystem can be localized.

Goals

• Define stable metric names, types, units, counting points, and bounded labels.
• Separate logical request outcomes from physical transport attempts, retries,
  and failovers.
• Cover common engine, RPC, RDMA, TCP/HP-TCP, SHM, segment, selection,
  admission, staging, and worker paths.
• Preserve /health compatibility while adding /live, /ready, and a
  versioned /diagnostics schema.
• Add stable structured event names and process-local correlation identifiers.
• Retain errors, timeouts, cancellations, and configured slow traces in addition
  to deterministic head sampling.
• Make logical-request and physical-attempt accounting independently
  checkable, and expose bounded queue/capacity and suspected-stuck signals.
• Separate software retries, TCP retransmissions, RDMA completion status, and
  protocol framing/correlation failures.
• Keep TENT_METRICS_ENABLED=OFF valid and keep tracing disabled by default.
• Ship dashboard, alert, and runbook assets with the public contract.

Non-goals

• Import an internal monitoring framework, exporter, or private metric catalog.
• Add internal-only transports or private backend state to the community API.
• Put peer IDs, addresses, request IDs, policy names, segment names, connection
  IDs, NIC names, or QP numbers in Prometheus labels.
• Emit one span per RDMA slice or completion.
• Change selection, admission, retry, failover, or transfer correctness behavior.
• Add automatic remediation or transport quarantine policy.
• Introduce wire-protocol trace propagation in this first slice.
• Add a payload checksum or other wire-format field in this stack. That needs a
  separate compatibility RFC covering negotiation, mixed versions, CPU cost,
  and failure semantics.
• Replace the operator CLI/diagnostic bundle proposed in #2864.
• Unify Classic Transfer Engine metrics from #2713/#2958.

Public metrics contract

Metrics pipeline self-observability

Stage 0 (#4938) makes the telemetry plane observable:

• tent_metrics_build_info;
• tent_metrics_runtime_enabled;
• tent_metrics_http_server_up;
• tent_metrics_http_start_failures_total;
• tent_metrics_serialization_attempts_total;
• tent_metrics_serialization_failures_total;
• tent_metrics_last_success_timestamp_seconds.

These families remain visible before transfer traffic and while runtime transfer
recording is disabled. A completely unreachable target must still be detected
with Prometheus up; an in-process metric cannot report an endpoint that cannot
be scraped.

Logical requests versus physical attempts

Existing read/write request, byte, failure, size, and latency families retain
their logical-request meaning and final-transport attribution. New physical
attempt families record the transport that actually executed each attempt:

• tent_transport_attempts_total;
• tent_transport_attempt_failures_total;
• tent_transport_attempt_latency_us;
• tent_transport_failover_total;
• tent_inflight_attempts.

A recovered RDMA-to-TCP request therefore records one successful logical TCP
outcome, one failed RDMA attempt, one successful TCP attempt, and one failover.

The accounting families are deliberately paired:

tent_logical_requests_started_total
  = sum(tent_logical_request_outcomes_total)
    + tent_logical_requests_inflight

tent_transport_attempts_total
  = sum(tent_transport_attempt_outcomes_total)
    + tent_inflight_attempts

The equalities are process-local and apply to matching operation/transport
dimensions. They make a lost terminal edge visible instead of inferring it
from legacy success/failure counters. Paired state updates remain active across
runtime sampling changes so a gauge cannot drift when only one half is seen.

Bounded failure model

Terminal failures use a shared phase/reason taxonomy. Phases are limited to:

metadata admission selection connect register submit queue post poll
completion retry failover notify cleanup shutdown

Reasons are limited to:

timeout unavailable queue_full resource_exhausted protocol stale_generation
wc_error cq_error mr_error peer_closed canceled internal

Detailed native status text remains in the owning subsystem's existing log path;
it is not copied into metric labels.

Implemented subsystem signals

• Common engine: terminal failure phase/reason and backend memory
  register/unregister operation, byte, and outcome counters.
• Worker and staging: last-progress timestamps, staging task/byte gauges,
  deferred-cleanup gauges, cleanup timeouts, and quarantined batches.
• RPC: physical method/outcome attempts, attempt latency, and actual retries.
• RDMA data path: slice lifecycle, post/completion outcomes, completion latency,
  software timeout, software retries/retry exhaustion, categorized non-success
  WC status (including retry and RNR retry exhaustion), CQ poll errors, and
  worker/posted in-flight gauges.
• RDMA lifecycle: endpoint construction/readiness/retirement/destruction, data
  and notification QP transition failures, and bounded notification events.
• TCP and HP-TCP: queue, busy-worker, admitted-task/byte and connection gauges;
  attempt, retry, timeout, peer-close, terminal outcome, queue wait, and attempt
  latency. HP-TCP additionally samples Linux TCP_INFO retransmission deltas,
  smoothed RTT, and congestion window, and classifies malformed or inconsistent
  protocol frames.
• Suspected stuck requests: an opt-in threshold emits one bounded
  phase/transport/operation counter and one correlated structured warning per
  logical request without changing request behavior.
• SHM: bounded relocation-cache/backing-object mapping events plus physical copy
  outcome and latency.
• Segment metadata: snapshot publication/mutation, registry publish/delete,
  remote fetch/cache/reset/invalidation, and generation advance.
• Selection: policy miss, empty/unavailable candidates, hint rejection or
  unavailability, selection, fallback, exhaustion, and metadata refresh.
• Admission: admit/reject/dispatch/deadline-drop/cancel events, wait latency,
  and outstanding owner/byte gauges by bounded owner kind.

The complete metric table, labels, units, and cardinality notes are in
docs/source/design/tent/metrics.md.

Liveness, readiness, and diagnostics

The endpoints share the existing metrics HTTP server:

• /health: retained as 200 OK plain-text compatibility and only proves the
  HTTP event loop is serving;
• /live: versioned JSON process liveness and remains 200 while the endpoint
  can respond;
• /ready: versioned JSON readiness, 200 only when mandatory checks pass and
  503 otherwise;
• /diagnostics: the same redacted snapshot, always 200 so it can be collected
  during an incident.

Readiness reasons are a closed set:

engine_diagnostics_unavailable
engine_unavailable
control_plane_unavailable
no_transport_available
progress_worker_unavailable
progress_worker_stalled

/ready uses only bounded aggregate state and is O(number of transport kinds),
so it is safe for frequent probes. /diagnostics intentionally retains the
more detailed O(number of live tasks) scan and must not be used as the readiness
probe.

The progress-worker stall decision requires both old outstanding work and stale
progress, so a legitimately long request does not make the engine unready while
the worker continues to advance.

Diagnostics schema version 1 contains build identity, engine/control-plane
availability, bounded loaded-transport names, progress-worker state, runtime
queue aggregates, alive batch count, and oldest in-flight age. It omits request
and peer identifiers, segment names, network and memory addresses, keys,
credentials, payloads, and raw pointers.

Provider ownership is explicit: the engine registers and clears its diagnostics
provider with an owner token, and snapshot invocation is protected against
teardown races.

Readiness and in-process metrics cannot report after the process has exited or
crashed. A missing /live/scrape target is therefore correlated with the
supervisor's exit status, stderr/startup logs, and a core dump or sanitizer
report. tent_metrics_build_info identifies the binary while it is alive; it
does not replace external crash retention.

Structured logs and correlation

The initial stable events are:

tent.request.failed
tent.request.slow
tent.request.stuck
tent.transport.attempt_failed
tent.transport.failover
tent.transport.selection_failed

They use process-local monotonic request_id, batch_id, task_id, and
attempt_id values, a 128-bit trace_id, bounded transport/operation/phase/
reason/status values, duration, failover count where applicable, and
configuration generation.

Raw BatchID pointers are never logged as correlation IDs. Tests assert that
structured events contain neither source addresses nor target offsets.
observability/slow_request_threshold_us controls slow-request logs.
observability/stuck_request_threshold_us controls the first still-pending
threshold crossing. Both default to zero (disabled). A stuck event is
diagnostic: it neither cancels a request nor proves a deadlock.

Network loss, retransmission, and protocol integrity

The public signals intentionally keep three retry layers separate:

• tent_rdma_retries_total is a TENT software slice re-enqueue;
• tent_tcp_retransmissions_total is a delta of Linux
  TCP_INFO.tcpi_total_retrans, sampled every 256 completed HP-TCP operations
  and once before connection close;
• tent_rdma_wc_errors_total{category=~"retry_exceeded|rnr_retry_exceeded"} is
  the terminal WC status after the NIC/QP hardware retry budget is exhausted.

Per-packet RDMA retransmission counts are below the verbs completion boundary
and require NIC/vendor telemetry. Classic RPC-backed TCP does not expose its
socket handle at this layer, so host TCP telemetry remains the source for that
transport.

HP-TCP protocol counters distinguish invalid size, magic, version, reserved
fields, opcode, length, and status from response request_id_mismatch and
committed_length_mismatch. TCP already provides ordered, checksummed byte
delivery; these counters diagnose framing, mixed-version, correlation, or peer
implementation faults. End-to-end payload checksums are explicitly deferred to
a child RFC because they change the wire/data path and need mixed-version and
performance design, not just observability instrumentation.

Sampled traces and OpenMetrics exemplars

Tracing is backend-neutral in this slice and is emitted as completed structured
span records:

• one tent.logical_transfer root span per logical request;
• one tent.transport_attempt child span per physical attempt;
• deterministic head sampling using
  observability/trace_sample_rate_ppm (capped at one million);
• unconditional retention for errors, timeouts, cancellations, and requests
  slower than observability/trace_slow_threshold_us;
• parent retention when a child was retained;
• bounded retention reasons: head_sampled, slow, error, timeout, and
  linked_child.

Tracing is disabled by default. Retained spans are INFO logs, so deployments
that enable tracing also set log_level=info. The implementation does not add a
tracing dependency and does not change the TENT wire protocol. Cross-process
context propagation is a compatible future extension rather than an implicit
claim of this stage.

/metrics remains Prometheus 0.0.4 compatible. The opt-in
/metrics/openmetrics endpoint uses the OpenMetrics 1.0 content type, ends with
# EOF, and can attach the latest retained trace ID to the matching
tent_transport_attempt_latency_us +Inf bucket. Exemplar publication occurs
after the corresponding histogram observation and uses a bounded fixed-size
per-series store.

Cardinality, safety, and compatibility

• Metric labels are derived only from documented enums and fixed build identity.
• Dynamic cells allocate only label combinations that actually occur, within a
  closed maximum domain.
• High-cardinality correlation IDs appear only in logs/traces, never metrics.
• Diagnostics and new structured events do not expose payloads, credentials,
  raw pointers, addresses, target offsets, or peer-provided strings.
• Existing metric names and /health behavior remain compatible.
• Metrics-off builds provide API stubs and pass the selected regression suite.
• Trace generation cannot fail a transfer, and exemplar export is optional.

Operations assets

The stack includes:

• mooncake-transfer-engine/tent/monitoring/grafana/tent-overview.json: an
  importable 20-panel dashboard for telemetry health, logical requests,
  physical attempts, latency, failover, workers, RDMA, admission, staging,
  HP-TCP socket health, protocol errors, and stuck events;
• mooncake-transfer-engine/tent/monitoring/prometheus/tent-alerts.yaml: 18
  starter alerts covering scrape/serialization health, logical and physical
  failure rates, worker stalls, staging cleanup, RDMA errors/retry exhaustion,
  admission rejection, and selection exhaustion;
• docs/source/design/tent/observability-runbook.md: a symptom-first workflow
  that connects readiness, bounded metrics, structured logs, traces, and
  exemplars.

Alert thresholds are conservative examples and must be tuned to workload SLOs
and traffic floors. The rules assume normal Prometheus job and instance
labels and document the job-selector adjustment point.

Implementation and review plan

Integration stack

Draft PR #4939 contains the complete linear integration history below. It is a
continuously tested reference stack, not the intended merge unit:

Stage	Scope	Commit(s)
0	Metrics contract/self-observability	aae72ce1, 5c969fe1, 0c20b102 (review fixes; #4938)
1	Common failure, memory, worker, staging, RPC	8da2d4d9, c0fe1f27, 8311b35d
2	RDMA data path and lifecycle	14b794f9, 2a6b2fe5
3	TCP and HP-TCP	a376597b
4	SHM and segment	ac700ccf, 5c9e77b3
5	Selection and admission	48d4d6e2, 554f5cd1, 02d3fb4a
6	Liveness/readiness/diagnostics	b5a0ae2d, 32cf582f, c2efeae0
7	Structured correlation logs	b50027e9
8	Sampled spans and exemplars	2ead741d
9	Dashboard, alerts, runbook	28e4b6af
10	Lossless logical-request and physical-attempt outcome accounting	6f75ffd5
11	O(1) readiness and stream capacity health	1a62c0f4
12	TCP_INFO, RDMA WC, and HP-TCP protocol diagnostics	cdfa0ae5
13	Suspected-stuck events and expanded operational assets	c04cda84

The commit IDs describe the current integration snapshot based on community
main at b46254de and ending at 0c20b102. They will be refreshed in this
RFC if the Draft integration branch is rebased.

Extracted review slices

The four review-driven follow-up patches complete earlier functional areas and
will be folded into those owner slices rather than submitted as four trailing
fix-up PRs. In particular, Stage 11 is divided between TCP capacity and
readiness, Stage 12 between RDMA and HP-TCP, and Stage 13 between request-state
diagnostics and operational assets.

Review slice	Reviewable scope	Integration material incorporated
R0	Metrics contract and telemetry-pipeline self-observability	Stage 0 (aae72ce1) plus its review fixes (5c969fe1, 0c20b102), already open as #4938 (1bf03834, 639ec638, 80ee17d1)
R1	Common failure/memory/worker/staging/RPC metrics plus lossless logical-request and physical-attempt accounting	Stage 1 plus Stage 10 (6f75ffd5)
R2	RDMA data-path and endpoint lifecycle observability, including categorized WC errors and hardware retry exhaustion	Stage 2 plus the RDMA part of Stage 12 (cdfa0ae5)
R3	TCP and HP-TCP attempts, queue/capacity health, Linux TCP_INFO, and protocol-integrity diagnostics	Stage 3 plus the stream-capacity part of Stage 11 (1a62c0f4) and the HP-TCP part of Stage 12 (cdfa0ae5)
R4	SHM copy and segment-metadata observability	Stage 4
R5	Transport selection and admission observability	Stage 5
R6	Liveness, bounded O(transport kinds) readiness, and redacted diagnostics	Stage 6 plus the readiness part of Stage 11 (1a62c0f4)
R7	Structured correlation logs, slow requests, and once-per-request suspected-stuck diagnostics	Stage 7 plus the request-state part of Stage 13 (c04cda84)
R8	Sampled logical/attempt spans and OpenMetrics exemplars	Stage 8
R9	Dashboard, alerts, monitoring README, and symptom-first runbook	Stage 9 plus the operational-assets part of Stage 13 (c04cda84)

R0 and R1 establish the public metric and accounting foundations. R2 through
R5 then add subsystem instrumentation and may be reviewed in parallel after
their shared foundation is available. R6 follows the transport-state model;
R7 follows the request/attempt lifecycle; R8 follows correlation; and R9 is
last because it consumes the final names and semantics from all preceding
slices.

Each extracted PR will be created from the latest community main, with only
the commits or hunks needed for that dependency-complete slice. Cross-cutting
follow-up commits may therefore be split and folded during extraction. After a
predecessor merges, dependent slices will be rebased so already merged commits
are not repeated. The Draft integration PR remains open as the end-to-end
reference and is not intended for monolithic merge.

Review fixes may remain as explicit commits at the integration branch tip so
their validation history stays visible. During extraction they belong to, and
will be folded into, the review slice that owns the affected contract; for
example, 5c969fe1 and 0c20b102 are part of R0 rather than new trailing
feature stages.

Every extracted PR must be independently buildable and reviewable and include:

• its implementation, focused tests, public contract documentation, and
  AI-assistance disclosure in the same PR;
• metrics-enabled and metrics-compiled-out build/test coverage for affected
  paths;
• checks for counting points, conservation invariants, and bounded label
  cardinality where applicable;
• changed-line formatting and git diff --check;
• JSON/YAML parsing for operational assets and Sphinx/link validation when
  documentation tooling is available; and
• a focused hot-path smoke benchmark for instrumentation that changes request,
  attempt, posting, polling, or completion paths.

Production-topology performance acceptance and workload-specific alert
threshold tuning remain explicit merge gates for the slices that own those
surfaces; the development-container smoke results below are supporting evidence,
not a replacement for those gates.

Validation evidence

Baseline validation at Stage 9 after rebasing to b46254de:

• metrics-enabled full incremental build: 211/211 build steps completed;

• metrics-enabled cross-stage regression set: 20/20 passed;

• metrics-compiled-out regression set: 4/4 passed;

• pre-commit on all files changed from main: passed;

• Grafana JSON parse, alert YAML validation, repository links: passed;

• paired TCP-loopback submit benchmark, 4096 requests x 100 iterations x three
  runs: metrics-enabled median submit/completion deltas approximately
  +0.12%/+0.17% versus metrics compiled out in the development container.
  Review-driven delta validation:

• The Stage 0 review fix passed metrics-enabled and metrics-compiled-out focused
  builds and tests (2/2 in each configuration), changed-line clang-format, and
  git diff --check.

• Stage 10/11 outcome, readiness, stream-capacity, metrics HTTP, TCP and HP-TCP
  focused tests passed.

• Stage 12 metrics-on network/protocol tests passed 4/4 and metrics-off tests
  passed 2/2.

• Stage 13 final-format metrics-on and metrics-off recording tests each passed
  1/1; dashboard JSON and alert YAML parsed successfully.

The benchmark result is a smoke measurement, not a production-topology claim.
Extracted review PRs still require topology-specific performance acceptance and
workload-specific alert tuning. Sphinx was not rerun for the final documentation
because it is unavailable in the validation container; Stage 0's earlier Sphinx
build passed. The later delta passed changed-line clang-format, git diff --check, dashboard JSON parsing, and alert YAML parsing.

Relationship to existing work

• #4939 is the complete Draft integration stack for this RFC.
• #4938 is Stage 0 and the first independently reviewable PR.
• #4598 covers RDMA multi-rail allocation-intent and benchmark-window
  observability. This RFC does not duplicate per-NIC Prometheus labels or
  change selection policy.
• #2864 covers an operator CLI and sanitized diagnostic bundle. This RFC
  supplies the live in-process signals and snapshot that such tooling may
  consume; it does not create a competing CLI or archive format.
• #2713/#2958 cover Classic Transfer Engine metrics. This RFC is TENT-specific.

Review questions

1. Are the readiness checks and reason identifiers appropriate for intentionally
   single-transport deployments?
2. Are the four cross-cutting follow-up patches assigned to the right owner
   slices, especially the TCP/readiness split for Stage 11 and the
   RDMA/HP-TCP split for Stage 12?
3. Which public production topology should own the final metrics-only and
   low-rate-tracing performance gates?
4. Which compatible carrier and tracing backend should a later cross-process
   propagation RFC standardize?
5. Should payload integrity start with an optional HP-TCP payload checksum, or
   remain delegated to transport/NIC checks until a negotiated wire version is
   available?

Before submitting a new issue...

• Searched existing issues and PRs for TENT metrics, observability,
  diagnostics, tracing, health, readiness, and multi-rail statistics.
• Reviewed #4598, #2864, #2713, and #2958 and documented the non-overlapping
  responsibility of this RFC.

AI Assistance Disclosure

OpenAI Codex assisted with implementation, tests, documentation, validation,
and self-review. Each extracted review PR will retain the applicable disclosure
and exact validation evidence.

Originally created by @zxpdemonio on GitHub (Oct 26, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4940 ## Changes proposed ### Summary This RFC defines a production-oriented observability contract for TENT across four connected surfaces: 1. bounded Prometheus metrics for aggregation and alerting; 2. structured failure/slow-path logs with stable correlation identifiers; 3. compatible liveness plus meaningful readiness and redacted diagnostics; 4. sampled, process-local trace spans linked from OpenMetrics exemplars. The complete implementation is staged as 24 linear commits in draft integration PR #4939, rebased on community `main` at `b46254de`. Stage 0 is independently reviewable in #4938. Later review PRs will be extracted from the integration stack rather than merging #4939 monolithically. The operating objective is concrete: during an incident, an operator should be able to start from an alert or symptom, identify the failing phase and transport, retrieve a bounded diagnostic snapshot, and pivot from a metric to the correlated request/attempt logs or retained trace without adding high-cardinality metric labels. ### Status - All planned implementation stages are present in #4939. - The original metrics-enabled full build and 20-test cross-stage regression set pass. The four review-driven follow-up stages also pass their focused metrics-on and metrics-off build/test matrices. - Dashboard JSON and alert YAML parse successfully; the final dashboard has 20 panels and the alert bundle has 18 rules. - Production-topology performance acceptance and alert-threshold tuning remain work for the extracted review PRs. ## Motivation TENT already exposed request, byte, latency, attempt, and stage metrics, but an operator could not reliably answer several basic questions: - Is the metrics pipeline compiled in, enabled, bound, and serializing? - Did a logical request fail, or did a physical attempt fail and recover through failover? - Was the fault in metadata, admission, selection, connection, registration, submission, queueing, posting, polling, completion, retry, notification, cleanup, or shutdown? - Is an idle worker healthy, or is it stalled while work remains outstanding? - Which bounded subsystem signal should lead to the detailed request evidence? - Can a latency sample link to a retained trace without putting request IDs in Prometheus labels? - Is a retry a TENT software retry, a kernel TCP retransmission, or terminal RDMA hardware retry exhaustion? - Did HP-TCP reject a malformed frame, mismatch a response request ID, or observe a committed-length inconsistency? Without this contract, incidents require source inspection and ad-hoc logging before the failing subsystem can be localized. ## Goals - Define stable metric names, types, units, counting points, and bounded labels. - Separate logical request outcomes from physical transport attempts, retries, and failovers. - Cover common engine, RPC, RDMA, TCP/HP-TCP, SHM, segment, selection, admission, staging, and worker paths. - Preserve `/health` compatibility while adding `/live`, `/ready`, and a versioned `/diagnostics` schema. - Add stable structured event names and process-local correlation identifiers. - Retain errors, timeouts, cancellations, and configured slow traces in addition to deterministic head sampling. - Make logical-request and physical-attempt accounting independently checkable, and expose bounded queue/capacity and suspected-stuck signals. - Separate software retries, TCP retransmissions, RDMA completion status, and protocol framing/correlation failures. - Keep `TENT_METRICS_ENABLED=OFF` valid and keep tracing disabled by default. - Ship dashboard, alert, and runbook assets with the public contract. ## Non-goals - Import an internal monitoring framework, exporter, or private metric catalog. - Add internal-only transports or private backend state to the community API. - Put peer IDs, addresses, request IDs, policy names, segment names, connection IDs, NIC names, or QP numbers in Prometheus labels. - Emit one span per RDMA slice or completion. - Change selection, admission, retry, failover, or transfer correctness behavior. - Add automatic remediation or transport quarantine policy. - Introduce wire-protocol trace propagation in this first slice. - Add a payload checksum or other wire-format field in this stack. That needs a separate compatibility RFC covering negotiation, mixed versions, CPU cost, and failure semantics. - Replace the operator CLI/diagnostic bundle proposed in #2864. - Unify Classic Transfer Engine metrics from #2713/#2958. ## Public metrics contract ### Metrics pipeline self-observability Stage 0 (#4938) makes the telemetry plane observable: - `tent_metrics_build_info`; - `tent_metrics_runtime_enabled`; - `tent_metrics_http_server_up`; - `tent_metrics_http_start_failures_total`; - `tent_metrics_serialization_attempts_total`; - `tent_metrics_serialization_failures_total`; - `tent_metrics_last_success_timestamp_seconds`. These families remain visible before transfer traffic and while runtime transfer recording is disabled. A completely unreachable target must still be detected with Prometheus `up`; an in-process metric cannot report an endpoint that cannot be scraped. ### Logical requests versus physical attempts Existing read/write request, byte, failure, size, and latency families retain their logical-request meaning and final-transport attribution. New physical attempt families record the transport that actually executed each attempt: - `tent_transport_attempts_total`; - `tent_transport_attempt_failures_total`; - `tent_transport_attempt_latency_us`; - `tent_transport_failover_total`; - `tent_inflight_attempts`. A recovered RDMA-to-TCP request therefore records one successful logical TCP outcome, one failed RDMA attempt, one successful TCP attempt, and one failover. The accounting families are deliberately paired: ```text tent_logical_requests_started_total = sum(tent_logical_request_outcomes_total) + tent_logical_requests_inflight tent_transport_attempts_total = sum(tent_transport_attempt_outcomes_total) + tent_inflight_attempts ``` The equalities are process-local and apply to matching operation/transport dimensions. They make a lost terminal edge visible instead of inferring it from legacy success/failure counters. Paired state updates remain active across runtime sampling changes so a gauge cannot drift when only one half is seen. ### Bounded failure model Terminal failures use a shared phase/reason taxonomy. Phases are limited to: ```text metadata admission selection connect register submit queue post poll completion retry failover notify cleanup shutdown ``` Reasons are limited to: ```text timeout unavailable queue_full resource_exhausted protocol stale_generation wc_error cq_error mr_error peer_closed canceled internal ``` Detailed native status text remains in the owning subsystem's existing log path; it is not copied into metric labels. ### Implemented subsystem signals - Common engine: terminal failure phase/reason and backend memory register/unregister operation, byte, and outcome counters. - Worker and staging: last-progress timestamps, staging task/byte gauges, deferred-cleanup gauges, cleanup timeouts, and quarantined batches. - RPC: physical method/outcome attempts, attempt latency, and actual retries. - RDMA data path: slice lifecycle, post/completion outcomes, completion latency, software timeout, software retries/retry exhaustion, categorized non-success WC status (including retry and RNR retry exhaustion), CQ poll errors, and worker/posted in-flight gauges. - RDMA lifecycle: endpoint construction/readiness/retirement/destruction, data and notification QP transition failures, and bounded notification events. - TCP and HP-TCP: queue, busy-worker, admitted-task/byte and connection gauges; attempt, retry, timeout, peer-close, terminal outcome, queue wait, and attempt latency. HP-TCP additionally samples Linux TCP_INFO retransmission deltas, smoothed RTT, and congestion window, and classifies malformed or inconsistent protocol frames. - Suspected stuck requests: an opt-in threshold emits one bounded phase/transport/operation counter and one correlated structured warning per logical request without changing request behavior. - SHM: bounded relocation-cache/backing-object mapping events plus physical copy outcome and latency. - Segment metadata: snapshot publication/mutation, registry publish/delete, remote fetch/cache/reset/invalidation, and generation advance. - Selection: policy miss, empty/unavailable candidates, hint rejection or unavailability, selection, fallback, exhaustion, and metadata refresh. - Admission: admit/reject/dispatch/deadline-drop/cancel events, wait latency, and outstanding owner/byte gauges by bounded owner kind. The complete metric table, labels, units, and cardinality notes are in `docs/source/design/tent/metrics.md`. ## Liveness, readiness, and diagnostics The endpoints share the existing metrics HTTP server: - `/health`: retained as `200 OK` plain-text compatibility and only proves the HTTP event loop is serving; - `/live`: versioned JSON process liveness and remains `200` while the endpoint can respond; - `/ready`: versioned JSON readiness, `200` only when mandatory checks pass and `503` otherwise; - `/diagnostics`: the same redacted snapshot, always `200` so it can be collected during an incident. Readiness reasons are a closed set: ```text engine_diagnostics_unavailable engine_unavailable control_plane_unavailable no_transport_available progress_worker_unavailable progress_worker_stalled ``` `/ready` uses only bounded aggregate state and is O(number of transport kinds), so it is safe for frequent probes. `/diagnostics` intentionally retains the more detailed O(number of live tasks) scan and must not be used as the readiness probe. The progress-worker stall decision requires both old outstanding work and stale progress, so a legitimately long request does not make the engine unready while the worker continues to advance. Diagnostics schema version 1 contains build identity, engine/control-plane availability, bounded loaded-transport names, progress-worker state, runtime queue aggregates, alive batch count, and oldest in-flight age. It omits request and peer identifiers, segment names, network and memory addresses, keys, credentials, payloads, and raw pointers. Provider ownership is explicit: the engine registers and clears its diagnostics provider with an owner token, and snapshot invocation is protected against teardown races. Readiness and in-process metrics cannot report after the process has exited or crashed. A missing `/live`/scrape target is therefore correlated with the supervisor's exit status, stderr/startup logs, and a core dump or sanitizer report. `tent_metrics_build_info` identifies the binary while it is alive; it does not replace external crash retention. ## Structured logs and correlation The initial stable events are: ```text tent.request.failed tent.request.slow tent.request.stuck tent.transport.attempt_failed tent.transport.failover tent.transport.selection_failed ``` They use process-local monotonic `request_id`, `batch_id`, `task_id`, and `attempt_id` values, a 128-bit `trace_id`, bounded transport/operation/phase/ reason/status values, duration, failover count where applicable, and configuration generation. Raw `BatchID` pointers are never logged as correlation IDs. Tests assert that structured events contain neither source addresses nor target offsets. `observability/slow_request_threshold_us` controls slow-request logs. `observability/stuck_request_threshold_us` controls the first still-pending threshold crossing. Both default to zero (disabled). A stuck event is diagnostic: it neither cancels a request nor proves a deadlock. ## Network loss, retransmission, and protocol integrity The public signals intentionally keep three retry layers separate: - `tent_rdma_retries_total` is a TENT software slice re-enqueue; - `tent_tcp_retransmissions_total` is a delta of Linux `TCP_INFO.tcpi_total_retrans`, sampled every 256 completed HP-TCP operations and once before connection close; - `tent_rdma_wc_errors_total{category=~"retry_exceeded|rnr_retry_exceeded"}` is the terminal WC status after the NIC/QP hardware retry budget is exhausted. Per-packet RDMA retransmission counts are below the verbs completion boundary and require NIC/vendor telemetry. Classic RPC-backed TCP does not expose its socket handle at this layer, so host TCP telemetry remains the source for that transport. HP-TCP protocol counters distinguish invalid size, magic, version, reserved fields, opcode, length, and status from response `request_id_mismatch` and `committed_length_mismatch`. TCP already provides ordered, checksummed byte delivery; these counters diagnose framing, mixed-version, correlation, or peer implementation faults. End-to-end payload checksums are explicitly deferred to a child RFC because they change the wire/data path and need mixed-version and performance design, not just observability instrumentation. ## Sampled traces and OpenMetrics exemplars Tracing is backend-neutral in this slice and is emitted as completed structured span records: - one `tent.logical_transfer` root span per logical request; - one `tent.transport_attempt` child span per physical attempt; - deterministic head sampling using `observability/trace_sample_rate_ppm` (capped at one million); - unconditional retention for errors, timeouts, cancellations, and requests slower than `observability/trace_slow_threshold_us`; - parent retention when a child was retained; - bounded retention reasons: `head_sampled`, `slow`, `error`, `timeout`, and `linked_child`. Tracing is disabled by default. Retained spans are `INFO` logs, so deployments that enable tracing also set `log_level=info`. The implementation does not add a tracing dependency and does not change the TENT wire protocol. Cross-process context propagation is a compatible future extension rather than an implicit claim of this stage. `/metrics` remains Prometheus 0.0.4 compatible. The opt-in `/metrics/openmetrics` endpoint uses the OpenMetrics 1.0 content type, ends with `# EOF`, and can attach the latest retained trace ID to the matching `tent_transport_attempt_latency_us` `+Inf` bucket. Exemplar publication occurs after the corresponding histogram observation and uses a bounded fixed-size per-series store. ## Cardinality, safety, and compatibility - Metric labels are derived only from documented enums and fixed build identity. - Dynamic cells allocate only label combinations that actually occur, within a closed maximum domain. - High-cardinality correlation IDs appear only in logs/traces, never metrics. - Diagnostics and new structured events do not expose payloads, credentials, raw pointers, addresses, target offsets, or peer-provided strings. - Existing metric names and `/health` behavior remain compatible. - Metrics-off builds provide API stubs and pass the selected regression suite. - Trace generation cannot fail a transfer, and exemplar export is optional. ## Operations assets The stack includes: - `mooncake-transfer-engine/tent/monitoring/grafana/tent-overview.json`: an importable 20-panel dashboard for telemetry health, logical requests, physical attempts, latency, failover, workers, RDMA, admission, staging, HP-TCP socket health, protocol errors, and stuck events; - `mooncake-transfer-engine/tent/monitoring/prometheus/tent-alerts.yaml`: 18 starter alerts covering scrape/serialization health, logical and physical failure rates, worker stalls, staging cleanup, RDMA errors/retry exhaustion, admission rejection, and selection exhaustion; - `docs/source/design/tent/observability-runbook.md`: a symptom-first workflow that connects readiness, bounded metrics, structured logs, traces, and exemplars. Alert thresholds are conservative examples and must be tuned to workload SLOs and traffic floors. The rules assume normal Prometheus `job` and `instance` labels and document the job-selector adjustment point. ## Implementation and review plan ### Integration stack Draft PR #4939 contains the complete linear integration history below. It is a continuously tested reference stack, not the intended merge unit: | Stage | Scope | Commit(s) | |---|---|---| | 0 | Metrics contract/self-observability | `aae72ce1`, `5c969fe1`, `0c20b102` (review fixes; #4938) | | 1 | Common failure, memory, worker, staging, RPC | `8da2d4d9`, `c0fe1f27`, `8311b35d` | | 2 | RDMA data path and lifecycle | `14b794f9`, `2a6b2fe5` | | 3 | TCP and HP-TCP | `a376597b` | | 4 | SHM and segment | `ac700ccf`, `5c9e77b3` | | 5 | Selection and admission | `48d4d6e2`, `554f5cd1`, `02d3fb4a` | | 6 | Liveness/readiness/diagnostics | `b5a0ae2d`, `32cf582f`, `c2efeae0` | | 7 | Structured correlation logs | `b50027e9` | | 8 | Sampled spans and exemplars | `2ead741d` | | 9 | Dashboard, alerts, runbook | `28e4b6af` | | 10 | Lossless logical-request and physical-attempt outcome accounting | `6f75ffd5` | | 11 | O(1) readiness and stream capacity health | `1a62c0f4` | | 12 | TCP_INFO, RDMA WC, and HP-TCP protocol diagnostics | `cdfa0ae5` | | 13 | Suspected-stuck events and expanded operational assets | `c04cda84` | The commit IDs describe the current integration snapshot based on community `main` at `b46254de` and ending at `0c20b102`. They will be refreshed in this RFC if the Draft integration branch is rebased. ### Extracted review slices The four review-driven follow-up patches complete earlier functional areas and will be folded into those owner slices rather than submitted as four trailing fix-up PRs. In particular, Stage 11 is divided between TCP capacity and readiness, Stage 12 between RDMA and HP-TCP, and Stage 13 between request-state diagnostics and operational assets. | Review slice | Reviewable scope | Integration material incorporated | |---|---|---| | R0 | Metrics contract and telemetry-pipeline self-observability | Stage 0 (`aae72ce1`) plus its review fixes (`5c969fe1`, `0c20b102`), already open as #4938 (`1bf03834`, `639ec638`, `80ee17d1`) | | R1 | Common failure/memory/worker/staging/RPC metrics plus lossless logical-request and physical-attempt accounting | Stage 1 plus Stage 10 (`6f75ffd5`) | | R2 | RDMA data-path and endpoint lifecycle observability, including categorized WC errors and hardware retry exhaustion | Stage 2 plus the RDMA part of Stage 12 (`cdfa0ae5`) | | R3 | TCP and HP-TCP attempts, queue/capacity health, Linux TCP_INFO, and protocol-integrity diagnostics | Stage 3 plus the stream-capacity part of Stage 11 (`1a62c0f4`) and the HP-TCP part of Stage 12 (`cdfa0ae5`) | | R4 | SHM copy and segment-metadata observability | Stage 4 | | R5 | Transport selection and admission observability | Stage 5 | | R6 | Liveness, bounded O(transport kinds) readiness, and redacted diagnostics | Stage 6 plus the readiness part of Stage 11 (`1a62c0f4`) | | R7 | Structured correlation logs, slow requests, and once-per-request suspected-stuck diagnostics | Stage 7 plus the request-state part of Stage 13 (`c04cda84`) | | R8 | Sampled logical/attempt spans and OpenMetrics exemplars | Stage 8 | | R9 | Dashboard, alerts, monitoring README, and symptom-first runbook | Stage 9 plus the operational-assets part of Stage 13 (`c04cda84`) | R0 and R1 establish the public metric and accounting foundations. R2 through R5 then add subsystem instrumentation and may be reviewed in parallel after their shared foundation is available. R6 follows the transport-state model; R7 follows the request/attempt lifecycle; R8 follows correlation; and R9 is last because it consumes the final names and semantics from all preceding slices. Each extracted PR will be created from the latest community `main`, with only the commits or hunks needed for that dependency-complete slice. Cross-cutting follow-up commits may therefore be split and folded during extraction. After a predecessor merges, dependent slices will be rebased so already merged commits are not repeated. The Draft integration PR remains open as the end-to-end reference and is not intended for monolithic merge. Review fixes may remain as explicit commits at the integration branch tip so their validation history stays visible. During extraction they belong to, and will be folded into, the review slice that owns the affected contract; for example, `5c969fe1` and `0c20b102` are part of R0 rather than new trailing feature stages. Every extracted PR must be independently buildable and reviewable and include: - its implementation, focused tests, public contract documentation, and AI-assistance disclosure in the same PR; - metrics-enabled and metrics-compiled-out build/test coverage for affected paths; - checks for counting points, conservation invariants, and bounded label cardinality where applicable; - changed-line formatting and `git diff --check`; - JSON/YAML parsing for operational assets and Sphinx/link validation when documentation tooling is available; and - a focused hot-path smoke benchmark for instrumentation that changes request, attempt, posting, polling, or completion paths. Production-topology performance acceptance and workload-specific alert threshold tuning remain explicit merge gates for the slices that own those surfaces; the development-container smoke results below are supporting evidence, not a replacement for those gates. ## Validation evidence Baseline validation at Stage 9 after rebasing to `b46254de`: - metrics-enabled full incremental build: 211/211 build steps completed; - metrics-enabled cross-stage regression set: 20/20 passed; - metrics-compiled-out regression set: 4/4 passed; - pre-commit on all files changed from `main`: passed; - Grafana JSON parse, alert YAML validation, repository links: passed; - paired TCP-loopback submit benchmark, 4096 requests x 100 iterations x three runs: metrics-enabled median submit/completion deltas approximately `+0.12%`/`+0.17%` versus metrics compiled out in the development container. Review-driven delta validation: - The Stage 0 review fix passed metrics-enabled and metrics-compiled-out focused builds and tests (2/2 in each configuration), changed-line clang-format, and `git diff --check`. - Stage 10/11 outcome, readiness, stream-capacity, metrics HTTP, TCP and HP-TCP focused tests passed. - Stage 12 metrics-on network/protocol tests passed 4/4 and metrics-off tests passed 2/2. - Stage 13 final-format metrics-on and metrics-off recording tests each passed 1/1; dashboard JSON and alert YAML parsed successfully. The benchmark result is a smoke measurement, not a production-topology claim. Extracted review PRs still require topology-specific performance acceptance and workload-specific alert tuning. Sphinx was not rerun for the final documentation because it is unavailable in the validation container; Stage 0's earlier Sphinx build passed. The later delta passed changed-line clang-format, `git diff --check`, dashboard JSON parsing, and alert YAML parsing. ## Relationship to existing work - #4939 is the complete Draft integration stack for this RFC. - #4938 is Stage 0 and the first independently reviewable PR. - #4598 covers RDMA multi-rail allocation-intent and benchmark-window observability. This RFC does not duplicate per-NIC Prometheus labels or change selection policy. - #2864 covers an operator CLI and sanitized diagnostic bundle. This RFC supplies the live in-process signals and snapshot that such tooling may consume; it does not create a competing CLI or archive format. - #2713/#2958 cover Classic Transfer Engine metrics. This RFC is TENT-specific. ## Review questions 1. Are the readiness checks and reason identifiers appropriate for intentionally single-transport deployments? 2. Are the four cross-cutting follow-up patches assigned to the right owner slices, especially the TCP/readiness split for Stage 11 and the RDMA/HP-TCP split for Stage 12? 3. Which public production topology should own the final metrics-only and low-rate-tracing performance gates? 4. Which compatible carrier and tracing backend should a later cross-process propagation RFC standardize? 5. Should payload integrity start with an optional HP-TCP payload checksum, or remain delegated to transport/NIC checks until a negotiated wire version is available? ### Before submitting a new issue... - [x] Searched existing issues and PRs for TENT metrics, observability, diagnostics, tracing, health, readiness, and multi-rail statistics. - [x] Reviewed #4598, #2864, #2713, and #2958 and documented the non-overlapping responsibility of this RFC. ## AI Assistance Disclosure OpenAI Codex assisted with implementation, tests, documentation, validation, and self-review. Each extracted review PR will retain the applicable disclosure and exact validation evidence.

bsakaguchi added the

RFC

label 2026-10-29 21:24:29 +00:00

bsakaguchi commented 2026-10-29 21:24:29 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Oct 26, 2026):

Implementation progress: draft integration PR #4939 now contains commit 5d1caece, the first Stage 1 patch.

This patch adds:

• tent_terminal_failures_total{transport,phase,reason} at the canonical terminal task accounting point;
• tent_memory_operations_total{transport,operation,outcome} around real backend registration calls;
• tent_memory_operation_bytes_total{transport,operation,outcome} with batched-register byte semantics;
• metrics-enabled/compiled-out and success/failure regression coverage.

The rest of Stage 1 (oldest inflight age, worker progress, staging cleanup and control/metadata RPC signals) remains pending in the ordered commit stack.

<!-- gh-comment-id:6409558513 --> @zxpdemonio commented on GitHub (Oct 26, 2026): Implementation progress: draft integration PR #4939 now contains commit `5d1caece`, the first Stage 1 patch. This patch adds: - `tent_terminal_failures_total{transport,phase,reason}` at the canonical terminal task accounting point; - `tent_memory_operations_total{transport,operation,outcome}` around real backend registration calls; - `tent_memory_operation_bytes_total{transport,operation,outcome}` with batched-register byte semantics; - metrics-enabled/compiled-out and success/failure regression coverage. The rest of Stage 1 (oldest inflight age, worker progress, staging cleanup and control/metadata RPC signals) remains pending in the ordered commit stack.

bsakaguchi commented 2026-10-29 21:24:30 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 26, 2026):

Thanks for opening this issue, @zxpdemonio!

Field	Value
Issue	#4940
GitHub user ID	16206312
Reporter	@zxpdemonio

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:6409725572 --> @github-actions[bot] commented on GitHub (Oct 26, 2026): Thanks for opening this issue, @zxpdemonio! | Field | Value | |-------|-------| | **Issue** | #4940 | | **GitHub user ID** | `16206312` | | **Reporter** | @zxpdemonio | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:24:30 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Oct 26, 2026):

Stage 1 follow-up commit 81353e8e is now in draft integration PR #4939.

It adds worker-owned progress timestamps, staging in-flight task/byte gauges,
deferred cleanup resource gauges and the bounded shutdown-drain timeout
counter. Metrics-enabled and metrics-compiled-out builds both pass the metrics,
progress-worker and proxy-shutdown regression targets.

The oldest-inflight signal remains assigned to the versioned engine snapshot
stage so the metrics hot path does not maintain a second locked registry of
live attempts. Control/metadata RPC signals remain pending.

<!-- gh-comment-id:6409752947 --> @zxpdemonio commented on GitHub (Oct 26, 2026): Stage 1 follow-up commit `81353e8e` is now in draft integration PR #4939. It adds worker-owned progress timestamps, staging in-flight task/byte gauges, deferred cleanup resource gauges and the bounded shutdown-drain timeout counter. Metrics-enabled and metrics-compiled-out builds both pass the metrics, progress-worker and proxy-shutdown regression targets. The oldest-inflight signal remains assigned to the versioned engine snapshot stage so the metrics hot path does not maintain a second locked registry of live attempts. Control/metadata RPC signals remain pending.

bsakaguchi commented 2026-10-29 21:24:30 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Oct 26, 2026):

Stage 1 is now complete on draft integration PR #4939.

The final Stage 1 patch (86063abe) instruments the canonical CoroRpcAgent::callCoroutine() attempt loop with bounded physical-attempt metrics:

• tent_rpc_attempts_total{method,outcome};
• tent_rpc_retries_total{method};
• tent_rpc_attempt_latency_us{method,outcome}.

An idempotent logical call that retries once records two attempts and one retry. The retry counter advances only when the second attempt is actually issued. Outcomes distinguish connection establishment failures, RPC transport/protocol failures, and successful transport responses. Application-level failures returned inside a successful response remain owned by the calling subsystem's failure metric or log.

The method and outcome domains are closed enums, so peer addresses, endpoint identities, function IDs, and error strings never become metric labels. Metrics-disabled builds compile the attempt path out.

Validation for Stage 1c:

• metrics enabled: metrics recording, RPC reconnect, segment descriptor retry, and RPC handler isolation all pass (4/4);
• metrics compiled out: the same four regression targets pass (4/4);
• pre-commit passes for all touched files.

Oldest-inflight age remains assigned to Stage 6, where it can be derived from the engine-owned task snapshot instead of introducing a second hot-path task registry in metrics.

<!-- gh-comment-id:6409945805 --> @zxpdemonio commented on GitHub (Oct 26, 2026): Stage 1 is now complete on draft integration PR #4939. The final Stage 1 patch (`86063abe`) instruments the canonical `CoroRpcAgent::callCoroutine()` attempt loop with bounded physical-attempt metrics: - `tent_rpc_attempts_total{method,outcome}`; - `tent_rpc_retries_total{method}`; - `tent_rpc_attempt_latency_us{method,outcome}`. An idempotent logical call that retries once records two attempts and one retry. The retry counter advances only when the second attempt is actually issued. Outcomes distinguish connection establishment failures, RPC transport/protocol failures, and successful transport responses. Application-level failures returned inside a successful response remain owned by the calling subsystem's failure metric or log. The method and outcome domains are closed enums, so peer addresses, endpoint identities, function IDs, and error strings never become metric labels. Metrics-disabled builds compile the attempt path out. Validation for Stage 1c: - metrics enabled: metrics recording, RPC reconnect, segment descriptor retry, and RPC handler isolation all pass (4/4); - metrics compiled out: the same four regression targets pass (4/4); - pre-commit passes for all touched files. Oldest-inflight age remains assigned to Stage 6, where it can be derived from the engine-owned task snapshot instead of introducing a second hot-path task registry in metrics.

bsakaguchi commented 2026-10-29 21:24:30 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Oct 26, 2026):

Stage 2a is now available in draft PR #4939 as commit da8fcd1f ([TENT] Add RDMA data-path metrics).

This patch adds bounded, physical-slice RDMA diagnostics at the canonical worker owner:

• slice lifecycle events for enqueue, post attempt/error, path error, completion outcome, late completion and software timeout;
• retry and retry-exhaustion counters separated by endpoint, post and completion cause;
• CQ poll errors, RDMA worker progress, worker-owned and posted in-flight slice gauges;
• post-to-CQ latency split by read/write and bounded completion outcome.

The contract explicitly counts physical slices rather than logical requests and excludes peer IDs, addresses, connection identities and QP numbers from metric labels. Retry counters advance only when another attempt is actually enqueued; terminal exhaustion is a separate signal.

Validation:

• metrics enabled: 6/6 RDMA metrics/regression targets passed;
• metrics compiled out: 6/6 RDMA metrics/regression targets passed;
• pre-commit passed for all Stage 2a files.

Stage 2 remains open for endpoint/QP lifecycle and notification diagnostics.

<!-- gh-comment-id:6410212793 --> @zxpdemonio commented on GitHub (Oct 26, 2026): Stage 2a is now available in draft PR #4939 as commit `da8fcd1f` (`[TENT] Add RDMA data-path metrics`). This patch adds bounded, physical-slice RDMA diagnostics at the canonical worker owner: - slice lifecycle events for enqueue, post attempt/error, path error, completion outcome, late completion and software timeout; - retry and retry-exhaustion counters separated by endpoint, post and completion cause; - CQ poll errors, RDMA worker progress, worker-owned and posted in-flight slice gauges; - post-to-CQ latency split by read/write and bounded completion outcome. The contract explicitly counts physical slices rather than logical requests and excludes peer IDs, addresses, connection identities and QP numbers from metric labels. Retry counters advance only when another attempt is actually enqueued; terminal exhaustion is a separate signal. Validation: - metrics enabled: 6/6 RDMA metrics/regression targets passed; - metrics compiled out: 6/6 RDMA metrics/regression targets passed; - pre-commit passed for all Stage 2a files. Stage 2 remains open for endpoint/QP lifecycle and notification diagnostics.

bsakaguchi commented 2026-10-29 21:24:30 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Oct 26, 2026):

Stage 2 is complete in draft PR #4939.

Stage 2b is commit 9bb48750 ([TENT] Add RDMA endpoint metrics) and adds the remaining bounded RDMA lifecycle signals:

• endpoint construction, active/passive readiness, retirement, destruction and destroy-error events;
• failed data/notification QP transitions split by bounded QP type and state transition;
• notification receive posting/delivery, send posting/completion, invalid/dropped payload, post/completion/CQ errors, unknown-QP and channel-disabled events.

No endpoint name, peer address, connection identity or QP number is exposed as a Prometheus label. Expected flushes from already-retiring notification QPs are excluded from completion-error counts.

Validation:

• metrics enabled: 6/6 RDMA metrics/endpoint/notification regression targets passed;
• metrics compiled out: 6/6 targets passed;
• pre-commit passed for all Stage 2b files.

The ordered Stage 2 commits are da8fcd1f (slice/data path) and 9bb48750 (endpoint/QP/notification).

<!-- gh-comment-id:6410389617 --> @zxpdemonio commented on GitHub (Oct 26, 2026): Stage 2 is complete in draft PR #4939. Stage 2b is commit `9bb48750` (`[TENT] Add RDMA endpoint metrics`) and adds the remaining bounded RDMA lifecycle signals: - endpoint construction, active/passive readiness, retirement, destruction and destroy-error events; - failed data/notification QP transitions split by bounded QP type and state transition; - notification receive posting/delivery, send posting/completion, invalid/dropped payload, post/completion/CQ errors, unknown-QP and channel-disabled events. No endpoint name, peer address, connection identity or QP number is exposed as a Prometheus label. Expected flushes from already-retiring notification QPs are excluded from completion-error counts. Validation: - metrics enabled: 6/6 RDMA metrics/endpoint/notification regression targets passed; - metrics compiled out: 6/6 targets passed; - pre-commit passed for all Stage 2b files. The ordered Stage 2 commits are `da8fcd1f` (slice/data path) and `9bb48750` (endpoint/QP/notification).

bsakaguchi commented 2026-10-29 21:24:30 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Oct 26, 2026):

Stage 3 is complete in draft PR #4939 as commit 323d0446 ([TENT] Add TCP transport diagnostics).

The patch adds one bounded stream contract shared by TCP and HP-TCP:

• queue, physical attempt, connect, terminal outcome, actual retry and retry-exhaustion events;
• physical attempt latency split by transport, operation and bounded outcome;
• queue wait split by transport and operation;
• HP-TCP timeout and peer-close classification at the final async completion owner.

Counting semantics remain explicit: classic TCP queues logical tasks and records every data RPC issued by its retry loop as a physical attempt. HP-TCP queue/attempt metrics describe physical slice operations, while a logical retry is counted only after retry admission succeeds. Peer IDs, addresses, lane IDs and request IDs are not labels.

Validation:

• metrics enabled: 6/6 metrics, TCP and HP-TCP regression targets passed;
• metrics compiled out: 6/6 targets passed;
• pre-commit passed for all Stage 3 files.

<!-- gh-comment-id:6410597784 --> @zxpdemonio commented on GitHub (Oct 26, 2026): Stage 3 is complete in draft PR #4939 as commit `323d0446` (`[TENT] Add TCP transport diagnostics`). The patch adds one bounded stream contract shared by TCP and HP-TCP: - queue, physical attempt, connect, terminal outcome, actual retry and retry-exhaustion events; - physical attempt latency split by transport, operation and bounded outcome; - queue wait split by transport and operation; - HP-TCP timeout and peer-close classification at the final async completion owner. Counting semantics remain explicit: classic TCP queues logical tasks and records every data RPC issued by its retry loop as a physical attempt. HP-TCP queue/attempt metrics describe physical slice operations, while a logical retry is counted only after retry admission succeeds. Peer IDs, addresses, lane IDs and request IDs are not labels. Validation: - metrics enabled: 6/6 metrics, TCP and HP-TCP regression targets passed; - metrics compiled out: 6/6 targets passed; - pre-commit passed for all Stage 3 files.

bsakaguchi commented 2026-10-29 21:24:31 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Oct 26, 2026):

Stage 4 is complete on draft integration PR #4939.

• Stage 4a (80bd032e) adds bounded SHM relocation/mapping events plus synchronous physical copy outcomes and latency.
• Stage 4b (958d1c1d) adds local segment snapshot/mutation, registry publish/delete, remote descriptor fetch/cache/reset/invalidation, and cache-generation events.
• Metric labels exclude segment names/handles, peer identities, addresses, and shared-memory paths.
• Metrics-enabled validation: Stage 4a 2/2, Stage 4b 2/2.
• Metrics-compiled-out validation: Stage 4a 2/2, Stage 4b 2/2.
• Pre-commit passes on all Stage 4 files.

The next integration slice is Stage 5: transport selection and admission decisions.

<!-- gh-comment-id:6411122708 --> @zxpdemonio commented on GitHub (Oct 26, 2026): Stage 4 is complete on draft integration PR #4939. - Stage 4a (`80bd032e`) adds bounded SHM relocation/mapping events plus synchronous physical copy outcomes and latency. - Stage 4b (`958d1c1d`) adds local segment snapshot/mutation, registry publish/delete, remote descriptor fetch/cache/reset/invalidation, and cache-generation events. - Metric labels exclude segment names/handles, peer identities, addresses, and shared-memory paths. - Metrics-enabled validation: Stage 4a 2/2, Stage 4b 2/2. - Metrics-compiled-out validation: Stage 4a 2/2, Stage 4b 2/2. - Pre-commit passes on all Stage 4 files. The next integration slice is Stage 5: transport selection and admission decisions.

bsakaguchi commented 2026-10-29 21:24:31 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Oct 26, 2026):

Stage 5 is complete on draft integration PR #4939.

• Stage 5a (220950b1) records bounded selector decisions: policy miss, candidate empty/unavailable, hint rejection/unavailability, selected/fallback/exhausted, and metadata refresh.
• Stage 5b (05f6f048) records admission owner admit/reject/dispatch/deadline-drop/cancel events, queue wait latency, and paired outstanding owner/byte gauges by owner kind.
• Metric labels exclude policy names, request ids, queue owner ids, peers, and addresses.
• Metrics-enabled validation: Stage 5a 2/2, Stage 5b 3/3.
• Metrics-compiled-out validation: Stage 5a 2/2, Stage 5b 3/3.
• Pre-commit passes on all Stage 5 files.

The next integration slice is Stage 6: liveness, readiness, and the versioned/redacted diagnostics snapshot.

<!-- gh-comment-id:6411806482 --> @zxpdemonio commented on GitHub (Oct 26, 2026): Stage 5 is complete on draft integration PR #4939. - Stage 5a (`220950b1`) records bounded selector decisions: policy miss, candidate empty/unavailable, hint rejection/unavailability, selected/fallback/exhausted, and metadata refresh. - Stage 5b (`05f6f048`) records admission owner admit/reject/dispatch/deadline-drop/cancel events, queue wait latency, and paired outstanding owner/byte gauges by owner kind. - Metric labels exclude policy names, request ids, queue owner ids, peers, and addresses. - Metrics-enabled validation: Stage 5a 2/2, Stage 5b 3/3. - Metrics-compiled-out validation: Stage 5a 2/2, Stage 5b 3/3. - Pre-commit passes on all Stage 5 files. The next integration slice is Stage 6: liveness, readiness, and the versioned/redacted diagnostics snapshot.

bsakaguchi commented 2026-10-29 21:24:31 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Oct 26, 2026):

@alogfans The RFC and linked Draft implementation PR #4939 are now complete and ready for design review. The RFC has been updated to match the implemented scope: bounded subsystem metrics, versioned readiness/diagnostics, structured correlation logs, process-local sampled spans with OpenMetrics exemplars, and public dashboard/alert/runbook assets. The full stack is rebased on b46254de and passes the final metrics-on/off integration checks; production-topology performance acceptance and alert threshold tuning are explicitly left to the extracted review PRs. Could you please review the contract and proposed extraction sequence?

<!-- gh-comment-id:6414751623 --> @zxpdemonio commented on GitHub (Oct 26, 2026): @alogfans The RFC and linked Draft implementation PR #4939 are now complete and ready for design review. The RFC has been updated to match the implemented scope: bounded subsystem metrics, versioned readiness/diagnostics, structured correlation logs, process-local sampled spans with OpenMetrics exemplars, and public dashboard/alert/runbook assets. The full stack is rebased on b46254de and passes the final metrics-on/off integration checks; production-topology performance acceptance and alert threshold tuning are explicitly left to the extracted review PRs. Could you please review the contract and proposed extraction sequence?

bsakaguchi commented 2026-10-29 21:24:32 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Oct 26, 2026):

@alogfans The RFC and draft integration PR have been updated after the observability gap review. The added design covers lossless request/attempt accounting, O(1) readiness, stream capacity, TCP retransmissions, terminal RDMA hardware retry exhaustion, HP-TCP protocol errors, suspected-stuck requests, and the explicit payload-checksum boundary. Could you please review the RFC and the staged split in #4939?

<!-- gh-comment-id:6417298650 --> @zxpdemonio commented on GitHub (Oct 26, 2026): @alogfans The RFC and draft integration PR have been updated after the observability gap review. The added design covers lossless request/attempt accounting, O(1) readiness, stream capacity, TCP retransmissions, terminal RDMA hardware retry exhaustion, HP-TCP protocol errors, suspected-stuck requests, and the explicit payload-checksum boundary. Could you please review the RFC and the staged split in #4939?

bsakaguchi referenced this issue 2026-10-29 21:32:50 +00:00

[PR #1255] [MERGED] [CI] Force TCP for Mooncake EP Backend tests #2109

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#1255

No description provided.