[PR #339] [MERGED] observability: bound the heartbeat channel's connect attempt #361

New issue

Closed

opened 2026-10-29 14:22:36 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:22:36 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/339
Author: @nblintao
Created: 10/2/2026
Status: ✅ Merged
Merged: 10/3/2026
Merged by: @yingdi-shan

Base: main ← Head: fix/heartbeat-connect-timeout

📝 Commits (1)

01b6a97 observability: bound the heartbeat channel's connect attempt

📊 Changes

1 file changed (+9 additions, -1 deletions)

View changed files

📝 src/observability/reporter.rs (+9 -1)

📄 Description

The reporter's channel is built with connect_lazy() and no connect timeout, so when a reconnect's SYN goes unanswered the call is bounded only by the kernel's retry limit — over two minutes at the default tcp_syn_retries.

GRPC_CALL_TIMEOUT does not cover this. Request::set_timeout writes a grpc-timeout header that the channel's GrpcTimeout layer does enforce client-side, but it starts its timer in Service::call, and a call only reaches that after poll_ready has a connection. The connect itself happens inside Reconnect::poll_ready, outside any deadline.

A node refreshes every one of its sandbox bindings over this channel, so while a connect hangs nothing refreshes them and they expire — the data plane stops being able to route to sandboxes that are running and healthy.

Give the channel a 10s connect_timeout, the value already used for the other gRPC endpoints in this repository (src/image/buildkit.rs).

Wrapping the call in tokio::time::timeout would not substitute: dropping the future leaves the attempt in the channel, and later calls queue behind it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/339 **Author:** [@nblintao](https://github.com/nblintao) **Created:** 10/2/2026 **Status:** ✅ Merged **Merged:** 10/3/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `fix/heartbeat-connect-timeout` --- ### 📝 Commits (1) - [`01b6a97`](https://github.com/kvcache-ai/AgentENV/commit/01b6a97acc4a81214caeda8c924f8c00313f4277) observability: bound the heartbeat channel's connect attempt ### 📊 Changes **1 file changed** (+9 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `src/observability/reporter.rs` (+9 -1) </details> ### 📄 Description The reporter's channel is built with `connect_lazy()` and no connect timeout, so when a reconnect's SYN goes unanswered the call is bounded only by the kernel's retry limit — over two minutes at the default `tcp_syn_retries`. `GRPC_CALL_TIMEOUT` does not cover this. `Request::set_timeout` writes a `grpc-timeout` header that the channel's `GrpcTimeout` layer does enforce client-side, but it starts its timer in `Service::call`, and a call only reaches that after `poll_ready` has a connection. The connect itself happens inside `Reconnect::poll_ready`, outside any deadline. A node refreshes every one of its sandbox bindings over this channel, so while a connect hangs nothing refreshes them and they expire — the data plane stops being able to route to sandboxes that are running and healthy. - Give the channel a 10s `connect_timeout`, the value already used for the other gRPC endpoints in this repository (`src/image/buildkit.rs`). Wrapping the call in `tokio::time::timeout` would not substitute: dropping the future leaves the attempt in the channel, and later calls queue behind it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:22:36 +00:00