kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3142] [Bug]: HA etcd clients repeatedly receive too_many_pings GOAWAY since v0.3.12 #805

New issue

Closed

opened 2026-10-29 21:19:44 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:19:44 +00:00

Owner

Originally created by @Icedcoco on GitHub (Jul 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3142

Originally assigned to: @Icedcoco on GitHub.

Bug Report

Summary

In prefill/decode disaggregated deployments using etcd as the HA backend, Mooncake clients repeatedly log:

ERROR: [transport] Client received GoAway with error code ENHANCE_YOUR_CALM and debug data equal to ASCII "too_many_pings".

The issue is reproducible with Mooncake v0.3.12 and later. It was not observed with v0.3.11.post1. The apparent regression point is #2383.

The errors are usually more frequent shortly after startup and become less frequent over time as gRPC-Go increases the keepalive interval after each GOAWAY.

Environment

Mooncake: v0.3.12 and later
Deployment: prefill/decode disaggregation
HA backend: etcd
etcd client: go.etcd.io/etcd/client/v3 v3.5.21
gRPC-Go: google.golang.org/grpc v1.79.3
Reproduced with etcd v3.3.25
Multiple etcd endpoints make the issue easier to reproduce

Expected Behavior

Idle etcd connections should not be repeatedly closed and re-established during normal HA operation.

Actual Behavior

etcd periodically closes idle gRPC connections with:

ENHANCE_YOUR_CALM: too_many_pings

gRPC-Go then reconnects the affected SubChannel automatically. Mooncake continues running, but the connection churn produces error logs and unnecessary control-plane activity.

Setting the following etcd option does not resolve the issue:

--grpc-keepalive-min-time=1s

Reproduction

Start etcd with its normal client endpoint configuration. The issue is still reproducible if --grpc-keepalive-min-time=1s is added.
Start Mooncake master in HA mode with etcd as the HA backend.
Start a Mooncake client using the HA backend address.
Leave the deployment running with little or no control-plane activity.
Observe repeated too_many_pings errors in the client logs.

To confirm the affected endpoint, enable gRPC-Go logging before starting the client:

GRPC_GO_LOG_SEVERITY_LEVEL=info \
GRPC_GO_LOG_VERBOSITY_LEVEL=2 \
<mooncake client startup command>

Relevant logs can be filtered with:

rg 'original dial target|Subchannel picks|too_many_pings|received goaway|Connectivity change'

The resulting sequence shows that the affected channel targets an etcd endpoint:

Channel created with target etcd-endpoints://.../<etcd-ip>:2379
Subchannel picks address "<etcd-ip>:2379"
Client received GoAway ... "too_many_pings"
SubChannel: READY -> IDLE
Closing: "received goaway and there are no active streams"
SubChannel: IDLE -> CONNECTING -> READY

Root Cause

PR #2383 added transport-level gRPC keepalive to the Store etcd client to detect half-open connections after an etcd leader is suspended with SIGSTOP:

return clientv3.Config{
    Endpoints:            validEndpoints,
    DialTimeout:          5 * time.Second,
    DialKeepAliveTime:    10 * time.Second,
    DialKeepAliveTimeout: 3 * time.Second,
    PermitWithoutStream:  true,
}

The problematic option is:

PermitWithoutStream: true

It instructs gRPC-Go to send HTTP/2 PING frames even when the underlying transport has no active RPC streams.

etcd configures its gRPC enforcement policy with:

keepalive.EnforcementPolicy{
    MinTime:             configuredMinTime,
    PermitWithoutStream: false,
}

When a transport has an active RPC stream, etcd validates client PING intervals against --grpc-keepalive-min-time.

When a transport has no active streams, etcd does not use that configured minimum. Because PermitWithoutStream is false on the server, repeated idle PINGs are treated as policy violations. After enough violations, etcd sends:

GOAWAY
error code: ENHANCE_YOUR_CALM
debug data: too_many_pings

This also explains why lowering --grpc-keepalive-min-time does not fix the problem.

The issue is more visible with multiple endpoints because clientv3 uses round_robin and creates a SubChannel for each endpoint. A watch or lease stream may be active on one transport while the remaining transports are idle but still send PINGs.

After receiving too_many_pings, gRPC-Go closes the affected transport, reconnects it, and increases the keepalive interval for future connections. This explains why the errors become less frequent over time.

Impact

No process crashes, data corruption, or direct interruption of active data-path operations have been observed.

The GOAWAY normally targets a transport with no active RPC streams, so gRPC-Go can reconnect it automatically. However, the issue still causes:

Repeated TCP/gRPC connection establishment
Additional client and etcd CPU/network overhead
Noisy error logs
Potential connection storms in large multi-rank deployments
Possible transient latency for new etcd operations during reconnection
Possible delay in HA control-plane operations if reconnection overlaps with an etcd or master failover

The connection carrying an active leadership lease keepalive or master-view watch is normally not subject to the idle-stream enforcement path.

Proposed Fix

Remove PermitWithoutStream, or explicitly set it to false:

return clientv3.Config{
    Endpoints:            validEndpoints,
    DialTimeout:          5 * time.Second,
    DialKeepAliveTime:    storeDialKeepAliveTime,
    DialKeepAliveTimeout: storeDialKeepAliveTimeout,
    PermitWithoutStream:  false,
}

DialKeepAliveTime and DialKeepAliveTimeout should remain unchanged.

This should preserve the purpose of #2383. Leadership lease keepalive and watch operations use active RPC streams, so transport keepalive can still detect a half-open connection when the etcd leader is suspended. Only PINGs on completely idle transports are disabled.

Validation

The fix should be verified with:

An idle HA client connected to multiple etcd endpoints, confirming that no too_many_pings GOAWAY is received.
The #2383 etcd leader SIGSTOP/SIGKILL regression test, confirming that the master can still recover.
Master-view watch recovery after an etcd connection interruption.
Leadership lease keepalive failure and client reset/reconnection.
A comparison of SubChannel reconnection counts before and after the change.

Before submitting...

Searched for related issues and pull requests
Confirmed that v0.3.12 contains the apparent regression
Reproduced the issue with a minimal client
Verified the cause with a PermitWithoutStream=true/false comparison

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @Icedcoco on GitHub (Jul 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3142 Originally assigned to: @Icedcoco on GitHub. ### Bug Report ## Bug Report ### Summary In prefill/decode disaggregated deployments using etcd as the HA backend, Mooncake clients repeatedly log: ```text ERROR: [transport] Client received GoAway with error code ENHANCE_YOUR_CALM and debug data equal to ASCII "too_many_pings". ``` The issue is reproducible with Mooncake `v0.3.12` and later. It was not observed with `v0.3.11.post1`. The apparent regression point is #2383. The errors are usually more frequent shortly after startup and become less frequent over time as gRPC-Go increases the keepalive interval after each GOAWAY. ### Environment - Mooncake: `v0.3.12` and later - Deployment: prefill/decode disaggregation - HA backend: etcd - etcd client: `go.etcd.io/etcd/client/v3 v3.5.21` - gRPC-Go: `google.golang.org/grpc v1.79.3` - Reproduced with etcd `v3.3.25` - Multiple etcd endpoints make the issue easier to reproduce ### Expected Behavior Idle etcd connections should not be repeatedly closed and re-established during normal HA operation. ### Actual Behavior etcd periodically closes idle gRPC connections with: ```text ENHANCE_YOUR_CALM: too_many_pings ``` gRPC-Go then reconnects the affected SubChannel automatically. Mooncake continues running, but the connection churn produces error logs and unnecessary control-plane activity. Setting the following etcd option does not resolve the issue: ```bash --grpc-keepalive-min-time=1s ``` ### Reproduction 1. Start etcd with its normal client endpoint configuration. The issue is still reproducible if `--grpc-keepalive-min-time=1s` is added. 2. Start Mooncake master in HA mode with etcd as the HA backend. 3. Start a Mooncake client using the HA backend address. 4. Leave the deployment running with little or no control-plane activity. 5. Observe repeated `too_many_pings` errors in the client logs. To confirm the affected endpoint, enable gRPC-Go logging before starting the client: ```bash GRPC_GO_LOG_SEVERITY_LEVEL=info \ GRPC_GO_LOG_VERBOSITY_LEVEL=2 \ <mooncake client startup command> ``` Relevant logs can be filtered with: ```bash rg 'original dial target|Subchannel picks|too_many_pings|received goaway|Connectivity change' ``` The resulting sequence shows that the affected channel targets an etcd endpoint: ```text Channel created with target etcd-endpoints://.../<etcd-ip>:2379 Subchannel picks address "<etcd-ip>:2379" Client received GoAway ... "too_many_pings" SubChannel: READY -> IDLE Closing: "received goaway and there are no active streams" SubChannel: IDLE -> CONNECTING -> READY ``` ### Root Cause PR #2383 added transport-level gRPC keepalive to the Store etcd client to detect half-open connections after an etcd leader is suspended with `SIGSTOP`: ```go return clientv3.Config{ Endpoints: validEndpoints, DialTimeout: 5 * time.Second, DialKeepAliveTime: 10 * time.Second, DialKeepAliveTimeout: 3 * time.Second, PermitWithoutStream: true, } ``` The problematic option is: ```go PermitWithoutStream: true ``` It instructs gRPC-Go to send HTTP/2 PING frames even when the underlying transport has no active RPC streams. etcd configures its gRPC enforcement policy with: ```go keepalive.EnforcementPolicy{ MinTime: configuredMinTime, PermitWithoutStream: false, } ``` When a transport has an active RPC stream, etcd validates client PING intervals against `--grpc-keepalive-min-time`. When a transport has no active streams, etcd does not use that configured minimum. Because `PermitWithoutStream` is false on the server, repeated idle PINGs are treated as policy violations. After enough violations, etcd sends: ```text GOAWAY error code: ENHANCE_YOUR_CALM debug data: too_many_pings ``` This also explains why lowering `--grpc-keepalive-min-time` does not fix the problem. The issue is more visible with multiple endpoints because clientv3 uses `round_robin` and creates a SubChannel for each endpoint. A watch or lease stream may be active on one transport while the remaining transports are idle but still send PINGs. After receiving `too_many_pings`, gRPC-Go closes the affected transport, reconnects it, and increases the keepalive interval for future connections. This explains why the errors become less frequent over time. ### Impact No process crashes, data corruption, or direct interruption of active data-path operations have been observed. The GOAWAY normally targets a transport with no active RPC streams, so gRPC-Go can reconnect it automatically. However, the issue still causes: - Repeated TCP/gRPC connection establishment - Additional client and etcd CPU/network overhead - Noisy error logs - Potential connection storms in large multi-rank deployments - Possible transient latency for new etcd operations during reconnection - Possible delay in HA control-plane operations if reconnection overlaps with an etcd or master failover The connection carrying an active leadership lease keepalive or master-view watch is normally not subject to the idle-stream enforcement path. ### Proposed Fix Remove `PermitWithoutStream`, or explicitly set it to `false`: ```go return clientv3.Config{ Endpoints: validEndpoints, DialTimeout: 5 * time.Second, DialKeepAliveTime: storeDialKeepAliveTime, DialKeepAliveTimeout: storeDialKeepAliveTimeout, PermitWithoutStream: false, } ``` `DialKeepAliveTime` and `DialKeepAliveTimeout` should remain unchanged. This should preserve the purpose of #2383. Leadership lease keepalive and watch operations use active RPC streams, so transport keepalive can still detect a half-open connection when the etcd leader is suspended. Only PINGs on completely idle transports are disabled. ### Validation The fix should be verified with: 1. An idle HA client connected to multiple etcd endpoints, confirming that no `too_many_pings` GOAWAY is received. 2. The #2383 etcd leader `SIGSTOP`/`SIGKILL` regression test, confirming that the master can still recover. 3. Master-view watch recovery after an etcd connection interruption. 4. Leadership lease keepalive failure and client reset/reconnection. 5. A comparison of SubChannel reconnection counts before and after the change. ### Before submitting... - [x] Searched for related issues and pull requests - [x] Confirmed that `v0.3.12` contains the apparent regression - [x] Reproduced the issue with a minimal client - [x] Verified the cause with a `PermitWithoutStream=true/false` comparison ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:19:44 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:19:45 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 27, 2026):

Thanks for opening this issue, @Icedcoco!

Field	Value
Issue	#3142
GitHub user ID	`102317026`
Reporter	@Icedcoco

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 27, 2026): Thanks for opening this issue, @Icedcoco! | Field | Value | |-------|-------| | **Issue** | #3142 | | **GitHub user ID** | `102317026` | | **Reporter** | @Icedcoco | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:45 +00:00

Author

Owner

@whybeyoung commented on GitHub (Aug 4, 2026):

LGTM

@whybeyoung commented on GitHub (Aug 4, 2026): LGTM

bsakaguchi referenced this issue

2026-10-29 21:30:43 +00:00

[PR #805] [MERGED] [Misc] Mooncake EP & Mooncake Backend #1834