kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2704] [Bug]: Unhealthy store segment remains available and causes SGLang requests to fail (draft) #704

New issue

Closed

opened 2026-10-29 21:18:26 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:18:26 +00:00

Owner

Originally created by @greatwhole on GitHub (Jul 1, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2704

Originally assigned to: @Icedcoco on GitHub.

Bug Report

We are using Mooncake Store as the storage backend for SGLang in a Kubernetes deployment with RDMA enabled.

Recently, SGLang failed when accessing Mooncake Store through the Mooncake Store client. After investigation, the failure seemed to be related to only one store service. Other store services looked normal.

The problematic part is that after this store service became abnormal, the affected segment still seemed to be available in Mooncake Store metadata. As a result, requests involving this unhealthy store service continued to fail.

On the problematic store machine, dmesg kept printing RDMA-related QP local protection errors. We suspect that this store service or its underlying RDMA transport entered an abnormal state, but Mooncake Store did not exclude the affected segment from later requests.

Observed Behavior

We have multiple Mooncake Store services in the cluster.
Only one store service was affected by this issue.
Other store services looked normal.
Requests involving the affected store service kept failing.
The affected segment still seemed to be available in Mooncake Store metadata.
The issue cannot be reproduced stably at the moment.
The problematic store service did not recover automatically.

We also checked the restart history of this store service. It had restarted several times before without causing this issue. For the latest restart, Kubernetes events showed that it was killed under node-level memory pressure, likely because other containers on the same node had high memory usage. We mention this only as context, not as a confirmed root cause.

Kernel Logs

On the problematic store machine, dmesg repeatedly printed logs like the following:

Jun 30 15:01:31 maas-h20-a42h4kds-0001 kernel: mlx5_bond_0/1: QP 118513 error: local protection error (0x3a 0x0 0x95)
Jun 30 15:01:31 maas-h20-a42h4kds-0001 kernel: mlx5_bond_1/1: QP 122226 error: local protection error (0x3a 0x0 0x95)
Jun 30 15:01:31 maas-h20-a42h4kds-0001 kernel: mlx5_bond_1/1: QP 122227 error: local protection error (0x3a 0x0 0x95)
Jun 30 15:01:31 maas-h20-a42h4kds-0001 kernel: mlx5_bond_0/1: QP 118514 error: local protection error (0x3a 0x0 0x95)

The same pattern kept being printed continuously in dmesg.

Expected Behavior

When a store service becomes abnormal and can no longer serve requests correctly, Mooncake Store should avoid using the affected store service or segment for later requests.

More specifically, we expect Mooncake Store to:

detect that the affected store service or segment is no longer usable;
mark the affected segment as unavailable, or remove it from the available metadata;
avoid routing later requests to the unhealthy store service;
expose clear logs or metrics so users can diagnose this state.

Actual Behavior

The affected store service / segment still seemed to be considered available after it entered an abnormal state.

As a result, requests involving this unhealthy store service continued to fail.

Request

Could you help check whether Mooncake Store correctly handles this kind of partial store failure?

It would be helpful if Mooncake Store could invalidate or exclude the affected segment when a store service is still present but no longer able to serve RDMA requests correctly.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @greatwhole on GitHub (Jul 1, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2704 Originally assigned to: @Icedcoco on GitHub. ## Bug Report We are using Mooncake Store as the storage backend for SGLang in a Kubernetes deployment with RDMA enabled. Recently, SGLang failed when accessing Mooncake Store through the Mooncake Store client. After investigation, the failure seemed to be related to only one store service. Other store services looked normal. The problematic part is that after this store service became abnormal, the affected segment still seemed to be available in Mooncake Store metadata. As a result, requests involving this unhealthy store service continued to fail. On the problematic store machine, `dmesg` kept printing RDMA-related QP local protection errors. We suspect that this store service or its underlying RDMA transport entered an abnormal state, but Mooncake Store did not exclude the affected segment from later requests. ## Observed Behavior * We have multiple Mooncake Store services in the cluster. * Only one store service was affected by this issue. * Other store services looked normal. * Requests involving the affected store service kept failing. * The affected segment still seemed to be available in Mooncake Store metadata. * The issue cannot be reproduced stably at the moment. * The problematic store service did not recover automatically. We also checked the restart history of this store service. It had restarted several times before without causing this issue. For the latest restart, Kubernetes events showed that it was killed under node-level memory pressure, likely because other containers on the same node had high memory usage. We mention this only as context, not as a confirmed root cause. ## Kernel Logs On the problematic store machine, `dmesg` repeatedly printed logs like the following: ```text Jun 30 15:01:31 maas-h20-a42h4kds-0001 kernel: mlx5_bond_0/1: QP 118513 error: local protection error (0x3a 0x0 0x95) Jun 30 15:01:31 maas-h20-a42h4kds-0001 kernel: mlx5_bond_1/1: QP 122226 error: local protection error (0x3a 0x0 0x95) Jun 30 15:01:31 maas-h20-a42h4kds-0001 kernel: mlx5_bond_1/1: QP 122227 error: local protection error (0x3a 0x0 0x95) Jun 30 15:01:31 maas-h20-a42h4kds-0001 kernel: mlx5_bond_0/1: QP 118514 error: local protection error (0x3a 0x0 0x95) ``` The same pattern kept being printed continuously in `dmesg`. ## Expected Behavior When a store service becomes abnormal and can no longer serve requests correctly, Mooncake Store should avoid using the affected store service or segment for later requests. More specifically, we expect Mooncake Store to: * detect that the affected store service or segment is no longer usable; * mark the affected segment as unavailable, or remove it from the available metadata; * avoid routing later requests to the unhealthy store service; * expose clear logs or metrics so users can diagnose this state. ## Actual Behavior The affected store service / segment still seemed to be considered available after it entered an abnormal state. As a result, requests involving this unhealthy store service continued to fail. ## Request Could you help check whether Mooncake Store correctly handles this kind of partial store failure? It would be helpful if Mooncake Store could invalidate or exclude the affected segment when a store service is still present but no longer able to serve RDMA requests correctly. ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:18:26 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:18:27 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 1, 2026):

Thanks for opening this issue, @greatwhole!

Field	Value
Issue	#2704
GitHub user ID	`16155174`
Reporter	@greatwhole

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 1, 2026): Thanks for opening this issue, @greatwhole! | Field | Value | |-------|-------| | **Issue** | #2704 | | **GitHub user ID** | `16155174` | | **Reporter** | @greatwhole | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:18:27 +00:00

Author

Owner

@whybeyoung commented on GitHub (Jul 1, 2026):

@stmatengss

@whybeyoung commented on GitHub (Jul 1, 2026): @stmatengss

bsakaguchi commented

2026-10-29 21:18:27 +00:00

Author

Owner

@Icedcoco commented on GitHub (Jul 2, 2026):

Thanks for the detailed report.

From the current description, my initial guess is that this may be a data-plane health propagation issue rather than a pure metadata-server issue. The store service may still be alive from the master's point of view, so the segment remains mounted and its completed replicas can still be returned by metadata queries. At the same time, the RDMA path for that store service appears to be unhealthy, so requests that select replicas on that segment continue to fail.

I have not confirmed this yet from logs, so I would avoid calling it the root cause at this point.

Could you share a bit more information from one failure window?

Mooncake commit/version.
Mooncake master logs around the failure, especially lines containing:
client_expired, unmount_expired_mem_segment, Cleaned up HTTP metadata, MountSegment, ReMountSegment, QuerySegmentStatus.
Store/client logs from the affected store service around the same time.
A failed key's replica metadata, if available, showing which segment/endpoint is still being selected.
Whether the affected store service was still able to ping/remount to master after the RDMA errors started.
The values of MOONCAKE_LOCAL_HOSTNAME, MOONCAKE_MASTER, MOONCAKE_TE_META_DATA_SERVER, MOONCAKE_PROTOCOL, and the RDMA-related env vars.
Whether the segment name / endpoint stayed the same across the Kubernetes restarts.

The main thing I would like to distinguish is:

the client/store process expired and the master cleanup did not fully remove its replicas/metadata;
or the process stayed alive from the master's point of view, but its RDMA data path was broken, so master kept returning replicas on an unhealthy segment.

These two cases need different fixes.

@Icedcoco commented on GitHub (Jul 2, 2026): Thanks for the detailed report. From the current description, my initial guess is that this may be a data-plane health propagation issue rather than a pure metadata-server issue. The store service may still be alive from the master's point of view, so the segment remains mounted and its completed replicas can still be returned by metadata queries. At the same time, the RDMA path for that store service appears to be unhealthy, so requests that select replicas on that segment continue to fail. I have not confirmed this yet from logs, so I would avoid calling it the root cause at this point. Could you share a bit more information from one failure window? 1. Mooncake commit/version. 2. Mooncake master logs around the failure, especially lines containing: `client_expired`, `unmount_expired_mem_segment`, `Cleaned up HTTP metadata`, `MountSegment`, `ReMountSegment`, `QuerySegmentStatus`. 3. Store/client logs from the affected store service around the same time. 4. A failed key's replica metadata, if available, showing which segment/endpoint is still being selected. 5. Whether the affected store service was still able to ping/remount to master after the RDMA errors started. 6. The values of `MOONCAKE_LOCAL_HOSTNAME`, `MOONCAKE_MASTER`, `MOONCAKE_TE_META_DATA_SERVER`, `MOONCAKE_PROTOCOL`, and the RDMA-related env vars. 7. Whether the segment name / endpoint stayed the same across the Kubernetes restarts. The main thing I would like to distinguish is: - the client/store process expired and the master cleanup did not fully remove its replicas/metadata; - or the process stayed alive from the master's point of view, but its RDMA data path was broken, so master kept returning replicas on an unhealthy segment. These two cases need different fixes.

bsakaguchi referenced this issue

2026-10-29 21:30:05 +00:00

[PR #704] [MERGED] Add ability to do RDMA without nvidia-peermem #1758