kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4625] [Bug] Classic RDMA client (real_client.cpp): crash and silent write failures on peer-membership change #1174

New issue

Open

opened 2026-10-29 21:23:37 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-29 21:23:37 +00:00

Owner

Originally created by @sivabreddy on GitHub (Oct 13, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4625

Summary

Two distinct, reproducible problems in the "classic" RDMA client
(real_client.cpp), both triggered by a change in who's in the client pool
(a pod joining or leaving, by any method):

A crash. The client segfaults, taking down the entire host process, not
just the Mooncake connection.
A silent write failure. A Put can fail with TRANSFER_FAIL with no
error surfaced to the caller — the data is never actually written to the
shared store, but nothing above the Transfer Engine layer is told this
happened.

Both are reproducible on the latest PyPI release at time of testing
(mooncake-transfer-engine==0.3.13.post1, 2026-08-31). We believe both may
share a root cause with the already-fixed #3595 (a stale/self-referential
peer reference surviving a membership change), but in the older/classic
transport, which does not appear to carry that fix.

Environment

Mooncake version: mooncake-transfer-engine 0.3.13.post1 (PyPI,
installed via uv pip install mooncake-transfer-engine, no source
modifications)
Integration: vLLM v0.17.1 + LMCache v0.4.1-gdfc914cba
(LMCacheConnectorV1), using Mooncake Store as LMCache's remote backend
(remote_url: mooncakestore://...) — not the vLLM-native
MooncakeStoreConnector
Deployment: Kubernetes, 2-3 vLLM pods across 2 physical nodes, Mooncake
master + metadata server as separate lightweight pods in the same
namespace
Transport: RDMA (RoCEv2), Mellanox mlx5 devices, GID index 3
OS / platform: Linux containers (RHEL/UBI-based vLLM image)

mooncake-config.yaml used throughout (templated; local_hostname set to
each pod's own IP at container start):

chunk_size: 256
remote_url: "mooncakestore://<master-host>:50051/"
remote_serde: "naive"
local_cpu: True
max_local_cpu_size: 100
extra_config:
  local_hostname: "<pod IP, from $POD_IP>"
  metadata_server: "http://<metadata-host>:8080/metadata"
  master_server_address: "<master-host>:50051"
  protocol: "rdma"
  device_name: "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_6,mlx5_7,mlx5_8,mlx5_9"
  global_segment_size: 17179869184
  local_buffer_size: 8589934592
  transfer_timeout: 1

Of the 8 listed devices, only mlx5_0 has a real network attachment in our
environment; the other 7 fail to open at startup with a clear, non-fatal
warning and are excluded from the data path. This is expected and not itself
the bug — it becomes relevant in one test below.

Issue 1: segfault on peer-membership change

What we did (1): confirm the mechanism works normally

Deployed 2 vLLM+Mooncake pods (Pod A, Pod B) on separate nodes. Pod A stored a
prefix. Pod B requested the same prefix.

What we found (1)

A genuine cross-pod RDMA retrieve, confirmed from LMCache's own log line on
Pod B (Inference Engine computed tokens: 0 — Pod B never computed this
itself — followed by a real Retrieved N tokens line). No errors. This is
the expected, working behavior.

What we did (2): change pool membership, then send a real request

Changed which pods were in the pool, by each of these methods independently:

Force-deleted one pod (kubectl delete pod --grace-period=0 --force) and
let Kubernetes replace it.
Gracefully deleted one pod (kubectl delete pod, default grace period,
clean SIGTERM) and let Kubernetes replace it.
Simply added a brand-new pod to the same pool, with no deletion at all.

After each change, sent a real inference request that would cause a Mooncake
Get or Put — to either the new/replacement pod or an existing
pod that was never itself replaced.

What we found (2)

Whichever pod's turn it was to perform that next real operation segfaulted
— every time, regardless of which of the three membership-change methods was
used, and regardless of whether it was the new pod or an untouched existing
pod. This reproduced independently on 4 separate occasions. Exactly one
crash occurred per membership-change event. The affected process self-healed
after one container restart (10-15 seconds) and was stable again until the
next membership change elsewhere in the pool.

Crash log, one example (a replacement pod's first retrieve attempt):

(APIServer pid=1) INFO:     Application startup complete.
E1008 17:59:52.045347   870 real_client.cpp:2287] Client is not initialized
E1008 17:59:52.045735   870 real_client.cpp:2287] Client is not initialized
E1008 17:59:52.045954   870 real_client.cpp:2287] Client is not initialized
(EngineCore_DP0 pid=483) LMCache INFO: Reqid: ..., Total tokens 907, Inference Engine computed tokens: 0, LMCache hit tokens: 768, need to load: 768
E1008 17:59:52.051139  1057 real_client.cpp:3154] Client is not initialized
!!!!!!! Segfault encountered !!!!!!!
  File ".../runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
(APIServer pid=1) ERROR [core_client.py:691] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.

A second example (a replacement pod's own client failing to initialize at
all, before any successful operation) shows the retry-then-crash shape more
fully:

W1008 10:00:39.777729   167 real_client.cpp:858] Failed to create client on port 13476, retry 1/20
W1008 10:00:39.987325   167 real_client.cpp:858] Failed to create client on port 14180, retry 2/20
... (18 more retries, a different ephemeral port each time) ...
W1008 10:00:45.268965   167 real_client.cpp:858] Failed to create client on port 12411, retry 20/20
!!!!!!! Segfault encountered !!!!!!!
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.

What we did (3): test a candidate fix (MC_USE_TENT)

Found MC_USE_TENT as a string inside the installed mooncake/engine.so via
strings, hypothesizing it would opt into the newer TENT RDMA transport,
which already has a fix (#3598/#3603) for a structurally similar crash
(#3595: SIGSEGV in RailMonitor::updateBestMapping() after peer restart,
caused by a use-after-free of a cached remote Topology pointer). Set
MC_USE_TENT=1 in the pod environment and re-ran the identical
reproduction from step (2).

What we found (3)

No effect. The crash signature stayed in real_client.cpp, identical to
before. Either this wheel's TENT transport is not wired to that
flag, needs a different value, or is not compiled into this release at all.
We could not confirm which, since we only have the compiled wheel, not the
source. This is a ruled-out fix attempt, not a working one.

Our hypothesis for issue 1 (not confirmed against source)

The pattern — retries exhausted, then a crash referencing a "not
initialized" client state — closely matches #3595's description of a stale
reference surviving a peer-topology change and being dereferenced on the
failure-handling path without revalidation. If real_client.cpp holds a
similar cached pointer/reference to peer or segment state that becomes
invalid when a peer joins or leaves, and the retry-exhaustion path in this
file doesn't revalidate it before use, that would explain this exact
signature.

Issue 2: silent Put failure (TRANSFER_FAIL), no crash

What we did (4): store data, then check for errors beyond the success log

On an already-stable pool (no crash in progress), stored a prefix on one pod
and read that pod's complete log output for the request, not just the
single "Stored" confirmation line.

What we found (4)

The LMCache-level log line reports success regardless:

(LMCache) [req_id=...] Stored 512 out of total 512 tokens. size: 0.0625 GB, cost 2.0607 ms...

But immediately following it, in the same pod's log:

E1009 01:56:14.921368  1020 real_client.cpp:2176] Put operation failed with error: TRANSFER_FAIL
E1009 01:56:14.961222  1020 real_client.cpp:2176] Put operation failed with error: TRANSFER_FAIL

This failure is never surfaced anywhere above the Transfer Engine layer.
The calling LMCache integration only checks whether the local CPU-tier
store succeeded — it does not check whether the remote Mooncake Put
succeeded, so a failed remote write is reported identically to a successful
one. A later Get for this same data, from a different pod, then correctly
reports a miss — not because the master forgot good data, but because the
data was never actually written to the shared tier in the first place.

Full context from one reproduction, showing the Put failure immediately
preceded by a cascade of RDMA handshake failures against devices with no
real network attachment in our environment (expected/known — see
Environment above):

E1009 01:56:14.885888   903 transfer_metadata.cpp:1610] Handshake rejected by <peer>: Local RDMA endpoint unavailable for mlx5_1 <- <self>@mlx5_0
E1009 01:56:14.885928   903 rdma_endpoint.cpp:462] Active handshake RPC failed; resetting endpoint. rc=-200, ...
W1009 01:56:14.887291   903 worker_pool.cpp:477] Worker: Cannot make connection for endpoint: <peer>@mlx5_1, pausing peer rail and retrying through an alternate peer RNIC
W1009 01:56:14.887295   903 worker_pool.cpp:1395] Rail paused: peer=<peer>@mlx5_1 error_count=5 pause_ms=30000
E1009 01:56:14.891184   903 rdma_endpoint.cpp:1194] [Handshake] Failed to modify QP to RTR, ...: No such device
... (the same pattern repeats for mlx5_2, mlx5_3, mlx5_6, mlx5_7 in turn) ...
E1009 01:56:14.903507   903 transfer_metadata.cpp:1610] Handshake rejected by <peer>: Local RDMA endpoint unavailable for mlx5_7 <- <self>@mlx5_0
E1009 01:56:14.907513   903 rdma_endpoint.cpp:462] Active handshake RPC failed; resetting endpoint. rc=-200, ...
E1009 01:56:14.258356   997 transfer_task.cpp:800] Batch 139824606495840 completed with task failures: task_ids=[0, 1, 2]
E1009 01:56:14.258715   997 real_client.cpp:2176] Put operation failed with error: TRANSFER_FAIL

With transfer_timeout: 1 (1 second) in our config, cycling through 5-7
known-broken devices before falling back to the one working device does not
always complete in time.

What we did (5): test a candidate fix (reduce device_name to the one working device)

Hypothesizing the timeout was the issue, redeployed with device_name
reduced from the full 8-device list to just "mlx5_0" — the one device that
actually works — removing all the wasted retry cycles entirely. Re-ran the
identical store/retrieve reproduction from step (4).

What we found (5)

Did not fix it — and surfaced a different, more concerning signature for
the same TRANSFER_FAIL:

E1009 02:04:44.243228   905 rdma_endpoint.cpp:1194] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num:
  local=<pod-ip>:14082@mlx5_0, peer=<pod-ip>:14082@mlx5_0, qp_index=0,
  local_qp=26297, peer_qp=26297, local_gid=..., local_gid_index=3, peer_gid=...,
  peer_lid=0, path_mtu=5, port_num=1: No data available
W1009 02:04:44.243474   905 worker_pool.cpp:477] Worker: Cannot make connection for endpoint: <pod-ip>:14082@mlx5_0, no alternate peer RNIC is available; retrying without pausing peer rail
... (repeats ~10 times, a new QP number each time) ...
E1009 02:04:44.258356   997 transfer_task.cpp:800] Batch ... completed with task failures
E1009 02:04:44.258715   997 real_client.cpp:2176] Put operation failed with error: TRANSFER_FAIL

local and peer are the identical address (<pod-ip>:14082@mlx5_0
on both sides) — the engine is attempting to establish an RDMA connection to
itself. We separately observed a similarly-shaped self-collision under
tensor-parallel (TP=4) workers sharing one pod (distinct workers registering
distinct clients, causing address collisions) — but this reproduction used
TP=1, a single worker, with only one device configured, so whatever
produces this self-referential peer address is not specific to multi-worker
tensor parallelism, and reducing the device list does not avoid it. This is
a ruled-out fix attempt, not a working one — we reverted back to the full
8-device list, which carries no additional risk beyond the already-documented
handshake retries.

Our hypothesis for issue 2 (not confirmed against source)

Given both issues produce connection errors referencing stale or
self-referential peer/endpoint state, specifically following a change in
pool membership or device configuration, we suspect they may share a root
cause in how the Transfer Engine selects and tracks peer connection state —
but we have not confirmed this against source, only against observed
behavior, since we only have the compiled wheel.

What does mitigate (not fix) issue 1

Not a code question, included for completeness since it shaped how
seriously we treat this: issue 1's crash is a one-time, self-healing event
per membership-change — the affected pod recovers automatically in 10-15
seconds. Running at least 3 client pods, replacing them one at a time
(never all at once), kept at least 2 pods continuously healthy and serving
real traffic through 3 full replacement cycles in our testing, including
one cycle where the crash genuinely fired. This does not reduce how often
the crash happens — it only ensures redundant capacity survives it. Issue 2
has no equivalent mitigation we could find; it is silent by nature.

Suggested next step

Given the structural similarity to #3595, the most likely fix for issue 1 is
the same one applied there, ported to real_client.cpp: hold any cached
peer/segment reference the failure-handling path touches as a
shared_ptr/revalidated reference rather than a raw pointer that can outlive
its backing object across a membership change, or re-validate it on the
retry-exhaustion path before use. Issue 2's self-referential peer address
(step 5's finding) suggests a related bug in peer/endpoint selection that
may be worth investigating together with issue 1, though we have not
confirmed they share a code path. Happy to test a patched build against the
exact reproductions above if one becomes available.

Found while validating a production KV-cache deployment plan. Happy to share more raw logs or test a patched build against the exact reproductions above.

Originally created by @sivabreddy on GitHub (Oct 13, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4625 ## Summary Two distinct, reproducible problems in the "classic" RDMA client (`real_client.cpp`), both triggered by a change in who's in the client pool (a pod joining or leaving, by any method): 1. **A crash.** The client segfaults, taking down the entire host process, not just the Mooncake connection. 2. **A silent write failure.** A `Put` can fail with `TRANSFER_FAIL` with no error surfaced to the caller — the data is never actually written to the shared store, but nothing above the Transfer Engine layer is told this happened. Both are reproducible on the latest PyPI release at time of testing (`mooncake-transfer-engine==0.3.13.post1`, 2026-08-31). We believe both may share a root cause with the already-fixed #3595 (a stale/self-referential peer reference surviving a membership change), but in the older/classic transport, which does not appear to carry that fix. ## Environment - **Mooncake version**: `mooncake-transfer-engine` 0.3.13.post1 (PyPI, installed via `uv pip install mooncake-transfer-engine`, no source modifications) - **Integration**: vLLM v0.17.1 + LMCache `v0.4.1-gdfc914cba` (`LMCacheConnectorV1`), using Mooncake Store as LMCache's remote backend (`remote_url: mooncakestore://...`) — not the vLLM-native `MooncakeStoreConnector` - **Deployment**: Kubernetes, 2-3 vLLM pods across 2 physical nodes, Mooncake master + metadata server as separate lightweight pods in the same namespace - **Transport**: RDMA (RoCEv2), Mellanox mlx5 devices, `GID index 3` - **OS / platform**: Linux containers (RHEL/UBI-based vLLM image) `mooncake-config.yaml` used throughout (templated; `local_hostname` set to each pod's own IP at container start): ```yaml chunk_size: 256 remote_url: "mooncakestore://<master-host>:50051/" remote_serde: "naive" local_cpu: True max_local_cpu_size: 100 extra_config: local_hostname: "<pod IP, from $POD_IP>" metadata_server: "http://<metadata-host>:8080/metadata" master_server_address: "<master-host>:50051" protocol: "rdma" device_name: "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_6,mlx5_7,mlx5_8,mlx5_9" global_segment_size: 17179869184 local_buffer_size: 8589934592 transfer_timeout: 1 ``` Of the 8 listed devices, only `mlx5_0` has a real network attachment in our environment; the other 7 fail to open at startup with a clear, non-fatal warning and are excluded from the data path. This is expected and not itself the bug — it becomes relevant in one test below. --- ## Issue 1: segfault on peer-membership change ### What we did (1): confirm the mechanism works normally Deployed 2 vLLM+Mooncake pods (Pod A, Pod B) on separate nodes. Pod A stored a prefix. Pod B requested the same prefix. ### What we found (1) A genuine cross-pod RDMA retrieve, confirmed from LMCache's own log line on Pod B (`Inference Engine computed tokens: 0` — Pod B never computed this itself — followed by a real `Retrieved N tokens` line). No errors. This is the expected, working behavior. ### What we did (2): change pool membership, then send a real request Changed which pods were in the pool, by each of these methods independently: - Force-deleted one pod (`kubectl delete pod --grace-period=0 --force`) and let Kubernetes replace it. - Gracefully deleted one pod (`kubectl delete pod`, default grace period, clean SIGTERM) and let Kubernetes replace it. - Simply added a brand-new pod to the same pool, with no deletion at all. After each change, sent a real inference request that would cause a Mooncake `Get` or `Put` — to **either** the new/replacement pod **or** an existing pod that was never itself replaced. ### What we found (2) **Whichever pod's turn it was to perform that next real operation segfaulted — every time, regardless of which of the three membership-change methods was used, and regardless of whether it was the new pod or an untouched existing pod.** This reproduced independently on 4 separate occasions. Exactly one crash occurred per membership-change event. The affected process self-healed after one container restart (10-15 seconds) and was stable again until the next membership change elsewhere in the pool. Crash log, one example (a replacement pod's first retrieve attempt): ``` (APIServer pid=1) INFO: Application startup complete. E1008 17:59:52.045347 870 real_client.cpp:2287] Client is not initialized E1008 17:59:52.045735 870 real_client.cpp:2287] Client is not initialized E1008 17:59:52.045954 870 real_client.cpp:2287] Client is not initialized (EngineCore_DP0 pid=483) LMCache INFO: Reqid: ..., Total tokens 907, Inference Engine computed tokens: 0, LMCache hit tokens: 768, need to load: 768 E1008 17:59:52.051139 1057 real_client.cpp:3154] Client is not initialized !!!!!!! Segfault encountered !!!!!!! File ".../runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd (APIServer pid=1) ERROR [core_client.py:691] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client. vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. ``` A second example (a replacement pod's own client failing to initialize at all, before any successful operation) shows the retry-then-crash shape more fully: ``` W1008 10:00:39.777729 167 real_client.cpp:858] Failed to create client on port 13476, retry 1/20 W1008 10:00:39.987325 167 real_client.cpp:858] Failed to create client on port 14180, retry 2/20 ... (18 more retries, a different ephemeral port each time) ... W1008 10:00:45.268965 167 real_client.cpp:858] Failed to create client on port 12411, retry 20/20 !!!!!!! Segfault encountered !!!!!!! vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. ``` ### What we did (3): test a candidate fix (MC_USE_TENT) Found `MC_USE_TENT` as a string inside the installed `mooncake/engine.so` via `strings`, hypothesizing it would opt into the newer TENT RDMA transport, which already has a fix (#3598/#3603) for a structurally similar crash (#3595: SIGSEGV in `RailMonitor::updateBestMapping()` after peer restart, caused by a use-after-free of a cached remote `Topology` pointer). Set `MC_USE_TENT=1` in the pod environment and re-ran the identical reproduction from step (2). ### What we found (3) **No effect.** The crash signature stayed in `real_client.cpp`, identical to before. Either this wheel's TENT transport is not wired to that flag, needs a different value, or is not compiled into this release at all. We could not confirm which, since we only have the compiled wheel, not the source. This is a ruled-out fix attempt, not a working one. ### Our hypothesis for issue 1 (not confirmed against source) The pattern — retries exhausted, then a crash referencing a "not initialized" client state — closely matches #3595's description of a stale reference surviving a peer-topology change and being dereferenced on the failure-handling path without revalidation. If `real_client.cpp` holds a similar cached pointer/reference to peer or segment state that becomes invalid when a peer joins or leaves, and the retry-exhaustion path in this file doesn't revalidate it before use, that would explain this exact signature. --- ## Issue 2: silent Put failure (TRANSFER_FAIL), no crash ### What we did (4): store data, then check for errors beyond the success log On an already-stable pool (no crash in progress), stored a prefix on one pod and read that pod's **complete** log output for the request, not just the single "Stored" confirmation line. ### What we found (4) The LMCache-level log line reports success regardless: ``` (LMCache) [req_id=...] Stored 512 out of total 512 tokens. size: 0.0625 GB, cost 2.0607 ms... ``` But immediately following it, in the same pod's log: ``` E1009 01:56:14.921368 1020 real_client.cpp:2176] Put operation failed with error: TRANSFER_FAIL E1009 01:56:14.961222 1020 real_client.cpp:2176] Put operation failed with error: TRANSFER_FAIL ``` **This failure is never surfaced anywhere above the Transfer Engine layer.** The calling LMCache integration only checks whether the *local* CPU-tier store succeeded — it does not check whether the remote Mooncake `Put` succeeded, so a failed remote write is reported identically to a successful one. A later `Get` for this same data, from a different pod, then correctly reports a miss — not because the master forgot good data, but because the data was never actually written to the shared tier in the first place. Full context from one reproduction, showing the Put failure immediately preceded by a cascade of RDMA handshake failures against devices with no real network attachment in our environment (expected/known — see Environment above): ``` E1009 01:56:14.885888 903 transfer_metadata.cpp:1610] Handshake rejected by <peer>: Local RDMA endpoint unavailable for mlx5_1 <- <self>@mlx5_0 E1009 01:56:14.885928 903 rdma_endpoint.cpp:462] Active handshake RPC failed; resetting endpoint. rc=-200, ... W1009 01:56:14.887291 903 worker_pool.cpp:477] Worker: Cannot make connection for endpoint: <peer>@mlx5_1, pausing peer rail and retrying through an alternate peer RNIC W1009 01:56:14.887295 903 worker_pool.cpp:1395] Rail paused: peer=<peer>@mlx5_1 error_count=5 pause_ms=30000 E1009 01:56:14.891184 903 rdma_endpoint.cpp:1194] [Handshake] Failed to modify QP to RTR, ...: No such device ... (the same pattern repeats for mlx5_2, mlx5_3, mlx5_6, mlx5_7 in turn) ... E1009 01:56:14.903507 903 transfer_metadata.cpp:1610] Handshake rejected by <peer>: Local RDMA endpoint unavailable for mlx5_7 <- <self>@mlx5_0 E1009 01:56:14.907513 903 rdma_endpoint.cpp:462] Active handshake RPC failed; resetting endpoint. rc=-200, ... E1009 01:56:14.258356 997 transfer_task.cpp:800] Batch 139824606495840 completed with task failures: task_ids=[0, 1, 2] E1009 01:56:14.258715 997 real_client.cpp:2176] Put operation failed with error: TRANSFER_FAIL ``` With `transfer_timeout: 1` (1 second) in our config, cycling through 5-7 known-broken devices before falling back to the one working device does not always complete in time. ### What we did (5): test a candidate fix (reduce device_name to the one working device) Hypothesizing the timeout was the issue, redeployed with `device_name` reduced from the full 8-device list to just `"mlx5_0"` — the one device that actually works — removing all the wasted retry cycles entirely. Re-ran the identical store/retrieve reproduction from step (4). ### What we found (5) **Did not fix it — and surfaced a different, more concerning signature for the same `TRANSFER_FAIL`:** ``` E1009 02:04:44.243228 905 rdma_endpoint.cpp:1194] [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: local=<pod-ip>:14082@mlx5_0, peer=<pod-ip>:14082@mlx5_0, qp_index=0, local_qp=26297, peer_qp=26297, local_gid=..., local_gid_index=3, peer_gid=..., peer_lid=0, path_mtu=5, port_num=1: No data available W1009 02:04:44.243474 905 worker_pool.cpp:477] Worker: Cannot make connection for endpoint: <pod-ip>:14082@mlx5_0, no alternate peer RNIC is available; retrying without pausing peer rail ... (repeats ~10 times, a new QP number each time) ... E1009 02:04:44.258356 997 transfer_task.cpp:800] Batch ... completed with task failures E1009 02:04:44.258715 997 real_client.cpp:2176] Put operation failed with error: TRANSFER_FAIL ``` **`local` and `peer` are the identical address** (`<pod-ip>:14082@mlx5_0` on both sides) — the engine is attempting to establish an RDMA connection to **itself**. We separately observed a similarly-shaped self-collision under tensor-parallel (TP=4) workers sharing one pod (distinct workers registering distinct clients, causing address collisions) — but **this reproduction used TP=1, a single worker, with only one device configured**, so whatever produces this self-referential peer address is not specific to multi-worker tensor parallelism, and reducing the device list does not avoid it. This is a ruled-out fix attempt, not a working one — we reverted back to the full 8-device list, which carries no additional risk beyond the already-documented handshake retries. ### Our hypothesis for issue 2 (not confirmed against source) Given both issues produce connection errors referencing stale or self-referential peer/endpoint state, specifically following a change in pool membership or device configuration, we suspect they may share a root cause in how the Transfer Engine selects and tracks peer connection state — but we have not confirmed this against source, only against observed behavior, since we only have the compiled wheel. --- ## What does mitigate (not fix) issue 1 Not a code question, included for completeness since it shaped how seriously we treat this: issue 1's crash is a one-time, self-healing event per membership-change — the affected pod recovers automatically in 10-15 seconds. Running at least 3 client pods, replacing them one at a time (never all at once), kept at least 2 pods continuously healthy and serving real traffic through 3 full replacement cycles in our testing, including one cycle where the crash genuinely fired. This does not reduce how often the crash happens — it only ensures redundant capacity survives it. Issue 2 has no equivalent mitigation we could find; it is silent by nature. ## Suggested next step Given the structural similarity to #3595, the most likely fix for issue 1 is the same one applied there, ported to `real_client.cpp`: hold any cached peer/segment reference the failure-handling path touches as a `shared_ptr`/revalidated reference rather than a raw pointer that can outlive its backing object across a membership change, or re-validate it on the retry-exhaustion path before use. Issue 2's self-referential peer address (step 5's finding) suggests a related bug in peer/endpoint selection that may be worth investigating together with issue 1, though we have not confirmed they share a code path. Happy to test a patched build against the exact reproductions above if one becomes available. --- *Found while validating a production KV-cache deployment plan. Happy to share more raw logs or test a patched build against the exact reproductions above.*

bsakaguchi added the

Transfer Engine

bug

labels

2026-10-29 21:23:37 +00:00

bsakaguchi commented

2026-10-29 21:23:37 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 13, 2026):

Thanks for opening this issue, @sivabreddy!

Field	Value
Issue	#4625
GitHub user ID	`113816799`
Reporter	@sivabreddy

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 13, 2026): Thanks for opening this issue, @sivabreddy! | Field | Value | |-------|-------| | **Issue** | #4625 | | **GitHub user ID** | `113816799` | | **Reporter** | @sivabreddy | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:23:38 +00:00

Author

Owner

@sivabreddy commented on GitHub (Oct 13, 2026):

Follow-up: retested with a corrected RDMA device list and checked TENT directly from source

Posting an update since we found something that changes the picture from the original report above.

1. Is TENT available as a way around this? No — confirmed from source, not guessed

We'd read about TENT (the newer failure-handling transport) and wondered if it was reachable from
the published package. Cloned kvcache-ai/Mooncake fresh and checked directly:

mooncake-common/common.cmake: option(USE_TENT "option for building Mooncake TENT" OFF) —
TENT is a CMake build-time flag, default OFF, not a runtime environment variable.
This confirms why our earlier MC_USE_TENT=1 attempt (mentioned in the original report) had no
effect — there is no such runtime flag in this transport. The published
mooncake-transfer-engine PyPI wheel is a standard build (USE_TENT not set), so TENT is not
present in it regardless of environment variables.

2. Our original device-list mismatch, corrected — the crash got worse, not better

The original report used a device_name list of 8 devices, of which only 1 (mlx5_0) actually
had a routable RoCEv2 identity on our cluster — we wondered if this mismatch itself was a
contributing factor. Re-checked fresh: still true, no change, confirmed via ibv_devinfo and a
scan of /sys/class/infiniband/*/ports/1/gids/* on both nodes.

Redeployed with device_name corrected to "mlx5_0" only (the one real device). Result:

All three client pods failed to establish a working Mooncake RDMA client at startup — 20/20
connection retries exhausted on every pod, 0/3 successful:

W1013 11:51:04.798380  1450 real_client.cpp:858] Failed to create client on port 12628, retry 1/20
...
W1013 11:52:28.016795  1450 real_client.cpp:858] Failed to create client on port 13693, retry 20/20
E1013 11:52:28.116932  1450 real_client.cpp:865] Failed to create client after 20 retries
E1013 11:52:28.118700  1450 real_client.cpp:3379] Client is not initialized

A separate issue found here: the Mooncake master's own metrics showed Clients: 0 for the
entire test, even though every pod logged MooncakeConnector initialized successfully on the
LMCache side. LMCache's MooncakestoreConnector appears to report success regardless of whether
the underlying C++ client actually has a working connection — there's no visible warning anywhere
that the connector is non-functional. (We realize this specific piece may belong on the LMCache
side rather than here, flagging for completeness since it compounds the diagnosis difficulty.)

The crash itself still reproduced, and more readily than before — not gated behind a
membership-change event anymore, but firing on literally the first real cache operation:

LMCache INFO: Reqid: ..., LMCache hit tokens: 256, need to load: 256
E1013 12:09:16.147480  1949 real_client.cpp:2287] Client is not initialized
E1013 12:09:16.701977  2350 real_client.cpp:3154] Client is not initialized
!!!!!!! Segfault encountered !!!!!!!

Confirmed independently on 2 of the 3 pods (identical file/line numbers, identical trigger: a
lookup reporting a partial local hit that needs to load the remainder from the remote tier). One
pod was also force-deleted and replaced to confirm the original membership-change trigger
specifically — same result, crashes immediately on the replacement pod's first real request.
Total: 5 independent crash observations across 3 pods, 100% reproduction rate, 0 successful real
cache operations observed anywhere in this retest.

3. What this changes about our read of the original report

Correcting the device-list mismatch does not fix the crash — if anything, it removes the
one device that was letting any connection succeed at all in our original 8-device config,
without touching whatever is fragile in real_client.cpp's client-creation/reconnection path.
We could not re-isolate the silent-write-failure bug from the original report under this
config, since the crash now pre-empts it entirely (the engine dies before a store can ever
complete) — not evidence that bug is fixed, just that this config's failure mode is more severe
and reaches a different point first.

Happy to share the full reproduction (manifests, complete logs) if useful, or test against a
patched build if one becomes available.

@sivabreddy commented on GitHub (Oct 13, 2026): ## Follow-up: retested with a corrected RDMA device list and checked TENT directly from source Posting an update since we found something that changes the picture from the original report above. ### 1. Is TENT available as a way around this? No — confirmed from source, not guessed We'd read about TENT (the newer failure-handling transport) and wondered if it was reachable from the published package. Cloned `kvcache-ai/Mooncake` fresh and checked directly: - `mooncake-common/common.cmake`: `option(USE_TENT "option for building Mooncake TENT" OFF)` — **TENT is a CMake build-time flag, default OFF**, not a runtime environment variable. - This confirms why our earlier `MC_USE_TENT=1` attempt (mentioned in the original report) had no effect — there is no such runtime flag in this transport. The published `mooncake-transfer-engine` PyPI wheel is a standard build (`USE_TENT` not set), so TENT is not present in it regardless of environment variables. ### 2. Our original device-list mismatch, corrected — the crash got worse, not better The original report used a `device_name` list of 8 devices, of which only 1 (`mlx5_0`) actually had a routable RoCEv2 identity on our cluster — we wondered if this mismatch itself was a contributing factor. Re-checked fresh: still true, no change, confirmed via `ibv_devinfo` and a scan of `/sys/class/infiniband/*/ports/1/gids/*` on both nodes. Redeployed with `device_name` corrected to `"mlx5_0"` only (the one real device). Result: **All three client pods failed to establish a working Mooncake RDMA client at startup — 20/20 connection retries exhausted on every pod, 0/3 successful:** ``` W1013 11:51:04.798380 1450 real_client.cpp:858] Failed to create client on port 12628, retry 1/20 ... W1013 11:52:28.016795 1450 real_client.cpp:858] Failed to create client on port 13693, retry 20/20 E1013 11:52:28.116932 1450 real_client.cpp:865] Failed to create client after 20 retries E1013 11:52:28.118700 1450 real_client.cpp:3379] Client is not initialized ``` **A separate issue found here:** the Mooncake master's own metrics showed `Clients: 0` for the entire test, even though every pod logged `MooncakeConnector initialized successfully` on the LMCache side. LMCache's `MooncakestoreConnector` appears to report success regardless of whether the underlying C++ client actually has a working connection — there's no visible warning anywhere that the connector is non-functional. (We realize this specific piece may belong on the LMCache side rather than here, flagging for completeness since it compounds the diagnosis difficulty.) **The crash itself still reproduced, and more readily than before — not gated behind a membership-change event anymore, but firing on literally the first real cache operation:** ``` LMCache INFO: Reqid: ..., LMCache hit tokens: 256, need to load: 256 E1013 12:09:16.147480 1949 real_client.cpp:2287] Client is not initialized E1013 12:09:16.701977 2350 real_client.cpp:3154] Client is not initialized !!!!!!! Segfault encountered !!!!!!! ``` Confirmed independently on 2 of the 3 pods (identical file/line numbers, identical trigger: a lookup reporting a partial local hit that needs to load the remainder from the remote tier). One pod was also force-deleted and replaced to confirm the original membership-change trigger specifically — same result, crashes immediately on the replacement pod's first real request. Total: 5 independent crash observations across 3 pods, 100% reproduction rate, 0 successful real cache operations observed anywhere in this retest. ### 3. What this changes about our read of the original report - Correcting the device-list mismatch does **not** fix the crash — if anything, it removes the one device that was letting any connection succeed at all in our original 8-device config, without touching whatever is fragile in `real_client.cpp`'s client-creation/reconnection path. - We could not re-isolate the silent-write-failure bug from the original report under this config, since the crash now pre-empts it entirely (the engine dies before a store can ever complete) — not evidence that bug is fixed, just that this config's failure mode is more severe and reaches a different point first. Happy to share the full reproduction (manifests, complete logs) if useful, or test against a patched build if one becomes available.

bsakaguchi commented

2026-10-29 21:23:38 +00:00

Author

Owner

@xiaodouzi666 commented on GitHub (Oct 19, 2026):

Hi, I'll post some fix measures later today, if you have any questions about that please do not hesitate to leave a comment, thx!

@xiaodouzi666 commented on GitHub (Oct 19, 2026): Hi, I'll post some fix measures later today, if you have any questions about that please do not hesitate to leave a comment, thx!

bsakaguchi commented

2026-10-29 21:23:38 +00:00

Author

Owner

@sivabreddy commented on GitHub (Oct 19, 2026):

Update after a re-test: the failed writes came from our RDMA configuration; the crash is the null-client path that #4633 fixes.

On 2026-10-17 we re-tested with the same versions: vLLM 0.17.1, LMCache 0.4.1 (LMCacheConnectorV1, mooncakestore://), Mooncake 0.3.13.post1 and Llama-3.1-8B TP1. We used the same LMCache YAML and the HTTP metadata server, with 3 pods on 2 nodes. Several RDMA pods on each node share one RoCE device. Differences from the original run: an Ubuntu CUDA image with pip, and LMCache 0.4.1 built from source.

1. What was wrong on our side

A fixed GID index. We set MC_GID_INDEX=3. A cluster policy also injected NCCL_IB_GID_INDEX=3, which Mooncake uses when MC_GID_INDEX is not set (mooncake-transfer-engine/src/config.cpp L144-145). When pods share one RDMA device, each pod also sees the GID entries of the other pods. So a fixed index points to at most one pod's own GID, or to another pod's GID. Without a fixed index, findBestGidIndex ranks GIDs that have a readable sysfs netdev first, and each pod selected its own IPv4 GID (indexes 11–27).
RDMA devices without a network in device_name. We listed 8 devices, and only one has a network.

2. Re-test with automatic GID selection

	8-device list	connected device only
Requests during and after 3 membership changes (force delete, delete, new pod)	51 of 51 answered	51 of 51 answered
Segfaults / engine restarts / `Failed to create client` retries	0 / 0 / 0	0 / 0 / 0
`Put ... TRANSFER_FAIL` log lines (4 pods)	320 (master rate metrics: no completed `PutEnd`)	0
Cross-pod loads	none	yes (2,560 of 2,560 tokens)

Store-only test (Python client, 4 clients, 2 per node, 16 keys of 4 MB each, P2PHANDSHAKE, connected device only): with MC_GID_INDEX=3, 16 of 16 puts failed on every client and 0 of 64 keys were readable; with automatic selection, 64 of 64 keys were readable.

3. The crash: #4633 matches our logs
In our crash logs the process first fails to create the client (Failed to create client ... retry 20/20) or logs Client is not initialized; in one log the last Mooncake line before the segfault is real_client.cpp:3154] Client is not initialized (batch_get_buffer_internal, 0.3.13.post1). In 0.3.13.post1, RealClient::batch_get_buffer (L3366-3373) passes an always-true success predicate to execute_timed_operation, so the observer calls client_->ObserveTransferOperation(...) with a null client_. The same pattern is in the other 7 batch wrappers (L4024, 4086, 4377, 4656, 5435, 6053, 6097) and is still on main. This is exactly what #4633 fixes (thanks @pjdurden). So the chain in our case was: fixed GID index → client creation fails → LMCache keeps calling the batch API (it does not check the setup() result) → null client_ in the metrics observer → segfault. With automatic GID selection the client is created and the crash path is not reached, which is why the re-test had 0 crashes. We did not re-run the crash setup with #4633 applied.

4. What we suggest

Merge #4633 (or an equivalent null check). It turns the crash into per-key errors.
Docs: docs/source/design/transfer-engine/index.md L473 says the default of MC_GID_INDEX is 3. In the code the default is -1 = automatic selection (mooncake-transfer-engine/include/config.h L46). Proposed text: "MC_GID_INDEX (or NCCL_IB_GID_INDEX when MC_GID_INDEX is not set): GID index per device. Default: automatic selection, which prefers a GID that has a network device. Do not set a fixed index when several containers share one RDMA device: the GID table holds entries for the interfaces of all containers on the node." We can send this as a PR.
For users: when several pods share one RDMA device, do not set MC_GID_INDEX / NCCL_IB_GID_INDEX, and list only RDMA devices that have a network in device_name.
The "silent" part (LMCache logs Stored ... after a failed Mooncake Put, and continues after a failed setup()) is on the LMCache side; we will raise it there. We withdraw the #3595 hypothesis.

@xiaodouzi666 thank you for the offer. #4633 already covers the crash; feel free to use or improve anything above. The membership change itself was not the cause.

We will close this issue once #4633 (or an equivalent) is merged, unless you prefer to close it earlier.

@sivabreddy commented on GitHub (Oct 19, 2026): **Update after a re-test: the failed writes came from our RDMA configuration; the crash is the null-client path that #4633 fixes.** On 2026-10-17 we re-tested with the same versions: vLLM 0.17.1, LMCache 0.4.1 (`LMCacheConnectorV1`, `mooncakestore://`), Mooncake 0.3.13.post1 and Llama-3.1-8B TP1. We used the same LMCache YAML and the HTTP metadata server, with 3 pods on 2 nodes. Several RDMA pods on each node share one RoCE device. Differences from the original run: an Ubuntu CUDA image with pip, and LMCache 0.4.1 built from source. **1. What was wrong on our side** - **A fixed GID index.** We set `MC_GID_INDEX=3`. A cluster policy also injected `NCCL_IB_GID_INDEX=3`, which Mooncake uses when `MC_GID_INDEX` is not set (`mooncake-transfer-engine/src/config.cpp` L144-145). When pods share one RDMA device, each pod also sees the GID entries of the other pods. So a fixed index points to at most one pod's own GID, or to another pod's GID. Without a fixed index, `findBestGidIndex` ranks GIDs that have a readable sysfs netdev first, and each pod selected its own IPv4 GID (indexes 11–27). - **RDMA devices without a network in `device_name`.** We listed 8 devices, and only one has a network. **2. Re-test with automatic GID selection** | | 8-device list | connected device only | |---|---|---| | Requests during and after 3 membership changes (force delete, delete, new pod) | 51 of 51 answered | 51 of 51 answered | | Segfaults / engine restarts / `Failed to create client` retries | 0 / 0 / 0 | 0 / 0 / 0 | | `Put ... TRANSFER_FAIL` log lines (4 pods) | 320 (master rate metrics: no completed `PutEnd`) | 0 | | Cross-pod loads | none | yes (2,560 of 2,560 tokens) | Store-only test (Python client, 4 clients, 2 per node, 16 keys of 4 MB each, P2PHANDSHAKE, connected device only): with `MC_GID_INDEX=3`, 16 of 16 puts failed on every client and 0 of 64 keys were readable; with automatic selection, 64 of 64 keys were readable. **3. The crash: #4633 matches our logs** In our crash logs the process first fails to create the client (`Failed to create client ... retry 20/20`) or logs `Client is not initialized`; in one log the last Mooncake line before the segfault is `real_client.cpp:3154] Client is not initialized` (`batch_get_buffer_internal`, 0.3.13.post1). In 0.3.13.post1, `RealClient::batch_get_buffer` (L3366-3373) passes an always-true success predicate to `execute_timed_operation`, so the observer calls `client_->ObserveTransferOperation(...)` with a null `client_`. The same pattern is in the other 7 batch wrappers (L4024, 4086, 4377, 4656, 5435, 6053, 6097) and is still on `main`. This is exactly what #4633 fixes (thanks @pjdurden). So the chain in our case was: fixed GID index → client creation fails → LMCache keeps calling the batch API (it does not check the `setup()` result) → null `client_` in the metrics observer → segfault. With automatic GID selection the client is created and the crash path is not reached, which is why the re-test had 0 crashes. We did not re-run the crash setup with #4633 applied. **4. What we suggest** - Merge #4633 (or an equivalent null check). It turns the crash into per-key errors. - Docs: `docs/source/design/transfer-engine/index.md` L473 says the default of `MC_GID_INDEX` is 3. In the code the default is -1 = automatic selection (`mooncake-transfer-engine/include/config.h` L46). Proposed text: "`MC_GID_INDEX` (or `NCCL_IB_GID_INDEX` when `MC_GID_INDEX` is not set): GID index per device. Default: automatic selection, which prefers a GID that has a network device. Do not set a fixed index when several containers share one RDMA device: the GID table holds entries for the interfaces of all containers on the node." We can send this as a PR. - For users: when several pods share one RDMA device, do not set `MC_GID_INDEX` / `NCCL_IB_GID_INDEX`, and list only RDMA devices that have a network in `device_name`. - The "silent" part (LMCache logs `Stored ...` after a failed Mooncake Put, and continues after a failed `setup()`) is on the LMCache side; we will raise it there. We withdraw the #3595 hypothesis. @xiaodouzi666 thank you for the offer. #4633 already covers the crash; feel free to use or improve anything above. The membership change itself was not the cause. We will close this issue once #4633 (or an equivalent) is merged, unless you prefer to close it earlier.

bsakaguchi commented

2026-10-29 21:23:38 +00:00

Author

Owner

@xiaodouzi666 commented on GitHub (Oct 26, 2026):

Hi, I believe this crash has been solved, in order to solve that, I implemented the fix under these two PRs (#4932, #4933), please take a look, and I also added the test procedure inside, I believe this could work as well in your device, however, I'm still not so clear about what caused your setup failure, if you could share more details that would be even better. If there's any other problem, plz let me know :))

@xiaodouzi666 commented on GitHub (Oct 26, 2026): Hi, I believe this crash has been solved, in order to solve that, I implemented the fix under these two PRs (#4932, #4933), please take a look, and I also added the test procedure inside, I believe this could work as well in your device, however, I'm still not so clear about what caused your setup failure, if you could share more details that would be even better. If there's any other problem, plz let me know :))

bsakaguchi commented

2026-10-29 21:23:38 +00:00

Author

Owner

@sivabreddy commented on GitHub (Oct 26, 2026):

Thank you @xiaodouzi666 for #4932 and #4933. I tested both today.

Why setup() failed on our side (details in my comment of 2026-10-19 above): we had set MC_GID_INDEX=3 (and a cluster policy injected NCCL_IB_GID_INDEX=3) while several pods on each node share one RoCE device, so index 3 did not point to the pod's own GID, and we listed 8 RDMA devices in device_name where only one has a network. Client creation then failed on all 20 retries. With automatic GID selection and only the connected device, the client is created and we had no crash.

Test of #4932 + #4933. Two wheels built with the same flags (-DUSE_CUDA=OFF -DWITH_STORE=ON -DWITH_TE=ON, Python 3.10, Ubuntu 22.04, x86_64): main at ac4d29bf (merge base) and #4933 head 69c49020 (includes #4932). For each case and each API, a new Python process calls MooncakeDistributedStore.setup(...) and then one batch API with dummy buffers.

S1: master not reachable (tcp, master 127.0.0.1:1): the path of this issue ("Client is not initialized").
S2: master reachable, RDMA device that does not exist (rdma, mlx5_99).

API after the failed `setup()` (S1 and S2)	`main`	#4932 + #4933
`batch_get_buffer`	SIGSEGV	`[None, None]`
`get_into_ranges`	SIGSEGV	`[[[-600]]]`
`batch_put_from`	SIGSEGV	`[-600]`
`batch_get_into`	SIGSEGV	`[-600]`
`batch_get_into_multi_buffers`	SIGSEGV	`[-600]`
`batch_upsert_from`	`[-600]`	`[-600]`
`batch_put_from_multi_buffers`	`[]`	`[]`

So 5 of the 7 batch APIs in the Python binding crash on main in both cases, and none crash with the PRs. A correct setup (local master, tcp) gives setup() == 0 and a working put/get with both builds.

#4933 also works as described: the retry line now names the reason (Failed to create client on port 13897: RPC_FAIL, retry 1/20 in S1, ...: INTERNAL_ERROR, retry 1/20 in S2), and in S1 setup() returns -900 (RPC_FAIL) instead of -1. In our original logs only the retry line was visible, so this would have shown the cause at once.

Two small notes:

#4932 and #4633 change the same 8 observers in the same way; maybe they can be combined so that only one PR needs review.
With an uninitialized client, batch_put_from_multi_buffers returns an empty list, not one error per key (same on main), so a caller cannot see which keys failed. Not related to the crash.

I will close this issue when one of the fixes is merged, unless you prefer to close it earlier.

@sivabreddy commented on GitHub (Oct 26, 2026): Thank you @xiaodouzi666 for #4932 and #4933. I tested both today. **Why `setup()` failed on our side** (details in my comment of 2026-10-19 above): we had set `MC_GID_INDEX=3` (and a cluster policy injected `NCCL_IB_GID_INDEX=3`) while several pods on each node share one RoCE device, so index 3 did not point to the pod's own GID, and we listed 8 RDMA devices in `device_name` where only one has a network. Client creation then failed on all 20 retries. With automatic GID selection and only the connected device, the client is created and we had no crash. **Test of #4932 + #4933.** Two wheels built with the same flags (`-DUSE_CUDA=OFF -DWITH_STORE=ON -DWITH_TE=ON`, Python 3.10, Ubuntu 22.04, x86_64): `main` at `ac4d29bf` (merge base) and #4933 head `69c49020` (includes #4932). For each case and each API, a new Python process calls `MooncakeDistributedStore.setup(...)` and then one batch API with dummy buffers. - S1: master not reachable (`tcp`, master `127.0.0.1:1`): the path of this issue ("Client is not initialized"). - S2: master reachable, RDMA device that does not exist (`rdma`, `mlx5_99`). | API after the failed `setup()` (S1 and S2) | `main` | #4932 + #4933 | |---|---|---| | `batch_get_buffer` | SIGSEGV | `[None, None]` | | `get_into_ranges` | SIGSEGV | `[[[-600]]]` | | `batch_put_from` | SIGSEGV | `[-600]` | | `batch_get_into` | SIGSEGV | `[-600]` | | `batch_get_into_multi_buffers` | SIGSEGV | `[-600]` | | `batch_upsert_from` | `[-600]` | `[-600]` | | `batch_put_from_multi_buffers` | `[]` | `[]` | So 5 of the 7 batch APIs in the Python binding crash on `main` in both cases, and none crash with the PRs. A correct setup (local master, `tcp`) gives `setup() == 0` and a working `put`/`get` with both builds. #4933 also works as described: the retry line now names the reason (`Failed to create client on port 13897: RPC_FAIL, retry 1/20` in S1, `...: INTERNAL_ERROR, retry 1/20` in S2), and in S1 `setup()` returns `-900` (`RPC_FAIL`) instead of `-1`. In our original logs only the retry line was visible, so this would have shown the cause at once. Two small notes: - #4932 and #4633 change the same 8 observers in the same way; maybe they can be combined so that only one PR needs review. - With an uninitialized client, `batch_put_from_multi_buffers` returns an empty list, not one error per key (same on `main`), so a caller cannot see which keys failed. Not related to the crash. I will close this issue when one of the fixes is merged, unless you prefer to close it earlier.

bsakaguchi referenced this issue

2026-10-29 21:32:24 +00:00

[PR #1174] [CLOSED] [Store] refactor: config loading and CLI parsing by iflytek #2052