[GH-ISSUE #607] [Usage]: When using MooncakeDistributedStore in Python, after the written client is closed, other clients cannot query the written value #202

New issue

Closed

opened 2026-10-29 21:10:50 +00:00 by bsakaguchi · 12 comments

bsakaguchi commented 2026-10-29 21:10:50 +00:00

Owner

Copy link

Originally created by @hhwode on GitHub (Jul 9, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/607

Describe your usage question

when using at node 1：

# node 1
from mooncake.store import MooncakeDistributedStore
store = MooncakeDistributedStore()
store.setup("10.248.92.175", "http://10.248.253.230:8080/metadata", 3200 * 1024 * 1024, 512 * 1024 * 1024, "rdma", "mlx5_2", "10.248.92.175:54321")
key = "test_teardown_key"
test_data = b"Hello, World!"
store.put(key, test_data)
store.is_exist(key)  # ret=1, exist

using in node 2:

from mooncake.store import MooncakeDistributedStore
store = MooncakeDistributedStore()
store.setup("10.248.253.230", "http://10.248.253.230:8080/metadata", 3200 * 1024 * 1024, 512 * 1024 * 1024, "rdma", "mlx5_2", "10.248.92.175:54321")
key = "test_teardown_key"
store.is_exist(key)  # ret=1, exist when node 1 is runing
store.is_exist(key)  # ret=0, not exist when node 1 is closed

It is right for store.put, at document https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/mooncake-store-preview.md. The put operation will copy the replica to other client, but KV seems to only exist on the writing client, and it was deleted after being closed.

master service logs:

I0709 17:14:49.469707  4236 rpc_service.h:136] Master Metrics: Storage: 13.00 B / 3.12 GB (0.0%) | Keys: 1 | Requests (Success/Total): Put=2/2, Get=0/0, Exist=1/2, Del=0/0, DelAll=0/0,  | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B
I0709 17:14:59.469836  4236 rpc_service.h:136] Master Metrics: Storage: 13.00 B / 6.25 GB (0.0%) | Keys: 1 | Requests (Success/Total): Put=2/2, Get=0/0, Exist=1/2, Del=0/0, DelAll=0/0,  | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B
I0709 17:15:09.469965  4236 rpc_service.h:136] Master Metrics: Storage: 13.00 B / 6.25 GB (0.0%) | Keys: 1 | Requests (Success/Total): Put=2/2, Get=0/0, Exist=2/3, Del=0/0, DelAll=0/0,  | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B
I0709 17:15:19.470116  4236 rpc_service.h:136] Master Metrics: Storage: 0.00 B / 3.12 GB (0.0%) | Keys: 0 | Requests (Success/Total): Put=2/2, Get=0/0, Exist=2/3, Del=0/0, DelAll=0/0,  | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @hhwode on GitHub (Jul 9, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/607 ### Describe your usage question when using at node 1： ```python # node 1 from mooncake.store import MooncakeDistributedStore store = MooncakeDistributedStore() store.setup("10.248.92.175", "http://10.248.253.230:8080/metadata", 3200 * 1024 * 1024, 512 * 1024 * 1024, "rdma", "mlx5_2", "10.248.92.175:54321") key = "test_teardown_key" test_data = b"Hello, World!" store.put(key, test_data) store.is_exist(key) # ret=1, exist ``` using in node 2: ```python from mooncake.store import MooncakeDistributedStore store = MooncakeDistributedStore() store.setup("10.248.253.230", "http://10.248.253.230:8080/metadata", 3200 * 1024 * 1024, 512 * 1024 * 1024, "rdma", "mlx5_2", "10.248.92.175:54321") key = "test_teardown_key" store.is_exist(key) # ret=1, exist when node 1 is runing store.is_exist(key) # ret=0, not exist when node 1 is closed ``` It is right for store.put, at document [https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/mooncake-store-preview.md](url). The put operation will copy the replica to other client, but KV seems to only exist on the writing client, and it was deleted after being closed. master service logs: ``` I0709 17:14:49.469707 4236 rpc_service.h:136] Master Metrics: Storage: 13.00 B / 3.12 GB (0.0%) | Keys: 1 | Requests (Success/Total): Put=2/2, Get=0/0, Exist=1/2, Del=0/0, DelAll=0/0, | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B I0709 17:14:59.469836 4236 rpc_service.h:136] Master Metrics: Storage: 13.00 B / 6.25 GB (0.0%) | Keys: 1 | Requests (Success/Total): Put=2/2, Get=0/0, Exist=1/2, Del=0/0, DelAll=0/0, | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B I0709 17:15:09.469965 4236 rpc_service.h:136] Master Metrics: Storage: 13.00 B / 6.25 GB (0.0%) | Keys: 1 | Requests (Success/Total): Put=2/2, Get=0/0, Exist=2/3, Del=0/0, DelAll=0/0, | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B I0709 17:15:19.470116 4236 rpc_service.h:136] Master Metrics: Storage: 0.00 B / 3.12 GB (0.0%) | Keys: 0 | Requests (Success/Total): Put=2/2, Get=0/0, Exist=2/3, Del=0/0, DelAll=0/0, | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B ``` ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi 2026-10-29 21:10:50 +00:00

• closed this issue
• added the
  stale
  auto-closed
  labels

bsakaguchi commented 2026-10-29 21:10:51 +00:00

Author

Owner

Copy link

@xiaguan commented on GitHub (Jul 9, 2025):

Right now in the Python bindings, we default the replica number to 1 and try to allocate memory form itself first.
I think this may clear up the confusion you had.

I'll submit a PR soon to add support for ReplicaConfig in the Python bindings.

<!-- gh-comment-id:3051913595 --> @xiaguan commented on GitHub (Jul 9, 2025): Right now in the Python bindings, we default the replica number to 1 and try to allocate memory form itself first. I think this may clear up the confusion you had. I'll submit a PR soon to add support for ReplicaConfig in the Python bindings.

bsakaguchi commented 2026-10-29 21:10:51 +00:00

Author

Owner

Copy link

@hhwode commented on GitHub (Jul 9, 2025):

Right now in the Python bindings, we default the replica number to 1 and try to allocate memory form itself first. I think this may clear up the confusion you had.

I'll submit a PR soon to add support for ReplicaConfig in the Python bindings.

Is there any other way to configure the number of replicas on the Python side，and the document states that it is possible to offload kv to disk through MOONCAKE-STORAGE-ROOT_DIR. Does Python not support this.

<!-- gh-comment-id:3051939326 --> @hhwode commented on GitHub (Jul 9, 2025): > Right now in the Python bindings, we default the replica number to 1 and try to allocate memory form itself first. I think this may clear up the confusion you had. > > I'll submit a PR soon to add support for ReplicaConfig in the Python bindings. Is there any other way to configure the number of replicas on the Python side，and the document states that it is possible to offload kv to disk through MOONCAKE-STORAGE-ROOT_DIR. Does Python not support this.

bsakaguchi commented 2026-10-29 21:10:51 +00:00

Author

Owner

Copy link

@xiaguan commented on GitHub (Jul 9, 2025):

https://github.com/kvcache-ai/Mooncake/pull/608, Python binding's replica config PR has been submitted—just waiting on review and merge.

As for offloading to disk: yes, the Python client does support this. You just need to set the appropriate environment variable. That said, this feature is still experimental for now, and it's mainly intended to serve as a read cache for the client itself. @SgtPepperr , could u help answer this?

<!-- gh-comment-id:3052048082 --> @xiaguan commented on GitHub (Jul 9, 2025): https://github.com/kvcache-ai/Mooncake/pull/608, Python binding's replica config PR has been submitted—just waiting on review and merge. As for offloading to disk: yes, the Python client does support this. You just need to set the appropriate environment variable. That said, this feature is still experimental for now, and it's mainly intended to serve as a read cache for the client itself. @SgtPepperr , could u help answer this?

bsakaguchi commented 2026-10-29 21:10:51 +00:00

Author

Owner

Copy link

@xiaguan commented on GitHub (Jul 10, 2025):

The PR has been merged, but we haven't published a new release yet. If you need it right away, you can build from main. Python now supports setting ReplicateConfig directly when calling put() releated interface.

<!-- gh-comment-id:3055883739 --> @xiaguan commented on GitHub (Jul 10, 2025): The PR has been merged, but we haven't published a new release yet. If you need it right away, you can build from main. Python now supports setting `ReplicateConfig` directly when calling put() releated interface.

bsakaguchi commented 2026-10-29 21:10:51 +00:00

Author

Owner

Copy link

@LuyuZhang00 commented on GitHub (Jul 10, 2025):

test-script：mooncake-store/tests/stress_cluster_benchmark.py
During concurrent execution of the test-script as role=prefill on node1 and role=decode on node2, the error 'Transfer failed for batch xxxxx task 0 with status 6' occurred, which is confusing. @xiaguan

<!-- gh-comment-id:3056655225 --> @LuyuZhang00 commented on GitHub (Jul 10, 2025): test-script：mooncake-store/tests/stress_cluster_benchmark.py During concurrent execution of the test-script as role=prefill on node1 and role=decode on node2, the error 'Transfer failed for batch xxxxx task 0 with status 6' occurred, which is confusing. @xiaguan

bsakaguchi commented 2026-10-29 21:10:51 +00:00

Author

Owner

Copy link

@xiaguan commented on GitHub (Jul 10, 2025):

Could you share more detailed logs? And please checkout your config.

Status 6 means the transfer failed.

<!-- gh-comment-id:3056669352 --> @xiaguan commented on GitHub (Jul 10, 2025): Could you share more detailed logs? And please checkout your config. Status 6 means the transfer failed.

bsakaguchi commented 2026-10-29 21:10:51 +00:00

Author

Owner

Copy link

@LuyuZhang00 commented on GitHub (Jul 10, 2025):

example
E0710 09:48:23.776901 5615 store_py.cpp:1202] BatchGet failed for key 'key1199': TRANSFER_FAIL

parser.add_argument("--local-hostname", type=str, default="localhost", help="Local hostname")
in node1 :local-hostname is node 1 name,
in node2: I set node2, which localhost should set

<!-- gh-comment-id:3056696776 --> @LuyuZhang00 commented on GitHub (Jul 10, 2025): example `E0710 09:48:23.776901 5615 store_py.cpp:1202] BatchGet failed for key 'key1199': TRANSFER_FAIL ` parser.add_argument("--local-hostname", type=str, default="localhost", help="Local hostname") in node1 :local-hostname is node 1 name, in node2: I set node2, which localhost should set

bsakaguchi commented 2026-10-29 21:10:52 +00:00

Author

Owner

Copy link

@xiaguan commented on GitHub (Jul 10, 2025):

Is it possible that the prefill node exited during the data transfer? Since we're using the client's own memory for storage, that could explain it.

<!-- gh-comment-id:3056908818 --> @xiaguan commented on GitHub (Jul 10, 2025): Is it possible that the prefill node exited during the data transfer? Since we're using the client's own memory for storage, that could explain it.

bsakaguchi commented 2026-10-29 21:10:52 +00:00

Author

Owner

Copy link

@hhwode commented on GitHub (Jul 11, 2025):

#608, Python binding's replica config PR has been submitted—just waiting on review and merge.

As for offloading to disk: yes, the Python client does support this. You just need to set the appropriate environment variable. That said, this feature is still experimental for now, and it's mainly intended to serve as a read cache for the client itself. @SgtPepperr , could u help answer this?

The PR has been merged, but we haven't published a new release yet. If you need it right away, you can build from main. Python now supports setting ReplicateConfig directly when calling put() releated interface.

Thanks， i will try it.

<!-- gh-comment-id:3060318279 --> @hhwode commented on GitHub (Jul 11, 2025): > [#608](https://github.com/kvcache-ai/Mooncake/pull/608), Python binding's replica config PR has been submitted—just waiting on review and merge. > > As for offloading to disk: yes, the Python client does support this. You just need to set the appropriate environment variable. That said, this feature is still experimental for now, and it's mainly intended to serve as a read cache for the client itself. [@SgtPepperr](https://github.com/SgtPepperr) , could u help answer this? > The PR has been merged, but we haven't published a new release yet. If you need it right away, you can build from main. Python now supports setting `ReplicateConfig` directly when calling put() releated interface. Thanks， i will try it.

bsakaguchi commented 2026-10-29 21:10:52 +00:00

Author

Owner

Copy link

@stmatengss commented on GitHub (Nov 25, 2025):

When any client is shut down, we should provide an option to migrate data.

<!-- gh-comment-id:3573856322 --> @stmatengss commented on GitHub (Nov 25, 2025): When any client is shut down, we should provide an option to migrate data.

bsakaguchi commented 2026-10-29 21:10:52 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jun 30, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4839876424 --> @github-actions[bot] commented on GitHub (Jun 30, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented 2026-10-29 21:10:52 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 7, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:4900236058 --> @github-actions[bot] commented on GitHub (Jul 7, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue 2026-10-29 21:27:02 +00:00

[PR #202] [MERGED] [Feature] use randomly-selected port in TCP-based OOB communication #1392

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#202

No description provided.