kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2514] [Usage]: Inconsistent cluster_id default between master and store client #666

New issue

Closed

opened 2026-10-29 21:17:49 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:17:49 +00:00

Owner

Originally created by @mzygQAQ on GitHub (Jun 17, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2514

Originally assigned to: @mzygQAQ on GitHub.

Describe your usage question

When starting mooncake_master without --cluster_id and without MC_STORE_CLUSTER_ID env var, the master registers in etcd with key:

mooncake-store/mooncake_cluster/master_view

However, when mooncake_store_service.py (the Python store service with REST API) starts in HA mode, it fails with:

  E0617 15:21:50.715953 486225 etcd_helper.cpp:54] key=mooncake-store/mooncake/master_view, error=key not found in etcd
  E0617 15:21:50.715960 486225 client_service.cpp:294] No master is available in HA backend

The expected key uses "mooncake_cluster" as the cluster_id (matching DEFAULT_CLUSTER_ID = "mooncake_cluster" in types.h), but the client is looking up "mooncake" instead.

Checking the installed Python extension by pip3:

  $ strings store.so | grep -E "^mooncake$|^mooncake_cluster$|^mooncake-store/" | sort -u
  mooncake
  mooncake-store/

Note: The static library built from current HEAD:

  $ strings libmooncake_store.a | grep -E "^mooncake$|^mooncake_cluster$|^mooncake-store/" | sort -u
  mooncake
  mooncake-store/
  mooncake_cluster

The current source code has DEFAULT_CLUSTER_ID = "mooncake_cluster" (types.h:106), and the static library built from the same source correctly contains "mooncake_cluster". But the installed wheel's store.so does NOT contain
"mooncake_cluster" - only "mooncake" and "mooncake-store/".

This suggests the installed Python wheel was built from an older version where DEFAULT_CLUSTER_ID was just "mooncake" (without "_cluster"). Could you confirm what the correct DEFAULT_CLUSTER_ID should be and whether a new wheel
should be published?

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @mzygQAQ on GitHub (Jun 17, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2514 Originally assigned to: @mzygQAQ on GitHub. ### Describe your usage question When starting mooncake_master without --cluster_id and without MC_STORE_CLUSTER_ID env var, the master registers in etcd with key: ``` mooncake-store/mooncake_cluster/master_view ``` However, when mooncake_store_service.py (the Python store service with REST API) starts in HA mode, it fails with: ``` E0617 15:21:50.715953 486225 etcd_helper.cpp:54] key=mooncake-store/mooncake/master_view, error=key not found in etcd E0617 15:21:50.715960 486225 client_service.cpp:294] No master is available in HA backend ``` The expected key uses "mooncake_cluster" as the cluster_id (matching DEFAULT_CLUSTER_ID = "mooncake_cluster" in types.h), but the client is looking up "mooncake" instead. Checking the installed Python extension by pip3: ``` $ strings store.so | grep -E "^mooncake$|^mooncake_cluster$|^mooncake-store/" | sort -u mooncake mooncake-store/ ``` Note: The static library built from current HEAD: ``` $ strings libmooncake_store.a | grep -E "^mooncake$|^mooncake_cluster$|^mooncake-store/" | sort -u mooncake mooncake-store/ mooncake_cluster ``` The current source code has DEFAULT_CLUSTER_ID = "mooncake_cluster" (types.h:106), and the static library built from the same source correctly contains "mooncake_cluster". But the installed wheel's store.so does NOT contain "mooncake_cluster" - only "mooncake" and "mooncake-store/". This suggests the installed Python wheel was built from an older version where DEFAULT_CLUSTER_ID was just "mooncake" (without "_cluster"). Could you confirm what the correct DEFAULT_CLUSTER_ID should be and whether a new wheel should be published? ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:17:49 +00:00

bsakaguchi commented

2026-10-29 21:17:50 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 17, 2026):

Thanks for opening this issue, @mzygQAQ!

Field	Value
Issue	#2514
GitHub user ID	`30590325`
Reporter	@mzygQAQ

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jun 17, 2026): Thanks for opening this issue, @mzygQAQ! | Field | Value | |-------|-------| | **Issue** | #2514 | | **GitHub user ID** | `30590325` | | **Reporter** | @mzygQAQ | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:17:50 +00:00

Author

Owner

@mzygQAQ commented on GitHub (Jun 18, 2026):

$ pip list | grep moon
mooncake-transfer-engine-non-cuda 0.3.11.post1

@mzygQAQ commented on GitHub (Jun 18, 2026): ``` $ pip list | grep moon mooncake-transfer-engine-non-cuda 0.3.11.post1 ```

bsakaguchi commented

2026-10-29 21:17:50 +00:00

Author

Owner

@Icedcoco commented on GitHub (Jun 18, 2026):

Thanks for the detailed check. Your analysis looks right.

The intended default cluster_id is mooncake_cluster, matching DEFAULT_CLUSTER_ID in types.h. The mismatch you observed is caused by the released wheel mooncake-transfer-engine-non-cuda 0.3.11.post1 still using the older client-side fallback value mooncake.

This has already been fixed on main by #2244, which changed the HA client fallback to use DEFAULT_CLUSTER_ID instead of the hardcoded mooncake. A new wheel should be published to include this fix.

Before the new wheel is available, the workaround is to set the cluster id explicitly on both master and client side, for example:

export MC_STORE_CLUSTER_ID=mooncake_cluster

or pass the equivalent --cluster_id=mooncake_cluster when starting mooncake_master.

With both sides using the same cluster_id, the client should look up mooncake-store/mooncake_cluster/master_view and discover the HA master correctly.

@Icedcoco commented on GitHub (Jun 18, 2026): Thanks for the detailed check. Your analysis looks right. The intended default `cluster_id` is `mooncake_cluster`, matching `DEFAULT_CLUSTER_ID` in `types.h`. The mismatch you observed is caused by the released wheel `mooncake-transfer-engine-non-cuda 0.3.11.post1` still using the older client-side fallback value `mooncake`. This has already been fixed on `main` by #2244, which changed the HA client fallback to use `DEFAULT_CLUSTER_ID` instead of the hardcoded `mooncake`. A new wheel should be published to include this fix. Before the new wheel is available, the workaround is to set the cluster id explicitly on both master and client side, for example: ```bash export MC_STORE_CLUSTER_ID=mooncake_cluster ``` or pass the equivalent `--cluster_id=mooncake_cluster` when starting `mooncake_master`. With both sides using the same `cluster_id`, the client should look up `mooncake-store/mooncake_cluster/master_view` and discover the HA master correctly.

bsakaguchi commented

2026-10-29 21:17:50 +00:00

Author

Owner

@mzygQAQ commented on GitHub (Jun 18, 2026):

Thanks for the detailed check. Your analysis looks right.

The intended default cluster_id is mooncake_cluster, matching DEFAULT_CLUSTER_ID in types.h. The mismatch you observed is caused by the released wheel mooncake-transfer-engine-non-cuda 0.3.11.post1 still using the older client-side fallback value mooncake.

This has already been fixed on main by #2244, which changed the HA client fallback to use DEFAULT_CLUSTER_ID instead of the hardcoded mooncake. A new wheel should be published to include this fix.

Before the new wheel is available, the workaround is to set the cluster id explicitly on both master and client side, for example:

export MC_STORE_CLUSTER_ID=mooncake_cluster
or pass the equivalent --cluster_id=mooncake_cluster when starting mooncake_master.

With both sides using the same cluster_id, the client should look up mooncake-store/mooncake_cluster/master_view and discover the HA master correctly.

thanks !

@mzygQAQ commented on GitHub (Jun 18, 2026): > Thanks for the detailed check. Your analysis looks right. > > The intended default `cluster_id` is `mooncake_cluster`, matching `DEFAULT_CLUSTER_ID` in `types.h`. The mismatch you observed is caused by the released wheel `mooncake-transfer-engine-non-cuda 0.3.11.post1` still using the older client-side fallback value `mooncake`. > > This has already been fixed on `main` by [#2244](https://github.com/kvcache-ai/Mooncake/pull/2244), which changed the HA client fallback to use `DEFAULT_CLUSTER_ID` instead of the hardcoded `mooncake`. A new wheel should be published to include this fix. > > Before the new wheel is available, the workaround is to set the cluster id explicitly on both master and client side, for example: > > export MC_STORE_CLUSTER_ID=mooncake_cluster > or pass the equivalent `--cluster_id=mooncake_cluster` when starting `mooncake_master`. > > With both sides using the same `cluster_id`, the client should look up `mooncake-store/mooncake_cluster/master_view` and discover the HA master correctly. thanks !

bsakaguchi referenced this issue

2026-10-29 21:29:53 +00:00

[PR #666] [MERGED] [TransferEngine] Add known issue about accuracy problem #1732