kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2633] [RFC]: Host-aware Local-first Allocation for Mooncake Store #693

New issue

Closed

opened 2026-10-29 21:18:15 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:18:15 +00:00

Owner

Originally created by @yokinoshitayoki on GitHub (Jun 26, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2633

Changes proposed

Motivation

Mooncake Store currently supports preferred allocation through ReplicateConfig.preferred_segment and ReplicateConfig.preferred_segments. When no preferred segment is specified, memory replicas are allocated by the configured allocation strategy, which commonly falls back to a random segment.

For KV cache workloads, random placement also increases the failure blast radius of a store node. In the current Mooncake Store cluster, when a store node goes offline, either because of a crash or planned shutdown, all KV cache stored on that node is lost. KV cache reuse depends on a continuous prefix. If a block in the middle of a request's cached sequence is missing, the later cached blocks can no longer be reused even if they still exist on other store nodes. Those suffix blocks remain stored in the cluster but lose their practical value for prefix reuse.

With random placement, KV cache blocks produced by one writer may be spread across many store nodes. A single store node failure can therefore punch holes into cached sequences from many different writers and invalidate the suffixes behind those holes. Host-aware local-first placement reduces this blast radius in the common case by keeping a writer's cache on local store segments first and only using remote hosts when local capacity is insufficient.

Locality is also a motivation. The writer is usually an inference engine process running on a specific physical host. If the KV cache is stored on a remote host while local CPU memory is available, later transfers between the inference engine and Mooncake Store may cross the network unnecessarily. This increases latency and consumes network bandwidth.

The existing preferred segment interface can be used by an upper layer such as vLLM or SGLang, but it requires the upper layer or deployment system to know the exact Segment.name of the local store. This is fragile for a general-purpose Mooncake feature:

Segment.name is a logical string and has no required host/IP format.
A physical host may run multiple Mooncake clients and multiple store segments.
A requester client may not mount any segment in standalone-store mode.
Upper layers should not need to manually enumerate Mooncake store segments to get a reasonable local placement policy.

This RFC proposes a host-aware local-first allocation mode inside Mooncake Store. Mooncake master will maintain host-level topology metadata and use it to prefer local store segments for Put allocation. If local segments cannot satisfy the allocation, master will fall back to other hosts in a deterministic ordered sequence rather than choosing a random segment immediately.

Goals

Add a Mooncake-native notion of physical host identity, host_id.
Let master map writer clients to hosts and hosts to active store segments.
Reduce the failure blast radius of one store node by avoiding unnecessary cross-host scattering of one writer's KV cache.
Prefer segments on the writer's host when no explicit preferred segment is provided.

Fall back to remote hosts in a deterministic host-ordered sequence, for example:

writer on host0: host0, host1, host2, host3
writer on host1: host1, host2, host3, host0
writer on host2: host2, host3, host0, host1
writer on host3: host3, host0, host1, host2

Within the same host, rotate the starting segment by key hash so multiple local segments are used more evenly.
Preserve the current explicit preferred_segment(s) behavior.
Keep the first implementation at host granularity. NUMA/GPU-aware placement can be built later on top of the same metadata model.

Non-goals

This RFC does not propose NUMA-aware or GPU-aware placement.
This RFC does not change KV cache consistency or durability semantics.
This RFC does not guarantee load-optimal global balancing.
This RFC does not require vLLM or SGLang to pass store segment names.
This RFC does not change the meaning of Segment.name.
This RFC does not make random fallback unavailable for deployments that want the current behavior.

Current Behavior

Mooncake master currently tracks segment ownership by client and by segment name. The relevant relationships are:

client_id -> segment_ids
segment.name -> client_id
segment.name -> segment_id
segment.name -> allocator

Segment.name is the logical name used for preferred allocation. A normal memory segment uses the local client's local_hostname as Segment.name.

Segment.te_endpoint is used for transport addressing. It may contain an IP:port endpoint, but it is not a stable physical host identity. It can be affected by multi-NIC selection, container networking, hostname/IP mismatch, or protocol-specific behavior.

When ReplicateConfig.preferred_segment(s) is set, allocation tries those segments first and then falls back to the existing strategy. When no preferred segment is set, master does not currently infer which segments are local to the writer.

Proposed Change

1. Introduce `host_id`

Add a new logical field named host_id.

host_id:
  Stable physical host identity used for allocation affinity.

host_id is separate from local_hostname.

local_hostname:
  Existing Mooncake client logical name. It may become Segment.name.

host_id:
  Physical machine identity. All Mooncake clients on the same physical host should report the same host_id.

Example:

Physical host: hostA

vLLM rank 0:
  local_hostname = hostA-rank0
  host_id = hostA

vLLM rank 1:
  local_hostname = hostA-rank1
  host_id = hostA

Mooncake store process:
  local_hostname = hostA-store0
  host_id = hostA
  segment.name = hostA-store0

2. Resolve `host_id` in the Mooncake client

Mooncake client should resolve host_id without requiring inference engines to understand Mooncake segment placement.

Recommended priority:

Explicit environment variable:
```
export MOONCAKE_HOST_ID=hostA
```
Optional explicit field in Mooncake config or setup options.
Best-effort fallback generated by Mooncake, for example hostname or the existing local host identity.

The best-effort fallback should be documented as a convenience mode, not a strict correctness guarantee. In containerized or multi-NIC environments, operators should set MOONCAKE_HOST_ID explicitly.

3. Register client host metadata on master

Master needs to know the writer's host even when the writer does not mount a segment. Therefore storing host_id only on Segment is not enough.

Master should maintain:

client_host_id_:        client_id -> host_id
segment_host_id_:       segment_name -> host_id
segments_by_host_:      host_id -> ordered segment names

The client should register or refresh its host_id when it connects to master or during the existing client/master setup path. Store clients should also attach host_id to mounted segments, so the master can rebuild host-to-segment indexes when segments are mounted, remounted, unmounted, or restored from snapshot.

4. Build host-ordered preferred segments on Put

When master handles PutStart, the priority should be:

If config.preferred_segment is set, use it exactly as today.
Else if config.preferred_segments is set, use it exactly as today.
Else if host-aware local-first allocation is enabled, build a preferred segment list from client_id -> host_id -> segment_names.
Else use the existing allocation behavior.

Pseudo-code:

std::vector<std::string> preferred_segments;

if (!config.preferred_segment.empty()) {
    preferred_segments.push_back(config.preferred_segment);
} else if (!config.preferred_segments.empty()) {
    preferred_segments = config.preferred_segments;
} else if (local_first_enabled) {
    preferred_segments = BuildHostOrderedSegments(client_id, key);
}

allocation_strategy_->Allocate(
    allocator_manager,
    value_length,
    config.replica_num,
    preferred_segments);

5. Host fallback ordering

The host fallback order should be deterministic and stable.

For the first version, sort active host_ids lexicographically and rotate the list so the writer's host is first.

Example:

active hosts: host0, host1, host2, host3

Writer host orders:

host0 -> host0, host1, host2, host3
host1 -> host1, host2, host3, host0
host2 -> host2, host3, host0, host1
host3 -> host3, host0, host1, host2

This avoids the common random fallback problem where many writers may randomly fallback to the same remote segment at the same time.

6. Stable segment rotation within a host

If a host has multiple segments, always trying the first segment creates a local hotspot. Instead, the segment order inside each host should use a stable key-based rotation.

Example:

host0 segments sorted by name:
  host0-store0, host0-store1, host0-store2

start = Hash(key) % 3

if start = 1:
  host0-store1, host0-store2, host0-store0

The same rotation is applied independently for each host in the fallback chain.

The resulting segment list for writer on host1 may look like:

host1-store1, host1-store0,
host2-store0, host2-store1,
host3-store1, host3-store0,
host0-store0, host0-store1

This remains deterministic for the same key and topology, while spreading first-choice pressure across segments on the same host.

7. Fallback behavior

The allocation strategy already supports trying preferred segments first and then continuing with fallback allocation if the preferred list cannot satisfy the request.

With host-aware local-first allocation enabled, the preferred list can include all active segments in host fallback order. For replica_num = 1, this means:

try local host segments
try next host segments
try next host segments
...
return failure only if no segment can allocate

For the first implementation, this RFC targets the common KV cache case where replica_num = 1. For replica_num > 1, the behavior should be explicitly defined before enabling this policy by default. A future extension may place the first replica locally and spread additional replicas across different hosts.

Configuration

Add a client-side host identity variable:

MOONCAKE_HOST_ID=hostA

Add a master/client allocation policy switch. The exact naming can be discussed, but the intended semantics are:

MOONCAKE_LOCAL_FIRST_ALLOC=1
MOONCAKE_HOST_FALLBACK_POLICY=ordered

Initial default:

MOONCAKE_LOCAL_FIRST_ALLOC=0

The default can stay disabled for conservative backward compatibility. After the behavior is validated, the community can discuss whether local-first should become the default for Mooncake Store memory allocation.

Compatibility

This proposal is backward-compatible if introduced behind a feature flag.

Existing preferred_segment and preferred_segments keep their current semantics and override automatic local-first allocation.
Existing clients that do not provide host_id continue to use the current allocation behavior.
Existing Segment.name matching behavior is unchanged.
Existing deployments can keep using manual preferred segment configuration.
Snapshot and HA code must include the new host metadata or rebuild it from segment/client registration after recovery.

Failure and Edge Cases

Writer has no `host_id`

Fallback to current allocation behavior.

Writer host has no active store segment

Skip the writer host and continue with the next hosts in ordered fallback.

Store segment is full

Continue to the next segment in the generated preferred list.

Store host goes down

The host's unavailable segments should be removed from active allocator indexes. The generated preferred list should only include active segments.

Multiple clients on the same physical host

All clients must report the same host_id. This is the main reason to keep host_id separate from local_hostname.

Multi-NIC and container environments

Operators should set MOONCAKE_HOST_ID explicitly. Automatically derived hostnames or IPs are best-effort only.

Implementation Plan

This feature is intended to be implemented in one PR with a narrow scope: add host metadata, build automatic local-first preferred segment lists, and verify the allocation behavior. The PR should not include NUMA/GPU-aware placement or topology-aware remote ordering.

Metadata changes

Add host_id to Mooncake client configuration.
Resolve host_id from MOONCAKE_HOST_ID, explicit config, or best-effort fallback.
Extend the client/master registration or heartbeat path to record client_id -> host_id.
Add host_id to mounted segment metadata.

Maintain master indexes:

client_host_id_
segment_host_id_
segments_by_host_

Allocation policy

Add a helper to build host-ordered segment lists:

std::vector<std::string> BuildHostOrderedSegments(
    const UUID& writer_client_id,
    const std::string& key);

Sort active hosts deterministically.
Rotate host order so writer host is first.
Sort segments within each host deterministically.
Rotate segments within each host using Hash(key) % segment_count.
Use the generated list as preferred_segments only when explicit preferred segments are absent.

Observability

Add minimal logs or metrics for:

local-first allocation enabled/disabled
writer host_id missing
generated preferred segment count
allocation landed on local host vs remote host
fallback depth by host

This is important for validating whether the new policy actually reduces cross-host writes and whether ordered fallback creates pressure on specific hosts.

Tests

Add unit tests for:

host_id resolution priority.
master host indexes on mount, unmount, and remount.

BuildHostOrderedSegments with four hosts:

host0 -> host0, host1, host2, host3
host1 -> host1, host2, host3, host0
host2 -> host2, host3, host0, host1
host3 -> host3, host0, host1, host2

stable key-based segment rotation inside one host.
explicit preferred_segment(s) overriding automatic local-first allocation.
missing host_id falling back to current behavior.
local host with no active segment falling back to the next host.

Add integration tests for:

standalone-store mode where requester mounts no segment but still registers host_id.
multiple store segments on one host.
local segment full, remote ordered fallback succeeds.
segment unmount removes it from host-aware allocation candidates.

Alternatives Considered

Upper layer passes `preferred_segment`

vLLM and SGLang can manually set preferred_segment or preferred_segments. This works for controlled deployments, but it requires the upper layer or deployment system to know exact Mooncake segment names. It also does not solve automatic discovery of all local segments.

Infer host by `Segment.te_endpoint`

Master could parse IPs from te_endpoint and group segments by IP. This is fragile in multi-NIC, container, hostname/IP-mixed, and standalone requester scenarios. It also does not provide host identity for clients that do not mount segments.

Reuse `local_hostname` as host identity

This breaks when multiple clients on the same physical host use different logical names, for example hostA-rank0, hostA-rank1, and hostA-store0. host_id should be a separate field.

Keep random fallback

Random fallback is simple, but multiple writers can fallback to the same remote segment or host at the same time. Ordered host fallback provides more predictable behavior and avoids obvious random collision patterns.

Open for Discussion

Should MOONCAKE_LOCAL_FIRST_ALLOC be enabled by default after the feature is implemented, or remain opt-in?
Should the fallback host order always be lexicographic rotation, or should operators be able to configure a topology-aware order?
How should replica_num > 1 behave under local-first allocation?
Should host_id be stored in snapshots, rebuilt from client registration, or both?
What hash function should be used for stable segment rotation inside a host?
Should this feature reuse prefer_alloc_in_same_node, or introduce a more explicit policy name such as local_first_allocation?

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @yokinoshitayoki on GitHub (Jun 26, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2633 ### Changes proposed ## Motivation Mooncake Store currently supports preferred allocation through `ReplicateConfig.preferred_segment` and `ReplicateConfig.preferred_segments`. When no preferred segment is specified, memory replicas are allocated by the configured allocation strategy, which commonly falls back to a random segment. For KV cache workloads, random placement also increases the failure blast radius of a store node. In the current Mooncake Store cluster, when a store node goes offline, either because of a crash or planned shutdown, all KV cache stored on that node is lost. KV cache reuse depends on a continuous prefix. If a block in the middle of a request's cached sequence is missing, the later cached blocks can no longer be reused even if they still exist on other store nodes. Those suffix blocks remain stored in the cluster but lose their practical value for prefix reuse. With random placement, KV cache blocks produced by one writer may be spread across many store nodes. A single store node failure can therefore punch holes into cached sequences from many different writers and invalidate the suffixes behind those holes. Host-aware local-first placement reduces this blast radius in the common case by keeping a writer's cache on local store segments first and only using remote hosts when local capacity is insufficient. Locality is also a motivation. The writer is usually an inference engine process running on a specific physical host. If the KV cache is stored on a remote host while local CPU memory is available, later transfers between the inference engine and Mooncake Store may cross the network unnecessarily. This increases latency and consumes network bandwidth. The existing preferred segment interface can be used by an upper layer such as vLLM or SGLang, but it requires the upper layer or deployment system to know the exact `Segment.name` of the local store. This is fragile for a general-purpose Mooncake feature: 1. `Segment.name` is a logical string and has no required host/IP format. 2. A physical host may run multiple Mooncake clients and multiple store segments. 3. A requester client may not mount any segment in standalone-store mode. 4. Upper layers should not need to manually enumerate Mooncake store segments to get a reasonable local placement policy. This RFC proposes a host-aware local-first allocation mode inside Mooncake Store. Mooncake master will maintain host-level topology metadata and use it to prefer local store segments for Put allocation. If local segments cannot satisfy the allocation, master will fall back to other hosts in a deterministic ordered sequence rather than choosing a random segment immediately. ## Goals 1. Add a Mooncake-native notion of physical host identity, `host_id`. 2. Let master map writer clients to hosts and hosts to active store segments. 3. Reduce the failure blast radius of one store node by avoiding unnecessary cross-host scattering of one writer's KV cache. 4. Prefer segments on the writer's host when no explicit preferred segment is provided. 5. Fall back to remote hosts in a deterministic host-ordered sequence, for example: ```text writer on host0: host0, host1, host2, host3 writer on host1: host1, host2, host3, host0 writer on host2: host2, host3, host0, host1 writer on host3: host3, host0, host1, host2 ``` 6. Within the same host, rotate the starting segment by key hash so multiple local segments are used more evenly. 7. Preserve the current explicit `preferred_segment(s)` behavior. 8. Keep the first implementation at host granularity. NUMA/GPU-aware placement can be built later on top of the same metadata model. ## Non-goals 1. This RFC does not propose NUMA-aware or GPU-aware placement. 2. This RFC does not change KV cache consistency or durability semantics. 3. This RFC does not guarantee load-optimal global balancing. 4. This RFC does not require vLLM or SGLang to pass store segment names. 5. This RFC does not change the meaning of `Segment.name`. 6. This RFC does not make random fallback unavailable for deployments that want the current behavior. ## Current Behavior Mooncake master currently tracks segment ownership by client and by segment name. The relevant relationships are: ```text client_id -> segment_ids segment.name -> client_id segment.name -> segment_id segment.name -> allocator ``` `Segment.name` is the logical name used for preferred allocation. A normal memory segment uses the local client's `local_hostname` as `Segment.name`. `Segment.te_endpoint` is used for transport addressing. It may contain an IP:port endpoint, but it is not a stable physical host identity. It can be affected by multi-NIC selection, container networking, hostname/IP mismatch, or protocol-specific behavior. When `ReplicateConfig.preferred_segment(s)` is set, allocation tries those segments first and then falls back to the existing strategy. When no preferred segment is set, master does not currently infer which segments are local to the writer. ## Proposed Change ### 1. Introduce `host_id` Add a new logical field named `host_id`. ```text host_id: Stable physical host identity used for allocation affinity. ``` `host_id` is separate from `local_hostname`. ```text local_hostname: Existing Mooncake client logical name. It may become Segment.name. host_id: Physical machine identity. All Mooncake clients on the same physical host should report the same host_id. ``` Example: ```text Physical host: hostA vLLM rank 0: local_hostname = hostA-rank0 host_id = hostA vLLM rank 1: local_hostname = hostA-rank1 host_id = hostA Mooncake store process: local_hostname = hostA-store0 host_id = hostA segment.name = hostA-store0 ``` ### 2. Resolve `host_id` in the Mooncake client Mooncake client should resolve `host_id` without requiring inference engines to understand Mooncake segment placement. Recommended priority: 1. Explicit environment variable: ```bash export MOONCAKE_HOST_ID=hostA ``` 2. Optional explicit field in Mooncake config or setup options. 3. Best-effort fallback generated by Mooncake, for example hostname or the existing local host identity. The best-effort fallback should be documented as a convenience mode, not a strict correctness guarantee. In containerized or multi-NIC environments, operators should set `MOONCAKE_HOST_ID` explicitly. ### 3. Register client host metadata on master Master needs to know the writer's host even when the writer does not mount a segment. Therefore storing `host_id` only on `Segment` is not enough. Master should maintain: ```text client_host_id_: client_id -> host_id segment_host_id_: segment_name -> host_id segments_by_host_: host_id -> ordered segment names ``` The client should register or refresh its `host_id` when it connects to master or during the existing client/master setup path. Store clients should also attach `host_id` to mounted segments, so the master can rebuild host-to-segment indexes when segments are mounted, remounted, unmounted, or restored from snapshot. ### 4. Build host-ordered preferred segments on Put When master handles `PutStart`, the priority should be: 1. If `config.preferred_segment` is set, use it exactly as today. 2. Else if `config.preferred_segments` is set, use it exactly as today. 3. Else if host-aware local-first allocation is enabled, build a preferred segment list from `client_id -> host_id -> segment_names`. 4. Else use the existing allocation behavior. Pseudo-code: ```cpp std::vector<std::string> preferred_segments; if (!config.preferred_segment.empty()) { preferred_segments.push_back(config.preferred_segment); } else if (!config.preferred_segments.empty()) { preferred_segments = config.preferred_segments; } else if (local_first_enabled) { preferred_segments = BuildHostOrderedSegments(client_id, key); } allocation_strategy_->Allocate( allocator_manager, value_length, config.replica_num, preferred_segments); ``` ### 5. Host fallback ordering The host fallback order should be deterministic and stable. For the first version, sort active `host_id`s lexicographically and rotate the list so the writer's host is first. Example: ```text active hosts: host0, host1, host2, host3 ``` Writer host orders: ```text host0 -> host0, host1, host2, host3 host1 -> host1, host2, host3, host0 host2 -> host2, host3, host0, host1 host3 -> host3, host0, host1, host2 ``` This avoids the common random fallback problem where many writers may randomly fallback to the same remote segment at the same time. ### 6. Stable segment rotation within a host If a host has multiple segments, always trying the first segment creates a local hotspot. Instead, the segment order inside each host should use a stable key-based rotation. Example: ```text host0 segments sorted by name: host0-store0, host0-store1, host0-store2 start = Hash(key) % 3 if start = 1: host0-store1, host0-store2, host0-store0 ``` The same rotation is applied independently for each host in the fallback chain. The resulting segment list for writer on host1 may look like: ```text host1-store1, host1-store0, host2-store0, host2-store1, host3-store1, host3-store0, host0-store0, host0-store1 ``` This remains deterministic for the same key and topology, while spreading first-choice pressure across segments on the same host. ### 7. Fallback behavior The allocation strategy already supports trying preferred segments first and then continuing with fallback allocation if the preferred list cannot satisfy the request. With host-aware local-first allocation enabled, the preferred list can include all active segments in host fallback order. For `replica_num = 1`, this means: ```text try local host segments try next host segments try next host segments ... return failure only if no segment can allocate ``` For the first implementation, this RFC targets the common KV cache case where `replica_num = 1`. For `replica_num > 1`, the behavior should be explicitly defined before enabling this policy by default. A future extension may place the first replica locally and spread additional replicas across different hosts. ## Configuration Add a client-side host identity variable: ```bash MOONCAKE_HOST_ID=hostA ``` Add a master/client allocation policy switch. The exact naming can be discussed, but the intended semantics are: ```bash MOONCAKE_LOCAL_FIRST_ALLOC=1 MOONCAKE_HOST_FALLBACK_POLICY=ordered ``` Initial default: ```text MOONCAKE_LOCAL_FIRST_ALLOC=0 ``` The default can stay disabled for conservative backward compatibility. After the behavior is validated, the community can discuss whether local-first should become the default for Mooncake Store memory allocation. ## Compatibility This proposal is backward-compatible if introduced behind a feature flag. 1. Existing `preferred_segment` and `preferred_segments` keep their current semantics and override automatic local-first allocation. 2. Existing clients that do not provide `host_id` continue to use the current allocation behavior. 3. Existing `Segment.name` matching behavior is unchanged. 4. Existing deployments can keep using manual preferred segment configuration. 5. Snapshot and HA code must include the new host metadata or rebuild it from segment/client registration after recovery. ## Failure and Edge Cases ### Writer has no `host_id` Fallback to current allocation behavior. ### Writer host has no active store segment Skip the writer host and continue with the next hosts in ordered fallback. ### Store segment is full Continue to the next segment in the generated preferred list. ### Store host goes down The host's unavailable segments should be removed from active allocator indexes. The generated preferred list should only include active segments. ### Multiple clients on the same physical host All clients must report the same `host_id`. This is the main reason to keep `host_id` separate from `local_hostname`. ### Multi-NIC and container environments Operators should set `MOONCAKE_HOST_ID` explicitly. Automatically derived hostnames or IPs are best-effort only. ## Implementation Plan This feature is intended to be implemented in one PR with a narrow scope: add host metadata, build automatic local-first preferred segment lists, and verify the allocation behavior. The PR should not include NUMA/GPU-aware placement or topology-aware remote ordering. ### Metadata changes 1. Add `host_id` to Mooncake client configuration. 2. Resolve `host_id` from `MOONCAKE_HOST_ID`, explicit config, or best-effort fallback. 3. Extend the client/master registration or heartbeat path to record `client_id -> host_id`. 4. Add `host_id` to mounted segment metadata. 5. Maintain master indexes: ```text client_host_id_ segment_host_id_ segments_by_host_ ``` ### Allocation policy 1. Add a helper to build host-ordered segment lists: ```cpp std::vector<std::string> BuildHostOrderedSegments( const UUID& writer_client_id, const std::string& key); ``` 2. Sort active hosts deterministically. 3. Rotate host order so writer host is first. 4. Sort segments within each host deterministically. 5. Rotate segments within each host using `Hash(key) % segment_count`. 6. Use the generated list as `preferred_segments` only when explicit preferred segments are absent. ### Observability Add minimal logs or metrics for: ```text local-first allocation enabled/disabled writer host_id missing generated preferred segment count allocation landed on local host vs remote host fallback depth by host ``` This is important for validating whether the new policy actually reduces cross-host writes and whether ordered fallback creates pressure on specific hosts. ### Tests Add unit tests for: 1. `host_id` resolution priority. 2. master host indexes on mount, unmount, and remount. 3. `BuildHostOrderedSegments` with four hosts: ```text host0 -> host0, host1, host2, host3 host1 -> host1, host2, host3, host0 host2 -> host2, host3, host0, host1 host3 -> host3, host0, host1, host2 ``` 4. stable key-based segment rotation inside one host. 5. explicit `preferred_segment(s)` overriding automatic local-first allocation. 6. missing `host_id` falling back to current behavior. 7. local host with no active segment falling back to the next host. Add integration tests for: 1. standalone-store mode where requester mounts no segment but still registers `host_id`. 2. multiple store segments on one host. 3. local segment full, remote ordered fallback succeeds. 4. segment unmount removes it from host-aware allocation candidates. ## Alternatives Considered ### Upper layer passes `preferred_segment` vLLM and SGLang can manually set `preferred_segment` or `preferred_segments`. This works for controlled deployments, but it requires the upper layer or deployment system to know exact Mooncake segment names. It also does not solve automatic discovery of all local segments. ### Infer host by `Segment.te_endpoint` Master could parse IPs from `te_endpoint` and group segments by IP. This is fragile in multi-NIC, container, hostname/IP-mixed, and standalone requester scenarios. It also does not provide host identity for clients that do not mount segments. ### Reuse `local_hostname` as host identity This breaks when multiple clients on the same physical host use different logical names, for example `hostA-rank0`, `hostA-rank1`, and `hostA-store0`. `host_id` should be a separate field. ### Keep random fallback Random fallback is simple, but multiple writers can fallback to the same remote segment or host at the same time. Ordered host fallback provides more predictable behavior and avoids obvious random collision patterns. ## Open for Discussion 1. Should `MOONCAKE_LOCAL_FIRST_ALLOC` be enabled by default after the feature is implemented, or remain opt-in? 2. Should the fallback host order always be lexicographic rotation, or should operators be able to configure a topology-aware order? 3. How should `replica_num > 1` behave under local-first allocation? 4. Should `host_id` be stored in snapshots, rebuilt from client registration, or both? 5. What hash function should be used for stable segment rotation inside a host? 6. Should this feature reuse `prefer_alloc_in_same_node`, or introduce a more explicit policy name such as `local_first_allocation`? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:18:16 +00:00

bsakaguchi commented

2026-10-29 21:18:16 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 26, 2026):

Thanks for opening this issue, @yokinoshitayoki!

Field	Value
Issue	#2633
GitHub user ID	`113957881`
Reporter	@yokinoshitayoki

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jun 26, 2026): Thanks for opening this issue, @yokinoshitayoki! | Field | Value | |-------|-------| | **Issue** | #2633 | | **GitHub user ID** | `113957881` | | **Reporter** | @yokinoshitayoki | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:30:00 +00:00

[PR #693] [MERGED] [Store] fix: add client list metrics #1749