[PR #1084] [MERGED] feat[Store]: Add standalone deployment implementation for Client #2003

New issue

Closed

opened 2026-10-29 21:32:02 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:32:02 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1084
Author: @YiXR
Created: 11/20/2025
Status: ✅ Merged
Merged: 11/25/2025
Merged by: @stmatengss

Base: main ← Head: split-client

📝 Commits (6)

d8b7569 [Store]: Add independent deployment implementation for Client
757d212 [Doc]: mooncake-store: add introduce for client standalone mode
751d62b [CI]: Add dummy client test
54784f7 [Store]: Change dummy client setup into a new func
8f3ff99 [Store]: Add more log for dummy client
e914a78 Merge remote-tracking branch 'upstream/main'

📊 Changes

28 files changed (+2147 additions, -229 deletions)

View changed files

📝 doc/en/mooncake-store.md (+34 -4)
📝 doc/zh/mooncake-store.md (+33 -4)
📝 docs/source/design/mooncake-store.md (+34 -4)
📝 mooncake-integration/store/store_py.cpp (+121 -24)
📝 mooncake-store/include/client_buffer.hpp (+12 -4)
📝 mooncake-store/include/client_service.h (+0 -0)
➕ mooncake-store/include/dummy_client.h (+185 -0)
➕ mooncake-store/include/pyclient.h (+116 -0)
📝 mooncake-store/include/real_client.h (+94 -39)
📝 mooncake-store/include/utils.h (+93 -0)
📝 mooncake-store/src/CMakeLists.txt (+9 -4)
📝 mooncake-store/src/client_buffer.cpp (+28 -2)
📝 mooncake-store/src/client_service.cpp (+1 -1)
➕ mooncake-store/src/dummy_client.cpp (+479 -0)
📝 mooncake-store/src/real_client.cpp (+343 -74)
➕ mooncake-store/src/real_client_main.cpp (+74 -0)
📝 mooncake-store/tests/client_integration_test.cpp (+1 -1)
📝 mooncake-store/tests/e2e/client_wrapper.h (+1 -1)
📝 mooncake-store/tests/non_ha_reconnect_test.cpp (+1 -1)
📝 mooncake-store/tests/pybind_client_test.cpp (+38 -35)

...and 8 more files

📄 Description

Description

Thanks @wanyue-wy @stmatengss @XucSh for helping with architectural design.

Type of Change

Architecture Overview

Split Client into DummyClient and RealClient, which RealClient is an independent process.
Full Compatibility: Fully compatible with the old arch.
Flexible Deployment: After applying the detached deployment patch to Sglang（tmp patch：https://github.com/YiXR/sglang/tree/xinyi-client, the Client detached deployment mode can be enabled via configuration. The non-detached mode can still be chosen.
Zero-Copy Performance: Introduces shared memory to support zero-copy operations between RealClient and DummyClient, ensuring that performance is on par with the non-detached mode.
Resilience: The RealClient continues to run and provide Storage for the Master even after the DummyClient exits.

Code Modification Overview

File Creation/Change Description

Renamed client.cpp & client.h to client_service.cpp & client_service.h.
Added real_client_main.cpp as the main process for the RealClient.
Added dummy_client.cpp & dummy_client.h for the DummyClient.
Renamed pybind_client.cpp & pybind_client.h to real_client.cpp & real_client.h. All instances of class PyClient have been renamed to class RealClient.
Added pyclient.h, which defines a new base class PyClient. PyClient maintains the interface to ensure backward compatibility. Both RealClient & DummyClient inherit from it and implement the corresponding functions.

Key Implementation/Change Details

mooncake-integration/store/store_py.cpp

This is the entry point for the Python compatibility layer, where the decision to use detached deployment is made. The store in store_py is defined as a PyClient base class pointer. If the client detached mode is selected, the store is instantiated as a DummyClient; otherwise, it's instantiated as a RealClient.

mooncake-store/src/dummy_client.cpp

DummyClient Implementation: Implements the DummyClient, which is primarily responsible for connecting to the RealClient and initiating RPC calls. Its high-level interface inherits from PyClient to remain consistent with the old code.
Shared Memory Creation: The DummyClient is responsible for creating the shared memory. During its setup phase, it creates the shared memory and notifies the RealClient to mmap it and register it with the transfer engine.
Shared Memory Details:
a. The shared memory created by the DummyClient provides memory space for the L2 cache (like mem_pool_host in SGLang) and the client's local buffer (held by client_buffer_allocator_).
b. shm_size = local_buffer_size + mem_pool_host_size. The first half of the shared memory is used for the L2 cache, and the second half is for the client's local buffer.
c. During the DummyClient setup phase, the RealClient is notified to register the entire shared memory block with the transfer engine at once. Therefore, there is no need to later call registerLocalMemory separately for the L2 cache.
d. After notifying the RealClient to mmap the shared memory, the DummyClient immediately calls shm_unlink. This removes the shared memory file, preventing any other external processes from mapping it. Since the RealClient has already completed the mmap, it can continue to access this memory. Due to the kernel's automatic reclamation mechanism for shared memory, the memory will be automatically freed when its reference count drops to zero (i.e., when both DummyClient and RealClient unmap it).
Keep-Alive: The DummyClient has a Ping thread to maintain its connection with the RealClient.

mooncake-store/src/real_client.cpp

Code Reuse: The RealClient reuses all the implementation from the original pybind_client (including creating the global_segment, holding the transfer engine, communicating with the master, etc.).
RPC Service: The RealClient registers its internal functions with an RPC service so they can be called by the DummyClient.
Shared Memory Details:
a. When notified by the DummyClient to map the shared memory, the RealClient will mmap it and record both the DummyClient's side VA (Virtual Address) and its own RealClient side VA. This facilitates virtual address translation in subsequent zero-copy interface calls.
b. The RealClient uses the second half of the shared memory to create its client_buffer_allocator_.
Virtual Address Translation: For RPCs involving zero-copy or using the client local buffer for transfers, virtual addresses are passed as parameters/return values. The RealClient, when handling these functions, must use the previously recorded VAs to perform virtual address translation. This ensures that the virtual addresses it uses are valid in its own context, and the virtual addresses it returns to the DummyClient are valid in the DummyClient's context.
Health Check: The RealClient uses a Ping mechanism to continuously monitor the status of the DummyClient. If it detects that the DummyClient is unavailable, it will unmap the corresponding shared memory to ensure timely memory cleanup.

mooncake-store/src/client_buffer.cpp

External Buffer Support: ClientBufferAllocator now supports being constructed with an external buffer. This change is made to accommodate the requirement of creating a client_buffer_allocator_ using the pre-allocated shared memory.

Types
- Bug fix
- New feature
  - Transfer Engine
  - Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- Documentation update
- Other

How Has This Been Tested?

To support detached Client mode, the patch described above must be applied to Sglang. (Without the patch, Sglang can still start normally in non-detached mode).

Launching in Detached Mode
a. Start the master:

mooncake_master --enable_http_metadata_server=true --eviction_high_watermark_ratio=0.95

b. Start the RealClient (listens on port 50052 by default):

mooncake_client --global_segment_size=4GB

c. Start Sglang. Set MOONCAKE_MASTER to the RealClient's address and MOONCAKE_STANDALONE_STORAGE=1 to enable standalone client deployment.

MOONCAKE_MASTER=127.0.0.1:50052 \
MOONCAKE_STANDALONE_STORAGE=1 \
python3 -m sglang.launch_server \
    --enable-hierarchical-cache \
    --hicache-storage-backend mooncake \
    --model-path Qwen/Qwen3-0.6B

Checklist

I have performed a self-review of my own code.
I have updated the documentation.
I have added tests to prove my changes are effective.

TODO List

Split Client into DummyClient and RealClient.
Demo for SGLang.
Add CI for DummyClient.
Enable a single RealClient to serve multiple DummyClients. https://github.com/kvcache-ai/Mooncake/pull/1122
Add support for RPC metric collection in RealClient and DummyClient.
Implement a reconnection mechanism for RealClient and DummyClient. https://github.com/kvcache-ai/Mooncake/pull/1146
Introduce a standalone allocator for share mem, enabling Python applications like SGLang to directly allocate shared memory. https://github.com/kvcache-ai/Mooncake/pull/1177
Expand DummyClient interface support.
a. Currently supported:
- put
- put_batch
- put_parts
- remove
- removeByRegex
- removeAll
- isExist
- batchIsExist
- getSize
- get_buffer (replaced by get_buffer_info)
- batch_put_from
- batch_get_into
  Unsupported:
- batch_get_buffer
- get_into
- put_from
- put_from_with_metadata
- batch_put_from_multi_buffers
- batch_get_into_multi_buffers
- get_tensor
- put_tensor
- batch_put_tensor
- batch_get_tensor

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1084 **Author:** [@YiXR](https://github.com/YiXR) **Created:** 11/20/2025 **Status:** ✅ Merged **Merged:** 11/25/2025 **Merged by:** [@stmatengss](https://github.com/stmatengss) **Base:** `main` ← **Head:** `split-client` --- ### 📝 Commits (6) - [`d8b7569`](https://github.com/kvcache-ai/Mooncake/commit/d8b7569b1ec9e1d6690ebf85f1b16d93570cbb2d) [Store]: Add independent deployment implementation for Client - [`757d212`](https://github.com/kvcache-ai/Mooncake/commit/757d2129dd7c4be73ceea41f6fed45139377e4f2) [Doc]: mooncake-store: add introduce for client standalone mode - [`751d62b`](https://github.com/kvcache-ai/Mooncake/commit/751d62b9d03a7082505817115700e8c28f9cfc36) [CI]: Add dummy client test - [`54784f7`](https://github.com/kvcache-ai/Mooncake/commit/54784f73777da5fd28b324a49997f4e75e23b8a4) [Store]: Change dummy client setup into a new func - [`8f3ff99`](https://github.com/kvcache-ai/Mooncake/commit/8f3ff99a7a51931ff483c7f2584513a0c1dd408e) [Store]: Add more log for dummy client - [`e914a78`](https://github.com/kvcache-ai/Mooncake/commit/e914a780946b673232ca8f0a7cf0fea7a2fa2dac) Merge remote-tracking branch 'upstream/main' ### 📊 Changes **28 files changed** (+2147 additions, -229 deletions) <details> <summary>View changed files</summary> 📝 `doc/en/mooncake-store.md` (+34 -4) 📝 `doc/zh/mooncake-store.md` (+33 -4) 📝 `docs/source/design/mooncake-store.md` (+34 -4) 📝 `mooncake-integration/store/store_py.cpp` (+121 -24) 📝 `mooncake-store/include/client_buffer.hpp` (+12 -4) 📝 `mooncake-store/include/client_service.h` (+0 -0) ➕ `mooncake-store/include/dummy_client.h` (+185 -0) ➕ `mooncake-store/include/pyclient.h` (+116 -0) 📝 `mooncake-store/include/real_client.h` (+94 -39) 📝 `mooncake-store/include/utils.h` (+93 -0) 📝 `mooncake-store/src/CMakeLists.txt` (+9 -4) 📝 `mooncake-store/src/client_buffer.cpp` (+28 -2) 📝 `mooncake-store/src/client_service.cpp` (+1 -1) ➕ `mooncake-store/src/dummy_client.cpp` (+479 -0) 📝 `mooncake-store/src/real_client.cpp` (+343 -74) ➕ `mooncake-store/src/real_client_main.cpp` (+74 -0) 📝 `mooncake-store/tests/client_integration_test.cpp` (+1 -1) 📝 `mooncake-store/tests/e2e/client_wrapper.h` (+1 -1) 📝 `mooncake-store/tests/non_ha_reconnect_test.cpp` (+1 -1) 📝 `mooncake-store/tests/pybind_client_test.cpp` (+38 -35) _...and 8 more files_ </details> ### 📄 Description ## Description Thanks @wanyue-wy @stmatengss @XucSh for helping with architectural design.  ## Type of Change # Architecture Overview 1. Split Client into DummyClient and RealClient, which RealClient is an independent process. 2. **Full Compatibility**: Fully compatible with the old arch. 3. **Flexible Deployment**: After applying the detached deployment patch to Sglang（tmp patch：https://github.com/YiXR/sglang/tree/xinyi-client, the Client detached deployment mode can be enabled via configuration. The non-detached mode can still be chosen. 4. **Zero-Copy Performance**: Introduces shared memory to support zero-copy operations between `RealClient` and `DummyClient`, ensuring that performance is on par with the non-detached mode. 5. **Resilience**: The `RealClient` continues to run and provide Storage for the Master even after the `DummyClient` exits. # Code Modification Overview ### File Creation/Change Description 1. Renamed `client.cpp` & `client.h` to `client_service.cpp` & `client_service.h`. 2. Added `real_client_main.cpp` as the main process for the `RealClient`. 3. Added `dummy_client.cpp` & `dummy_client.h` for the `DummyClient`. 4. Renamed `pybind_client.cpp` & `pybind_client.h` to `real_client.cpp` & `real_client.h`. All instances of `class PyClient` have been renamed to `class RealClient`. 6. Added `pyclient.h`, which defines a new base class `PyClient`. `PyClient` maintains the interface to ensure backward compatibility. Both `RealClient` & `DummyClient` inherit from it and implement the corresponding functions. # Key Implementation/Change Details ### mooncake-integration/store/store_py.cpp 1. This is the entry point for the Python compatibility layer, where the decision to use detached deployment is made. The `store` in `store_py` is defined as a `PyClient` base class pointer. If the client detached mode is selected, the `store` is instantiated as a `DummyClient`; otherwise, it's instantiated as a `RealClient`. ### mooncake-store/src/dummy_client.cpp 1. **`DummyClient` Implementation**: Implements the `DummyClient`, which is primarily responsible for connecting to the `RealClient` and initiating RPC calls. Its high-level interface inherits from `PyClient` to remain consistent with the old code. 2. **Shared Memory Creation**: The `DummyClient` is responsible for creating the shared memory. During its setup phase, it creates the shared memory and notifies the `RealClient` to `mmap` it and register it with the transfer engine. 3. **Shared Memory Details**: a. The shared memory created by the `DummyClient` provides memory space for the L2 cache (like `mem_pool_host` in SGLang) and the client's local buffer (held by `client_buffer_allocator_`). b. `shm_size = local_buffer_size + mem_pool_host_size`. The first half of the shared memory is used for the L2 cache, and the second half is for the client's local buffer. c. During the `DummyClient` setup phase, the `RealClient` is notified to register the entire shared memory block with the transfer engine at once. Therefore, there is no need to later call `registerLocalMemory` separately for the L2 cache. d. After notifying the `RealClient` to `mmap` the shared memory, the `DummyClient` immediately calls `shm_unlink`. This removes the shared memory file, preventing any other external processes from mapping it. Since the `RealClient` has already completed the `mmap`, it can continue to access this memory. Due to the kernel's automatic reclamation mechanism for shared memory, the memory will be automatically freed when its reference count drops to zero (i.e., when both `DummyClient` and `RealClient` unmap it). 4. **Keep-Alive**: The `DummyClient` has a Ping thread to maintain its connection with the `RealClient`. ### mooncake-store/src/real_client.cpp 1. **Code Reuse**: The `RealClient` reuses all the implementation from the original `pybind_client` (including creating the `global_segment`, holding the transfer engine, communicating with the master, etc.). 2. **RPC Service**: The `RealClient` registers its internal functions with an RPC service so they can be called by the `DummyClient`. 3. **Shared Memory Details**: a. When notified by the `DummyClient` to map the shared memory, the `RealClient` will `mmap` it and record both the `DummyClient`'s side VA (Virtual Address) and its own `RealClient` side VA. This facilitates virtual address translation in subsequent zero-copy interface calls. b. The `RealClient` uses the second half of the shared memory to create its `client_buffer_allocator_`. 4. **Virtual Address Translation**: For RPCs involving zero-copy or using the client local buffer for transfers, virtual addresses are passed as parameters/return values. The `RealClient`, when handling these functions, must use the previously recorded VAs to perform virtual address translation. This ensures that the virtual addresses it uses are valid in its own context, and the virtual addresses it returns to the `DummyClient` are valid in the `DummyClient`'s context. 5. **Health Check**: The `RealClient` uses a Ping mechanism to continuously monitor the status of the `DummyClient`. If it detects that the `DummyClient` is unavailable, it will unmap the corresponding shared memory to ensure timely memory cleanup. ### mooncake-store/src/client_buffer.cpp 1. **External Buffer Support**: `ClientBufferAllocator` now supports being constructed with an external buffer. This change is made to accommodate the requirement of creating a `client_buffer_allocator_` using the pre-allocated shared memory. * Types - [ ] Bug fix - [x] New feature - [ ] Transfer Engine - [x] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [x] CI/CD - [x] Documentation update - [ ] Other ## How Has This Been Tested? 1. To support detached Client mode, the patch described above must be applied to Sglang. (Without the patch, Sglang can still start normally in non-detached mode). 2. **Launching in Detached Mode** a. Start the master: ```bash mooncake_master --enable_http_metadata_server=true --eviction_high_watermark_ratio=0.95 ``` b. Start the `RealClient` (listens on port 50052 by default): ```bash mooncake_client --global_segment_size=4GB ``` c. Start Sglang. Set `MOONCAKE_MASTER` to the `RealClient`'s address and `MOONCAKE_STANDALONE_STORAGE=1` to enable standalone client deployment. ```bash MOONCAKE_MASTER=127.0.0.1:50052 \ MOONCAKE_STANDALONE_STORAGE=1 \ python3 -m sglang.launch_server \ --enable-hierarchical-cache \ --hicache-storage-backend mooncake \ --model-path Qwen/Qwen3-0.6B ```  ## Checklist - [x] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. # TODO List - [x] Split Client into DummyClient and RealClient. - [x] Demo for SGLang. - [x] Add CI for DummyClient. - [x] Enable a single `RealClient` to serve multiple `DummyClient`s. https://github.com/kvcache-ai/Mooncake/pull/1122 - [ ] Add support for RPC metric collection in `RealClient` and `DummyClient`. - [x] Implement a reconnection mechanism for `RealClient` and `DummyClient`. https://github.com/kvcache-ai/Mooncake/pull/1146 - [x] Introduce a standalone allocator for share mem, enabling Python applications like SGLang to directly allocate shared memory. https://github.com/kvcache-ai/Mooncake/pull/1177 - [ ] Expand `DummyClient` interface support. a. Currently supported: - [x] `put` - [x] `put_batch` - [x] `put_parts` - [x] `remove` - [x] `removeByRegex` - [x] `removeAll` - [x] `isExist` - [x] `batchIsExist` - [x] `getSize` - [x] `get_buffer` (replaced by `get_buffer_info`) - [x] `batch_put_from` - [x] `batch_get_into` Unsupported: - [ ] `batch_get_buffer` - [ ] `get_into` - [ ] `put_from` - [ ] `put_from_with_metadata` - [ ] `batch_put_from_multi_buffers` - [ ] `batch_get_into_multi_buffers` - [ ] `get_tensor` - [ ] `put_tensor` - [ ] `batch_put_tensor` - [ ] `batch_get_tensor` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:32:02 +00:00