[PR #1122] [MERGED] feat[Store]: Add multi dummy clients support for real client

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#2024

bsakaguchi commented

2026-10-29 21:32:11 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1122
Author: @YiXR
Created: 11/26/2025
Status: ✅ Merged
Merged: 11/28/2025
Merged by: @ykwd

Base: main ← Head: xinyi-client

📝 Commits (2)

a660986 [Store]: Add multi dummy client support for real client
281105f [CI]: Add multi dummy client test

📊 Changes

6 files changed (+432 additions, -153 deletions)

View changed files

📝 mooncake-store/include/real_client.h (+74 -38)
📝 mooncake-store/src/dummy_client.cpp (+18 -16)
📝 mooncake-store/src/real_client.cpp (+232 -92)
📝 mooncake-store/src/real_client_main.cpp (+6 -6)
➕ mooncake-wheel/tests/test_multi_dummy_clients.py (+97 -0)
📝 scripts/run_tests.sh (+5 -1)

📄 Description

Description

Releated to https://github.com/kvcache-ai/Mooncake/pull/1084
Add multi dummy client support for real client.
Right now, a real client can hold multi dummy clients simultaneous.

Type of Change

Types
- Bug fix
- New feature
  - Transfer Engine
  - Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- Documentation update
- Other

How Has This Been Tested?

Start Mooncake Master

mooncake_master --enable_http_metadata_server=true --eviction_high_watermark_ratio=0.95

Start Mooncake Client (Real Client)

mooncake_client --global_segment_size 40GB

Start SGLang using Dummy Client (TP=2 means 2 Dummy Clients):

MOONCAKE_CLIENT=127.0.0.1:50052 \ MOONCAKE_STANDALONE_STORAGE=1 \ python3 -m sglang.launch_server \ --enable-hierarchical-cache \ --hicache-storage-backend mooncake\ --mem-fraction-static 0.8 \ --tp 2 \ --model-path Qwen/Qwen2.5-14B-Instruct

Start Benchmark

python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name sharegpt \ --dataset-path /root/ShareGPT_V3_unfiltered_cleaned_split.json \ --sharegpt-context-len 8192 \ --num-prompts 1024 \ --request-rate 128 \ --max-concurrency 100 \ --warmup-requests 16 \ --base-url http://localhost:30000

Performance

Test on Qwen/Qwen2.5-14B-Instruct，TP 2，NVIDIA A10

Metric	Before Client Separation	After Client Separation	Difference (%)
Throughput
Request throughput (req/s)	3.16	3.20	+1.27%
Total token throughput (tok/s)	1562.30	1584.85	+1.44%
End-to-End Latency
Mean E2E Latency (ms)	28480.76	28003.09	-1.68% (Better)
Median E2E Latency (ms)	22871.35	22494.87	-1.65% (Better)
Time to First Token (TTFT)
Mean TTFT (ms)	10405.14	9918.66	-4.68% (Better)
Median TTFT (ms)	9815.66	9652.91	-1.66% (Better)
Inter-Token Latency (ITL)
Mean ITL (ms)	93.20	93.24	+0.04%
Median ITL (ms)	61.37	61.33	-0.07% (Better)

Based on the benchmark results, the performance of the two deployment methods—before and after separating the client—is virtually identical.

Checklist

I have performed a self-review of my own code.
I have updated the documentation.
I have added tests to prove my changes are effective.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1122 **Author:** [@YiXR](https://github.com/YiXR) **Created:** 11/26/2025 **Status:** ✅ Merged **Merged:** 11/28/2025 **Merged by:** [@ykwd](https://github.com/ykwd) **Base:** `main` ← **Head:** `xinyi-client` --- ### 📝 Commits (2) - [`a660986`](https://github.com/kvcache-ai/Mooncake/commit/a660986d6cf57c983ba41d304865f5bdaf00ca8c) [Store]: Add multi dummy client support for real client - [`281105f`](https://github.com/kvcache-ai/Mooncake/commit/281105f72e2e40c5af92b3cf34788a845e27bee4) [CI]: Add multi dummy client test ### 📊 Changes **6 files changed** (+432 additions, -153 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-store/include/real_client.h` (+74 -38) 📝 `mooncake-store/src/dummy_client.cpp` (+18 -16) 📝 `mooncake-store/src/real_client.cpp` (+232 -92) 📝 `mooncake-store/src/real_client_main.cpp` (+6 -6) ➕ `mooncake-wheel/tests/test_multi_dummy_clients.py` (+97 -0) 📝 `scripts/run_tests.sh` (+5 -1) </details> ### 📄 Description ## Description Releated to https://github.com/kvcache-ai/Mooncake/pull/1084 Add multi dummy client support for real client. Right now, a real client can hold multi dummy clients simultaneous. ## Type of Change * Types - [ ] Bug fix - [x] New feature - [ ] Transfer Engine - [x] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [x] CI/CD - [ ] Documentation update - [ ] Other ## How Has This Been Tested? ### Start Mooncake Master `mooncake_master --enable_http_metadata_server=true --eviction_high_watermark_ratio=0.95` ### Start Mooncake Client (Real Client) `mooncake_client --global_segment_size 40GB` ### Start SGLang using Dummy Client (TP=2 means 2 Dummy Clients): `MOONCAKE_CLIENT=127.0.0.1:50052 \ MOONCAKE_STANDALONE_STORAGE=1 \ python3 -m sglang.launch_server \ --enable-hierarchical-cache \ --hicache-storage-backend mooncake\ --mem-fraction-static 0.8 \ --tp 2 \ --model-path Qwen/Qwen2.5-14B-Instruct` ### Start Benchmark `python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name sharegpt \ --dataset-path /root/ShareGPT_V3_unfiltered_cleaned_split.json \ --sharegpt-context-len 8192 \ --num-prompts 1024 \ --request-rate 128 \ --max-concurrency 100 \ --warmup-requests 16 \ --base-url http://localhost:30000` ### Performance Test on Qwen/Qwen2.5-14B-Instruct，TP 2，NVIDIA A10 | Metric | Before Client Separation | After Client Separation | Difference (%) | |---------------------------------|--------------------------|-------------------------|---------------------| | **Throughput** | | | | | Request throughput (req/s) | 3.16 | 3.20 | +1.27% | | Total token throughput (tok/s) | 1562.30 | 1584.85 | +1.44% | | **End-to-End Latency** | | | | | Mean E2E Latency (ms) | 28480.76 | 28003.09 | -1.68% (Better) | | Median E2E Latency (ms) | 22871.35 | 22494.87 | -1.65% (Better) | | **Time to First Token (TTFT)** | | | | | Mean TTFT (ms) | 10405.14 | 9918.66 | -4.68% (Better) | | Median TTFT (ms) | 9815.66 | 9652.91 | -1.66% (Better) | | **Inter-Token Latency (ITL)** | | | | | Mean ITL (ms) | 93.20 | 93.24 | +0.04% | | Median ITL (ms) | 61.37 | 61.33 | -0.07% (Better) | Based on the benchmark results, the performance of the two deployment methods—before and after separating the client—is virtually identical.  ## Checklist - [x] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [x] I have added tests to prove my changes are effective. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

closed this issue
added the
pull-request
label

Rows
Columns

[PR #1122] [MERGED] feat[Store]: Add multi dummy clients support for real client #2024