kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[PR #1226] [MERGED] Add Local Cache Mechanism for Mooncake Store Client #2088

New issue

Closed

opened 2026-10-29 21:32:41 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:32:41 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1226
Author: @Shichang-Zhang
Created: 12/17/2025
Status: ✅ Merged
Merged: 2/11/2026
Merged by: @stmatengss

Base: main ← Head: upstream-zsc-working

📝 Commits (10+)

36ee91b feat(Store): add local hot cache for client
4158e59 feat(Store): add client local hot cache log to show performance
9d4aa0d fix: local hot cache initialize bug
e9686ed fix(Store): Mooncake put slice is max 16MB， so make local hot cache block 16MB
6453b95 feat(Store): move local hot cache initialization to Client::Create
6768f3c feat(Store): local hot cache remove unused small block implementation
8848351 feat(Store): add client local hot cache unit test
602801e fix(Store): modify client local hot cache suit with v0.3.7
9d57b16 feat(Store): change local hot cache unit tes
19cca30 fix: initialize local hot cache with negative value

📊 Changes

9 files changed (+1871 additions, -10 deletions)

View changed files

📝 mooncake-integration/store/store_py.cpp (+0 -1)
📝 mooncake-store/include/client_service.h (+65 -0)
➕ mooncake-store/include/local_hot_cache.h (+199 -0)
📝 mooncake-store/src/CMakeLists.txt (+1 -0)
📝 mooncake-store/src/client_service.cpp (+251 -6)
➕ mooncake-store/src/local_hot_cache.cpp (+355 -0)
📝 mooncake-store/src/transfer_task.cpp (+35 -3)
📝 mooncake-store/tests/CMakeLists.txt (+1 -0)
➕ mooncake-store/tests/client_local_hot_cache_test.cpp (+964 -0)

📄 Description

Description

This PR introduces the client hot cache feature mentioned in Issue 1062.

This feature is enabled by setting the environment variable LOCAL_HOT_CACHE_SIZE (in bytes). The mooncake store client will allocate memory blocks for caching hot data on a per-request basis. The block size is configurable via the environment variable LOCAL_HOT_BLOCK_SIZE (in bytes), with a default value of 16MB. Memory allocation follows the configured block size unit. If LOCAL_HOT_CACHE_SIZE is set to less than the block size (either the default 16MB or the value specified by LOCAL_HOT_BLOCK_SIZE), zero, negative, or other invalid inputs (such as non-numeric strings), the client hot cache feature will be disabled.

For the standalone client scenario, the feature should be disabled since all things are in local storage.

For the architecture of multiple clients and a master, the feature is recommended for use. Users should adjust the LOCAL_HOT_CACHE_SIZE value based on their actual deployment context to strike an optimal balance: the performance gain from reduced cross-node data transfer outweighs the overhead of additional local cache memory.

Type of Change

Types
- Bug fix
- New feature
  - Transfer Engine
  - [√ ] Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- Documentation update
- Other

How Has This Been Tested?

Run unit test for the feature mooncake-store/tests/client_local_hot_cache_test.cpp.

Performance Test

Software

Name	Version
vLLM	v0.9.1
vLLM Ascend	v0.9.1
Mooncake	v0.3.7
EvalScope	v1.2.0
CANN	8.2RC1
Kubernetes	v1.33.1

Hardware

Name	Description
Machine	Atlas 800I A2
NPU	8 * 昇腾910B4
CPU	4 * 鲲鹏920
Memory	32G * 32
Network	8 * 200GE QSFP

Mooncake Performance Test

Environment

We run the test on a Kubernetes cluster that involves two Atlas 800I A2 machines.

Method

Start master-service pod as Mooncake Master and redis pod as the metadata server.
Start a Mooncake Client pod on node 1.
Call MooncakeDistributedStore:Put interface to store data (e.g., 1GB) on Mooncake Client 1.

3.1 Store 10 keys with corresponding data (e.g., 1GB total) in Mooncake.

3.2 All data slices are stored on node 1 since only one client exists.
Start another Mooncake Client pod with local hot cache (e.g., 1GB) enabled on node 2.
Call MooncakeDistributedStore:Get/BatchGet interface from Mooncake Client 2 to retrieve the stored data.

5.1 All data slices are on node 1, so all slices requested by Mooncake Client 2 will be transferred remotely.

5.2 Use TCP protocol to increase network transmission latency.

5.3 After retrieving data from remote node 1, the slices are stored in the local hot cache.
Record the latency of the Get/BatchGet calls.
Repeatedly call MooncakeDistributedStore:Get/BatchGet interface from Mooncake Client 2 and record the latency.
7.1 These requests result in cache hits.

Result

Method	Total Key Num	Data Size (MB)	Total Data Size (MB)	Local Cache Size (MB)	Get/BatchGet Per Round	Batch Size	Test Rounds	First Round Latency (ms)	Average Round Latency (ms)	Latency Delta (%)
get	25	10	250	0	25	-	10	767.64	881.02	-
get	25	10	250	400	25	-	10	1204.43	287.38	76.14
get	25	25	625	0	25	-	10	757.65	760.36	-
get	25	25	625	800	25	-	10	905.76	259.48	71.35
batchGet	25	10	250	0	5	5	10	489.24	463.28	-
batchGet	25	10	250	400	5	5	10	425.94	151.23	64.49
batchGet	25	25	625	0	5	5	10	433.15	440.47	-
batchGet	25	25	625	800	5	5	10	504.91	143.83	71.51

Note: The latency is measured for each key request, for BatchGet interface, first round latency is the average latency of first 2 key.

End to End Inference Performance Test

Environment

We run the test on a Kubernetes cluster that involves three Atlas 800I A2 machines. We deployed the inference service with vLLM v0 and Mooncake.

Method

Start master-service pod as Mooncake Master and redis pod as the metadata server.
Start 5 prefill pods and 5 decode pods running vLLM and Mooncake Client. Each client offers 30GB store memory.

2.1 Deploy 10 Mooncake Clients to increase the probability of remote slice data transmission.

2.2 Use TCP protocol and deploy across three machines to increase network transmission latency.

2.3 Prefill instances do not enable the local hot cache feature, while decode instances enable it.
Start proxy server pod for routing inference requests

3.1 Use the example proxy server from vLLM.
Start EvalScope script

4.1 Send 1000 inference requests at concurrency 5 or 10, with a fraction of exact duplicate requests to simulate hotspot/cache hits. Average input length is 4096 tokens and output length is fixed at 256 tokens.

4.2 Use a random dataset to vary prefix lengths, controlling prefix sharing across requests and thus the number of hot slices (and hit rate).

4.3 EvalScope summarizes and logs key performance metrics such as throughput, latency (p50/p95/p99), and success/error rates.

Test Script

vLLM prefill

export MOONCAKE_CONFIG_PATH=/app/mooncake_config.json
echo "{
    \"local_hostname\": \"$POD_IP\",
    \"metadata_server\": \"redis://redis:6379\",
    \"master_server_address\": \"mooncake-master:30089\",
    \"protocol\": \"tcp\",
    \"device_name\": \"\",
    \"global_segment_size\": 32212254720
}" > ${MOONCAKE_CONFIG_PATH}

VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3-8B \
  --port 8100 \
  --max-model-len 10000 \
  --gpu-memory-utilization 0.8 \
  --block-size 128 \
  --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer","kv_buffer_device":"npu"}'

Note: This is the start command of the prefill pod. So POD_IP is the node address of this pod.

vLLM decode

export MOONCAKE_CONFIG_PATH=/app/mooncake_config.json
echo "{
    \"local_hostname\": \"$POD_IP\",
    \"metadata_server\": \"redis://redis:6379\",
    \"master_server_address\": \"mooncake-master:30089\",
    \"protocol\": \"tcp\",
    \"device_name\": \"\",
    \"global_segment_size\": 32212254720
}" > ${MOONCAKE_CONFIG_PATH}

VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3-8B \
  --port 8200 \
  --max-model-len 10000 \
  --gpu-memory-utilization 0.8 \
  --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer","kv_buffer_device":"npu"}'

Note: This is the start command of the decode pod. So POD_IP is the node address of this pod.

EvalScope DataSet

import json
import random
import os
import numpy as np
from datetime import datetime
from typing import List, Dict
from modelscope import AutoTokenizer


# --- Core configuration ---
MODEL_NAME = "Qwen/Qwen3-8B"
REQUEST_LEN = 4096  # Target token length for each request
DATASET_SIZE = 1000  # Total number of requests
REPEAT_PERCENT = 0.5  # Ratio of identical (repeat) requests (0.0-1.0)

# Fixed seeds for reproducibility
RANDOM_SEED = 42

class IdenticalDatasetGenerator:
    """Dataset generator for identical/repeated requests."""
    
    def __init__(self, model_name: str = MODEL_NAME):
        """Initialize the generator and load the tokenizer."""
        self.model_name = model_name
        
        print(f"Loading tokenizer from ModelScope: {model_name} ...")
        self.tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
        if self.tokenizer.pad_token is None:
            self.tokenizer.pad_token = self.tokenizer.eos_token
        
        print(f"Tokenizer loaded. Vocab size: {self.tokenizer.vocab_size}")
        
    def generate_random_tokens(self, length: int) -> str:
        """
        Generate random text that encodes to the target number of tokens.
        A decode->encode roundtrip is used to approximate exact token count.
        
        Args:
            length: Target number of tokens.
            
        Returns:
            A text string (ideally encoding to exactly `length` tokens).
        """
        if length <= 0:
            return ""
        
        # Sample token IDs, decode to text, then re-encode to validate the count.
        token_ids = np.random.choice(
            self.tokenizer.vocab_size,
            size=length,
            replace=True
        ).tolist()
        
        text = self.tokenizer.decode(token_ids, skip_special_tokens=False)
        
        encoded_ids = self.tokenizer.encode(text, add_special_tokens=False)
        actual_length = len(encoded_ids)
        
        # If the roundtrip shrinks the token count, append additional tokens.
        if actual_length < length:
            needed = length - actual_length
            additional_ids = np.random.choice(
                self.tokenizer.vocab_size,
                size=needed,
                replace=True
            ).tolist()
            additional_text = self.tokenizer.decode(additional_ids, skip_special_tokens=False)
            text = text + additional_text
        
        return text
    
    def generate_request(self) -> str:
        """Generate a single request with REQUEST_LEN tokens."""
        return self.generate_random_tokens(REQUEST_LEN)
    
    def generate_dataset(self) -> List[Dict]:
        """
        Generate the full dataset.
        
        Returns:
            A list of items with `prompt`, plus metadata fields.
        """
        repeat_count = int(DATASET_SIZE * REPEAT_PERCENT)
        unique_count = DATASET_SIZE - repeat_count
        
        print("Generating identical-request dataset...")
        print(
            f"Config: total={DATASET_SIZE}, unique={unique_count}, identical={repeat_count} "
            f"({REPEAT_PERCENT:.1%}), tokens_per_request={REQUEST_LEN}"
        )
        
        dataset = []
        
        identical_request = self.generate_request()
        
        print(f"Generating {unique_count} unique requests...")
        unique_requests = []
        for i in range(unique_count):
            if (i + 1) % 100 == 0:
                print(f"Progress: {i+1}/{unique_count} unique requests generated")
            unique_request = self.generate_request()
            unique_requests.append(unique_request)
        
        print("Building dataset...")
        
        for i, request in enumerate(unique_requests):
            dataset.append({
                "prompt": request,
                "is_identical": False,
                "request_id": i,
            })
        
        for i in range(repeat_count):
            dataset.append({
                "prompt": identical_request,
                "is_identical": True,
                "request_id": unique_count + i,
            })
        
        print("Shuffling dataset...")
        random.shuffle(dataset)
        
        # Reassign request_id after shuffling.
        for i, item in enumerate(dataset):
            item["request_id"] = i
        
        print("Dataset generation complete.")
        return dataset
    
    def save_dataset(self, dataset: List[Dict], output_dir: str = "."):
        """
        Save the dataset to a JSON file (timestamped for uniqueness).
        
        Args:
            dataset: Dataset items.
            output_dir: Output directory.
        """
        os.makedirs(output_dir, exist_ok=True)
        
        identical_count = sum(1 for item in dataset if item.get("is_identical", False))
        
        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
        filename = f"identical_data_{DATASET_SIZE}total_{REQUEST_LEN}len_{REPEAT_PERCENT*100:.0f}pct_repeat_{timestamp}.json"
        filepath = os.path.join(output_dir, filename)
        
        print(f"Saving dataset to: {filepath}")
        
        with open(filepath, 'w', encoding='utf-8') as f:
            json.dump(dataset, f, ensure_ascii=False, indent=2)
        
        file_size_mb = os.path.getsize(filepath) / 1024 / 1024
        print(
            f"Saved. size={file_size_mb:.2f} MB, total={len(dataset)}, "
            f"identical={identical_count}, unique={len(dataset) - identical_count}"
        )
        return filepath


def main():
    random.seed(RANDOM_SEED)
    np.random.seed(RANDOM_SEED)
    
    generator = IdenticalDatasetGenerator(model_name=MODEL_NAME)
    dataset = generator.generate_dataset()
    generator.save_dataset(dataset, output_dir="./identical_dataset")


if __name__ == "__main__":
    main()

Result

Parallel	Identical Request Ratio	Local Cache Size (GB)	Throughput (tkn/s)	TTFT (s)	TPOT (s)	Throughput Delta	TTFT Delta	TPOT Delta
5	50	0	577.89	26.18	0.0520	0.00%	0.00%	0.00%
5	50	1	745.41	17.02	0.0531	28.99%	-35.00%	2.12%
5	50	2	820.39	13.94	0.0543	41.96%	-46.74%	4.42%
5	50	4	889.92	11.73	0.0544	53.99%	-55.20%	4.62%
10	50	0	772.65	42.06	0.0671	0.00%	0.00%	0.00%
10	50	1	956.47	27.94	0.0778	23.79%	-33.58%	15.95%
10	50	2	1037.45	25.33	0.0727	34.27%	-39.78%	8.35%
10	50	4	1105.28	22.66	0.0723	43.05%	-46.12%	7.75%
5	70	0	549.76	28.63	0.0508	0.00%	0.00%	0.00%
5	70	1	1029.84	9.13	0.0511	87.33%	-68.10%	0.59%
5	70	2	1139.46	6.51	0.0530	107.26%	-77.27%	4.33%
5	70	4	1108.51	7.44	0.0514	101.64%	-73.99%	1.18%
10	70	0	761.69	42.41	0.0681	0.00%	0.00%	0.00%
10	70	1	1401.02	14.25	0.0717	83.93%	-66.39%	5.29%
10	70	2	1537.34	13.44	0.0633	101.83%	-68.30%	-7.05%
10	70	4	1502.74	13.66	0.0656	97.29%	-67.78%	-3.67%

Checklist

I have performed a self-review of my own code.
I have updated the documentation.
I have added tests to prove my changes are effective.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1226 **Author:** [@Shichang-Zhang](https://github.com/Shichang-Zhang) **Created:** 12/17/2025 **Status:** ✅ Merged **Merged:** 2/11/2026 **Merged by:** [@stmatengss](https://github.com/stmatengss) **Base:** `main` ← **Head:** `upstream-zsc-working` --- ### 📝 Commits (10+) - [`36ee91b`](https://github.com/kvcache-ai/Mooncake/commit/36ee91b41075276414e64e747200ff4dfaa782dd) feat(Store): add local hot cache for client - [`4158e59`](https://github.com/kvcache-ai/Mooncake/commit/4158e595052daf05c7a6158ca23a3c6a7f23140f) feat(Store): add client local hot cache log to show performance - [`9d4aa0d`](https://github.com/kvcache-ai/Mooncake/commit/9d4aa0d51272547e410bafd7ea01e054e880f4e4) fix: local hot cache initialize bug - [`e9686ed`](https://github.com/kvcache-ai/Mooncake/commit/e9686ede8213f7d7848aaaa3a34f8d9e349a3e18) fix(Store): Mooncake put slice is max 16MB， so make local hot cache block 16MB - [`6453b95`](https://github.com/kvcache-ai/Mooncake/commit/6453b95e2d404880333dd90bdb0b9db83bda41e4) feat(Store): move local hot cache initialization to Client::Create - [`6768f3c`](https://github.com/kvcache-ai/Mooncake/commit/6768f3cdb29b66f9e05e3867662c7efa8081f607) feat(Store): local hot cache remove unused small block implementation - [`8848351`](https://github.com/kvcache-ai/Mooncake/commit/8848351feeb2186119c14042293dd0f2cbd4ce42) feat(Store): add client local hot cache unit test - [`602801e`](https://github.com/kvcache-ai/Mooncake/commit/602801ed19ec77cdb3affb79ea4120569b0b23a7) fix(Store): modify client local hot cache suit with v0.3.7 - [`9d57b16`](https://github.com/kvcache-ai/Mooncake/commit/9d57b16d5d862f2387bc067f10b2204f4a0853d7) feat(Store): change local hot cache unit tes - [`19cca30`](https://github.com/kvcache-ai/Mooncake/commit/19cca306be21f24ae5c512cfc03ebcdab1b358d2) fix: initialize local hot cache with negative value ### 📊 Changes **9 files changed** (+1871 additions, -10 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-integration/store/store_py.cpp` (+0 -1) 📝 `mooncake-store/include/client_service.h` (+65 -0) ➕ `mooncake-store/include/local_hot_cache.h` (+199 -0) 📝 `mooncake-store/src/CMakeLists.txt` (+1 -0) 📝 `mooncake-store/src/client_service.cpp` (+251 -6) ➕ `mooncake-store/src/local_hot_cache.cpp` (+355 -0) 📝 `mooncake-store/src/transfer_task.cpp` (+35 -3) 📝 `mooncake-store/tests/CMakeLists.txt` (+1 -0) ➕ `mooncake-store/tests/client_local_hot_cache_test.cpp` (+964 -0) </details> ### 📄 Description ## Description  This PR introduces the client hot cache feature mentioned in [Issue 1062](https://github.com/kvcache-ai/Mooncake/issues/1062). This feature is enabled by setting the environment variable `LOCAL_HOT_CACHE_SIZE` (in bytes). The mooncake store client will allocate memory blocks for caching hot data on a per-request basis. The block size is configurable via the environment variable `LOCAL_HOT_BLOCK_SIZE` (in bytes), with a default value of 16MB. Memory allocation follows the configured block size unit. If `LOCAL_HOT_CACHE_SIZE` is set to less than the block size (either the default 16MB or the value specified by `LOCAL_HOT_BLOCK_SIZE`), zero, negative, or other invalid inputs (such as non-numeric strings), the client hot cache feature will be disabled. For the standalone client scenario, the feature should be disabled since all things are in local storage. For the architecture of multiple clients and a master, the feature is recommended for use. Users should adjust the LOCAL_HOT_CACHE_SIZE value based on their actual deployment context to strike an optimal balance: the performance gain from reduced cross-node data transfer outweighs the overhead of additional local cache memory. ## Type of Change * Types - [ ] Bug fix - [ ] New feature - [ ] Transfer Engine - [√ ] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [ ] CI/CD - [ ] Documentation update - [ ] Other ## How Has This Been Tested? Run unit test for the feature `mooncake-store/tests/client_local_hot_cache_test.cpp`.  ## Performance Test ### Software | Name | Version | |------|---------| | vLLM | v0.9.1 | | vLLM Ascend | v0.9.1 | | Mooncake | v0.3.7 | | EvalScope | v1.2.0 | | CANN | 8.2RC1 | | Kubernetes | v1.33.1 | ### Hardware | Name | Description | |------|-------------| | Machine | Atlas 800I A2 | | NPU | 8 * 昇腾910B4 | | CPU | 4 * 鲲鹏920 | | Memory | 32G * 32 | | Network | 8 * 200GE QSFP | ### Mooncake Performance Test #### Environment We run the test on a Kubernetes cluster that involves two Atlas 800I A2 machines. #### Method 1. Start master-service pod as Mooncake Master and redis pod as the metadata server. 2. Start a Mooncake Client pod on node 1. 3. Call `MooncakeDistributedStore:Put` interface to store data (e.g., 1GB) on Mooncake Client 1. 3.1 Store 10 keys with corresponding data (e.g., 1GB total) in Mooncake. 3.2 All data slices are stored on node 1 since only one client exists. 4. Start another Mooncake Client pod with local hot cache (e.g., 1GB) enabled on node 2. 5. Call `MooncakeDistributedStore:Get/BatchGet` interface from Mooncake Client 2 to retrieve the stored data. 5.1 All data slices are on node 1, so all slices requested by Mooncake Client 2 will be transferred remotely. 5.2 Use TCP protocol to increase network transmission latency. 5.3 After retrieving data from remote node 1, the slices are stored in the local hot cache. 6. Record the latency of the Get/BatchGet calls. 7. Repeatedly call `MooncakeDistributedStore:Get/BatchGet` interface from Mooncake Client 2 and record the latency. 7.1 These requests result in cache hits. #### Result |Method|Total Key Num|Data Size (MB)|Total Data Size (MB)|Local Cache Size (MB)|Get/BatchGet Per Round|Batch Size|Test Rounds|First Round Latency (ms)|Average Round Latency (ms)|Latency Delta (%)| |---|---|---|---|---|---|---|---|---|---|---| |get|25|10|250|0|25|-|10|767.64|881.02|-| |get|25|10|250|400|25|-|10|1204.43|287.38|76.14| |get|25|25|625|0|25|-|10|757.65|760.36|-| |get|25|25|625|800|25|-|10|905.76|259.48|71.35| |batchGet|25|10|250|0|5|5|10|489.24|463.28|-| |batchGet|25|10|250|400|5|5|10|425.94|151.23|64.49| |batchGet|25|25|625|0|5|5|10|433.15|440.47|-| |batchGet|25|25|625|800|5|5|10|504.91|143.83|71.51| Note: The latency is measured for each key request, for `BatchGet` interface, first round latency is the average latency of first 2 key. ### End to End Inference Performance Test #### Environment We run the test on a Kubernetes cluster that involves three Atlas 800I A2 machines. We deployed the inference service with vLLM v0 and Mooncake. #### Method 1. Start master-service pod as Mooncake Master and redis pod as the metadata server. 2. Start 5 prefill pods and 5 decode pods running vLLM and Mooncake Client. Each client offers 30GB store memory. 2.1 Deploy 10 Mooncake Clients to increase the probability of remote slice data transmission. 2.2 Use TCP protocol and deploy across three machines to increase network transmission latency. 2.3 <u>Prefill instances do not enable the local hot cache feature, while decode instances enable it.</u> 3. Start proxy server pod for routing inference requests 3.1 Use the example proxy server from vLLM. 4. Start EvalScope script 4.1 Send 1000 inference requests at concurrency 5 or 10, with a fraction of exact duplicate requests to simulate hotspot/cache hits. Average input length is 4096 tokens and output length is fixed at 256 tokens. 4.2 Use a random dataset to vary prefix lengths, controlling prefix sharing across requests and thus the number of hot slices (and hit rate). 4.3 EvalScope summarizes and logs key performance metrics such as throughput, latency (p50/p95/p99), and success/error rates. #### Test Script - **vLLM prefill** ``` bash export MOONCAKE_CONFIG_PATH=/app/mooncake_config.json echo "{ \"local_hostname\": \"$POD_IP\", \"metadata_server\": \"redis://redis:6379\", \"master_server_address\": \"mooncake-master:30089\", \"protocol\": \"tcp\", \"device_name\": \"\", \"global_segment_size\": 32212254720 }" > ${MOONCAKE_CONFIG_PATH} VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \ --model Qwen/Qwen3-8B \ --port 8100 \ --max-model-len 10000 \ --gpu-memory-utilization 0.8 \ --block-size 128 \ --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer","kv_buffer_device":"npu"}' ``` Note: This is the start command of the prefill pod. So `POD_IP` is the node address of this pod. - **vLLM decode** ``` bash export MOONCAKE_CONFIG_PATH=/app/mooncake_config.json echo "{ \"local_hostname\": \"$POD_IP\", \"metadata_server\": \"redis://redis:6379\", \"master_server_address\": \"mooncake-master:30089\", \"protocol\": \"tcp\", \"device_name\": \"\", \"global_segment_size\": 32212254720 }" > ${MOONCAKE_CONFIG_PATH} VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \ --model Qwen/Qwen3-8B \ --port 8200 \ --max-model-len 10000 \ --gpu-memory-utilization 0.8 \ --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer","kv_buffer_device":"npu"}' ``` Note: This is the start command of the decode pod. So `POD_IP` is the node address of this pod. - **EvalScope DataSet** ``` python import json import random import os import numpy as np from datetime import datetime from typing import List, Dict from modelscope import AutoTokenizer # --- Core configuration --- MODEL_NAME = "Qwen/Qwen3-8B" REQUEST_LEN = 4096 # Target token length for each request DATASET_SIZE = 1000 # Total number of requests REPEAT_PERCENT = 0.5 # Ratio of identical (repeat) requests (0.0-1.0) # Fixed seeds for reproducibility RANDOM_SEED = 42 class IdenticalDatasetGenerator: """Dataset generator for identical/repeated requests.""" def __init__(self, model_name: str = MODEL_NAME): """Initialize the generator and load the tokenizer.""" self.model_name = model_name print(f"Loading tokenizer from ModelScope: {model_name} ...") self.tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) if self.tokenizer.pad_token is None: self.tokenizer.pad_token = self.tokenizer.eos_token print(f"Tokenizer loaded. Vocab size: {self.tokenizer.vocab_size}") def generate_random_tokens(self, length: int) -> str: """ Generate random text that encodes to the target number of tokens. A decode->encode roundtrip is used to approximate exact token count. Args: length: Target number of tokens. Returns: A text string (ideally encoding to exactly `length` tokens). """ if length <= 0: return "" # Sample token IDs, decode to text, then re-encode to validate the count. token_ids = np.random.choice( self.tokenizer.vocab_size, size=length, replace=True ).tolist() text = self.tokenizer.decode(token_ids, skip_special_tokens=False) encoded_ids = self.tokenizer.encode(text, add_special_tokens=False) actual_length = len(encoded_ids) # If the roundtrip shrinks the token count, append additional tokens. if actual_length < length: needed = length - actual_length additional_ids = np.random.choice( self.tokenizer.vocab_size, size=needed, replace=True ).tolist() additional_text = self.tokenizer.decode(additional_ids, skip_special_tokens=False) text = text + additional_text return text def generate_request(self) -> str: """Generate a single request with REQUEST_LEN tokens.""" return self.generate_random_tokens(REQUEST_LEN) def generate_dataset(self) -> List[Dict]: """ Generate the full dataset. Returns: A list of items with `prompt`, plus metadata fields. """ repeat_count = int(DATASET_SIZE * REPEAT_PERCENT) unique_count = DATASET_SIZE - repeat_count print("Generating identical-request dataset...") print( f"Config: total={DATASET_SIZE}, unique={unique_count}, identical={repeat_count} " f"({REPEAT_PERCENT:.1%}), tokens_per_request={REQUEST_LEN}" ) dataset = [] identical_request = self.generate_request() print(f"Generating {unique_count} unique requests...") unique_requests = [] for i in range(unique_count): if (i + 1) % 100 == 0: print(f"Progress: {i+1}/{unique_count} unique requests generated") unique_request = self.generate_request() unique_requests.append(unique_request) print("Building dataset...") for i, request in enumerate(unique_requests): dataset.append({ "prompt": request, "is_identical": False, "request_id": i, }) for i in range(repeat_count): dataset.append({ "prompt": identical_request, "is_identical": True, "request_id": unique_count + i, }) print("Shuffling dataset...") random.shuffle(dataset) # Reassign request_id after shuffling. for i, item in enumerate(dataset): item["request_id"] = i print("Dataset generation complete.") return dataset def save_dataset(self, dataset: List[Dict], output_dir: str = "."): """ Save the dataset to a JSON file (timestamped for uniqueness). Args: dataset: Dataset items. output_dir: Output directory. """ os.makedirs(output_dir, exist_ok=True) identical_count = sum(1 for item in dataset if item.get("is_identical", False)) timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") filename = f"identical_data_{DATASET_SIZE}total_{REQUEST_LEN}len_{REPEAT_PERCENT*100:.0f}pct_repeat_{timestamp}.json" filepath = os.path.join(output_dir, filename) print(f"Saving dataset to: {filepath}") with open(filepath, 'w', encoding='utf-8') as f: json.dump(dataset, f, ensure_ascii=False, indent=2) file_size_mb = os.path.getsize(filepath) / 1024 / 1024 print( f"Saved. size={file_size_mb:.2f} MB, total={len(dataset)}, " f"identical={identical_count}, unique={len(dataset) - identical_count}" ) return filepath def main(): random.seed(RANDOM_SEED) np.random.seed(RANDOM_SEED) generator = IdenticalDatasetGenerator(model_name=MODEL_NAME) dataset = generator.generate_dataset() generator.save_dataset(dataset, output_dir="./identical_dataset") if __name__ == "__main__": main() ``` #### Result | Parallel | Identical Request Ratio | Local Cache Size (GB) | Throughput (tkn/s) | TTFT (s) | TPOT (s) | Throughput Delta | TTFT Delta | TPOT Delta | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 5 | 50 | 0 | 577.89 | 26.18 | 0.0520 | 0.00% | 0.00% | 0.00% | | 5 | 50 | 1 | 745.41 | 17.02 | 0.0531 | 28.99% | -35.00% | 2.12% | | 5 | 50 | 2 | 820.39 | 13.94 | 0.0543 | 41.96% | -46.74% | 4.42% | | 5 | 50 | 4 | 889.92 | 11.73 | 0.0544 | 53.99% | -55.20% | 4.62% | | 10 | 50 | 0 | 772.65 | 42.06 | 0.0671 | 0.00% | 0.00% | 0.00% | | 10 | 50 | 1 | 956.47 | 27.94 | 0.0778 | 23.79% | -33.58% | 15.95% | | 10 | 50 | 2 | 1037.45 | 25.33 | 0.0727 | 34.27% | -39.78% | 8.35% | | 10 | 50 | 4 | 1105.28 | 22.66 | 0.0723 | 43.05% | -46.12% | 7.75% | | 5 | 70 | 0 | 549.76 | 28.63 | 0.0508 | 0.00% | 0.00% | 0.00% | | 5 | 70 | 1 | 1029.84 | 9.13 | 0.0511 | 87.33% | -68.10% | 0.59% | | 5 | 70 | 2 | 1139.46 | 6.51 | 0.0530 | 107.26% | -77.27% | 4.33% | | 5 | 70 | 4 | 1108.51 | 7.44 | 0.0514 | 101.64% | -73.99% | 1.18% | | 10 | 70 | 0 | 761.69 | 42.41 | 0.0681 | 0.00% | 0.00% | 0.00% | | 10 | 70 | 1 | 1401.02 | 14.25 | 0.0717 | 83.93% | -66.39% | 5.29% | | 10 | 70 | 2 | 1537.34 | 13.44 | 0.0633 | 101.83% | -68.30% | -7.05% | | 10 | 70 | 4 | 1502.74 | 13.66 | 0.0656 | 97.29% | -67.78% | -3.67% | ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:32:41 +00:00