[GH-ISSUE #1520] [Feature Request]: [Store] Add C2C cross-model KV Cache conversion support #467

New issue

Closed

opened 2026-10-29 21:15:02 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:15:02 +00:00

Owner

Originally created by @YiXR on GitHub (Feb 9, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1520

Originally assigned to: @ykwd, @stmatengss, @YiXR, @XucSh on GitHub.

Motivation

In production LLM serving, it is common to deploy multiple model sizes simultaneously (e.g., Qwen3-4B for complex queries and Qwen3-0.6B for simple ones). These models share the same tokenizer and similar transformer architectures, yet each must independently compute
its own KV Cache from scratch during prefill — even when serving the exact same prompt.

https://github.com/thu-nics/C2C addresses this by training lightweight MLP projectors that convert KV Cache between models with minimal quality loss. By integrating C2C at the Mooncake Store level, we can:

Eliminate redundant prefill: A single prefill on one model automatically generates usable KV Cache for all related models
Reduce TTFT (Time To First Token): Smaller models get "free" KV Cache from larger model's prefill, achieving near-zero prefill latency
Maximize cache utilization: Cross-model cache sharing dramatically increases effective cache hit rates in multi-model deployments
Zero framework changes: The conversion is fully transparent — SGLang and other frameworks require no code modifications

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         Mooncake Master                             │
│                                                                     │
│  ┌─────────────┐    C2C Config (models, rules, projector URLs)      │
│  │ c2c_config  │◄── Configured at startup, served via RPC           │
│  └──────┬──────┘                                                    │
│         │ GetC2CConfig()                                            │
└─────────┼───────────────────────────────────────────────────────────┘
          │
          ▼
┌─────────────────────────────────────────────────────────────────────┐
│                        Mooncake Client                              │
│                                                                     │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                    KVAutoConverter                            │   │
│  │                                                              │   │
│  │  1. Auto-download projector weights (HTTP, cached locally)   │   │
│  │  2. Auto-fetch model params from HuggingFace config.json     │   │
│  │  3. Hook batch_put_from → detect model → enqueue conversion  │   │
│  │  4. Worker pool: bf16→float→MLP project→float→bf16           │   │
│  │  5. Auto put_from(converted KV) back to store                │   │
│  │                                                              │   │
│  └──────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Data flow for a single request:

SGLang Instance A (Qwen3-4B)
       │
       │ batch_put_from("Qwen/Qwen3-4B_<hash>_0_k", kv_data)
       ▼
┌──────────────┐     match source_model_name     ┌──────────────────┐
│ Mooncake     │ ──────────────────────────────► │ KVAutoConverter   │
│ Store        │                                  │                  │
│              │     1. Group by (rule, k/v)      │  Worker Threads  │
│              │     2. Batch sgemm projection    │  ┌──┐┌──┐┌──┐   │
│              │     3. bf16 ↔ float (AVX2)       │  │W1││W2││Wn│   │
│              │                                  │  └──┘└──┘└──┘   │
│              │ ◄──────────────────────────────  │                  │
│              │  put_from("Qwen/Qwen3-0.6B_..") └──────────────────┘
│              │
└──────┬───────┘
       │
       │ get("Qwen/Qwen3-0.6B_<hash>_0_k")  →  Cache Hit!
       ▼
SGLang Instance B (Qwen3-0.6B)
       │
       ▼
  Skip prefill, directly decode  →  Near-zero TTFT

Conversion pipeline (per worker):

Source KV (bf16)                          Target KV (bf16)
[pages × layers × heads × dim]           [pages × layers' × heads' × dim']
       │                                          ▲
       ▼                                          │
┌─────────────┐                           ┌──────────────┐
│ bf16→float  │  (CPU Convert)            │ float→bf16   │
└──────┬──────┘                           └──────┬───────┘
       │                                          ▲
       ▼                                          │
┌──────────────────────────────────────────────────┐
│           C2C MLP Projector (per layer)           │
│                                                   │
│  src ──► Linear+GELU ──► Linear+GELU ──► Linear  │
│          [src→hidden]    [hidden→hidden]  [hidden→tgt] │
│                                              × gate│
│          (OpenBLAS sgemm, batch M=total_pages)     │
└───────────────────────────────────────────────────┘

References

https://github.com/thu-nics/C2C
C2C Paper: C2C: Scalable Multi-Model Serving via Cross-Model Prefill and Shared KV Cache (THU-NICS, 2025)

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @YiXR on GitHub (Feb 9, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1520 Originally assigned to: @ykwd, @stmatengss, @YiXR, @XucSh on GitHub. ## Motivation In production LLM serving, it is common to deploy multiple model sizes simultaneously (e.g., Qwen3-4B for complex queries and Qwen3-0.6B for simple ones). These models share the same tokenizer and similar transformer architectures, yet each must independently compute its own KV Cache from scratch during prefill — even when serving the exact same prompt. https://github.com/thu-nics/C2C addresses this by training lightweight MLP projectors that convert KV Cache between models with minimal quality loss. By integrating C2C at the Mooncake Store level, we can: - Eliminate redundant prefill: A single prefill on one model automatically generates usable KV Cache for all related models - Reduce TTFT (Time To First Token): Smaller models get "free" KV Cache from larger model's prefill, achieving near-zero prefill latency - Maximize cache utilization: Cross-model cache sharing dramatically increases effective cache hit rates in multi-model deployments - Zero framework changes: The conversion is fully transparent — SGLang and other frameworks require no code modifications ## Architecture ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Mooncake Master │ │ │ │ ┌─────────────┐ C2C Config (models, rules, projector URLs) │ │ │ c2c_config │◄── Configured at startup, served via RPC │ │ └──────┬──────┘ │ │ │ GetC2CConfig() │ └─────────┼───────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ Mooncake Client │ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ KVAutoConverter │ │ │ │ │ │ │ │ 1. Auto-download projector weights (HTTP, cached locally) │ │ │ │ 2. Auto-fetch model params from HuggingFace config.json │ │ │ │ 3. Hook batch_put_from → detect model → enqueue conversion │ │ │ │ 4. Worker pool: bf16→float→MLP project→float→bf16 │ │ │ │ 5. Auto put_from(converted KV) back to store │ │ │ │ │ │ │ └──────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘ ``` Data flow for a single request: SGLang Instance A (Qwen3-4B) │ │ batch_put_from("Qwen/Qwen3-4B_<hash>_0_k", kv_data) ▼ ┌──────────────┐ match source_model_name ┌──────────────────┐ │ Mooncake │ ──────────────────────────────► │ KVAutoConverter │ │ Store │ │ │ │ │ 1. Group by (rule, k/v) │ Worker Threads │ │ │ 2. Batch sgemm projection │ ┌──┐┌──┐┌──┐ │ │ │ 3. bf16 ↔ float (AVX2) │ │W1││W2││Wn│ │ │ │ │ └──┘└──┘└──┘ │ │ │ ◄────────────────────────────── │ │ │ │ put_from("Qwen/Qwen3-0.6B_..") └──────────────────┘ │ │ └──────┬───────┘ │ │ get("Qwen/Qwen3-0.6B_<hash>_0_k") → Cache Hit! ▼ SGLang Instance B (Qwen3-0.6B) │ ▼ Skip prefill, directly decode → Near-zero TTFT Conversion pipeline (per worker): Source KV (bf16) Target KV (bf16) [pages × layers × heads × dim] [pages × layers' × heads' × dim'] │ ▲ ▼ │ ┌─────────────┐ ┌──────────────┐ │ bf16→float │ (CPU Convert) │ float→bf16 │ └──────┬──────┘ └──────┬───────┘ │ ▲ ▼ │ ┌──────────────────────────────────────────────────┐ │ C2C MLP Projector (per layer) │ │ │ │ src ──► Linear+GELU ──► Linear+GELU ──► Linear │ │ [src→hidden] [hidden→hidden] [hidden→tgt] │ │ × gate│ │ (OpenBLAS sgemm, batch M=total_pages) │ └───────────────────────────────────────────────────┘ ## References - https://github.com/thu-nics/C2C - C2C Paper: C2C: Scalable Multi-Model Serving via Cross-Model Prefill and Shared KV Cache (THU-NICS, 2025) ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:15:02 +00:00