[GH-ISSUE #1860] [HELP] Any standard benches to trigger KVs loaded from mooncake store?

bsakaguchi commented

2026-10-29 21:15:58 +00:00

Owner

Originally created by @weike7 on GitHub (Apr 10, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1860

Hi guys,
I have deployed SGLang PD disaggregation + mooncake store: one node for P, one node for D, one node for mooncake store, and I can see KVs are saved to mooncake store node, but I never see who read KVs from store. Any benches to trigger KVs read from store?

Thanks!

Originally created by @weike7 on GitHub (Apr 10, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1860 Hi guys, I have deployed SGLang PD disaggregation + mooncake store: one node for P, one node for D, one node for mooncake store, and I can see KVs are saved to mooncake store node, but I never see who read KVs from store. Any benches to trigger KVs read from store? Thanks!

bsakaguchi

2026-10-29 21:15:58 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:15:59 +00:00

Author

Owner

@00fish0 commented on GitHub (Apr 10, 2026):

Hi @weike7,

The reason you might not be seeing any read operations from the Mooncake store is likely because the KV caches are still residing in SGLang's local L1 (GPU memory) or L2 (Host memory) caches. When a prefix match is found locally, SGLang will prioritize those over fetching from a remote store.

To manually trigger and verify the loading from Mooncake, you can flush the local caches on the SGLang P-node. This forces SGLang to look for the prefix in the remote Mooncake store for subsequent requests.

You can do this by sending a POST request to the /flush_cache endpoint of the P-node:

curl -X POST http://<p-node-ip>:<port>/flush_cache

@00fish0 commented on GitHub (Apr 10, 2026): Hi @weike7, The reason you might not be seeing any read operations from the Mooncake store is likely because the KV caches are still residing in SGLang's local L1 (GPU memory) or L2 (Host memory) caches. When a prefix match is found locally, SGLang will prioritize those over fetching from a remote store. To manually trigger and verify the loading from Mooncake, you can flush the local caches on the SGLang P-node. This forces SGLang to look for the prefix in the remote Mooncake store for subsequent requests. You can do this by sending a POST request to the /flush_cache endpoint of the P-node: ``` curl -X POST http://<p-node-ip>:<port>/flush_cache ```

bsakaguchi commented

2026-10-29 21:15:59 +00:00

Author

Owner

@weike7 commented on GitHub (Apr 11, 2026):

@00fish0 Thanks. Yes, the KVs are residing in L1 and L2, but I'd like a bench to trigger KV offload to L3 (mooncake store) to get the performance data for SGLang PD + mooncake store (DRAM or CXL memory pool), and compare it with SGLang PD + mooncake (SSD). Any such standard benches to simulate the real use cases to trigger L1/L2 full but must offload KVs to mooncake store?

BTW, I am also trying to understand how TE works, for SGLang PD + mooncake store. My current understanding is: D node never loads KVs from mooncake store node directly, but KVs are always loaded from store by prefill and then RDMA write to D node by TE. Am I right?

@weike7 commented on GitHub (Apr 11, 2026): @00fish0 Thanks. Yes, the KVs are residing in L1 and L2, but I'd like a bench to trigger KV offload to L3 (mooncake store) to get the performance data for SGLang PD + mooncake store (DRAM or CXL memory pool), and compare it with SGLang PD + mooncake (SSD). Any such standard benches to simulate the real use cases to trigger L1/L2 full but must offload KVs to mooncake store? BTW, I am also trying to understand how TE works, for SGLang PD + mooncake store. My current understanding is: D node never loads KVs from mooncake store node directly, but KVs are always loaded from store by prefill and then RDMA write to D node by TE. Am I right?

bsakaguchi commented

2026-10-29 21:16:00 +00:00

Author

Owner

@00fish0 commented on GitHub (Apr 11, 2026):

Hi @weike7, Thanks for bringing up these great questions!

Any such standard benches to simulate the real use cases to trigger L1/L2 full but must offload KVs to mooncake store?

@zhangzuo21 recently ran some tests for the SSD offload feature (PR #1835), which happens to cover this scenario. The test uses SGLang's built-in multiturn script bench_multiturn.py. The idea is to have multiple concurrent clients simulate multi-round conversations, with each round appending previous context. As rounds progress, the KV cache naturally overflows from L1 → L2 → L3 (Mooncake Store).

Here's the command used:

python3 benchmark/hicache/bench_multiturn.py \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 \
    --port 8189 \
    --disable-random-sample \
    --output-length 1 \
    --request-length 4096 \
    --num-clients 20 \
    --num-rounds 10 \
    --max-parallel 4 \
    --request-rate 16 \
    --ready-queue-policy random \
    --disable-auto-run \
    --enable-round-barrier

You can find the full benchmark setup and results in PR #1835.

My current understanding is: D node never loads KVs from mooncake store node directly, but KVs are always loaded from store by prefill and then RDMA write to D node by TE. Am I right?

Yes, your understanding is correct. In SGLang PD mode, the D node does not read KVs from the Mooncake Store directly. The P node loads KV cache from the Store (or computes it if not cached), then RDMA-writes it to the D node via the Transfer Engine.

@00fish0 commented on GitHub (Apr 11, 2026): Hi @weike7, Thanks for bringing up these great questions! > Any such standard benches to simulate the real use cases to trigger L1/L2 full but must offload KVs to mooncake store? @zhangzuo21 recently ran some tests for the SSD offload feature (PR #1835), which happens to cover this scenario. The test uses SGLang's built-in multiturn script [bench_multiturn.py](https://github.com/sgl-project/sglang/blob/main/benchmark/hicache/bench_multiturn.py). The idea is to have multiple concurrent clients simulate multi-round conversations, with each round appending previous context. As rounds progress, the KV cache naturally overflows from L1 → L2 → L3 (Mooncake Store). Here's the command used: ``` python3 benchmark/hicache/bench_multiturn.py \ --model-path $MODEL_PATH \ --host 127.0.0.1 \ --port 8189 \ --disable-random-sample \ --output-length 1 \ --request-length 4096 \ --num-clients 20 \ --num-rounds 10 \ --max-parallel 4 \ --request-rate 16 \ --ready-queue-policy random \ --disable-auto-run \ --enable-round-barrier ``` You can find the full benchmark setup and results in PR #1835. > My current understanding is: D node never loads KVs from mooncake store node directly, but KVs are always loaded from store by prefill and then RDMA write to D node by TE. Am I right? Yes, your understanding is correct. In SGLang PD mode, the D node does not read KVs from the Mooncake Store directly. The P node loads KV cache from the Store (or computes it if not cached), then RDMA-writes it to the D node via the Transfer Engine.

bsakaguchi commented

2026-10-29 21:16:00 +00:00

Author

Owner

@weike7 commented on GitHub (Apr 11, 2026):

@00fish0 Thanks. Actually, I run bench_multiturn.py last week, but I only see KVs offloaded from P node to store node, and I never see KVs loaded from store node. Let me try it again next Monday. BTW, any logs to show KVs loaded from store? I add logs in TE and rebuild it to see data rdma write to store node and D node.

On the other hand, I am thinking about if CXL memory pool is used for L3 cache, any plans to enable D node load KVs from CXL memory pool directly to bypass RDMA.

@weike7 commented on GitHub (Apr 11, 2026): @00fish0 Thanks. Actually, I run bench_multiturn.py last week, but I only see KVs offloaded from P node to store node, and I never see KVs loaded from store node. Let me try it again next Monday. BTW, any logs to show KVs loaded from store? I add logs in TE and rebuild it to see data rdma write to store node and D node. On the other hand, I am thinking about if CXL memory pool is used for L3 cache, any plans to enable D node load KVs from CXL memory pool directly to bypass RDMA.

bsakaguchi commented

2026-10-29 21:16:00 +00:00

Author

Owner

@00fish0 commented on GitHub (Apr 11, 2026):

Hi @weike7, regarding your questions about monitoring and triggering L3 reads:

To make the L3 (Mooncake) read effect more noticeable: You might want to try reducing the L1 (GPU memory) and L2 (Host memory) cache sizes in SGLang. This will cause the local caches to fill up much faster, forcing the system to fetch from the Mooncake remote store more frequently during your benchmarks.
How to verify KV reads from the logs:

Master Node Logs: The Master node prints the total number of Get requests every 10 seconds. If you see this count increasing, it confirms that read operations are being issued to Mooncake.
SGLang P-node Logs: You can look for the batch_get_into log entries in the P-node output. It should look something like this:

I20260404 16:15:33.901468 107973 real_client.cpp:2149] Time taken for batch_get_into: xxxus, ...

These two points should help you confirm whether the data path from Mooncake is being active during your bench_multiturn.py runs.

@00fish0 commented on GitHub (Apr 11, 2026): Hi @weike7, regarding your questions about monitoring and triggering L3 reads: 1. To make the L3 (Mooncake) read effect more noticeable: You might want to try reducing the L1 (GPU memory) and L2 (Host memory) cache sizes in SGLang. This will cause the local caches to fill up much faster, forcing the system to fetch from the Mooncake remote store more frequently during your benchmarks. 2. How to verify KV reads from the logs: - Master Node Logs: The Master node prints the total number of Get requests every 10 seconds. If you see this count increasing, it confirms that read operations are being issued to Mooncake. - SGLang P-node Logs: You can look for the batch_get_into log entries in the P-node output. It should look something like this: ``` I20260404 16:15:33.901468 107973 real_client.cpp:2149] Time taken for batch_get_into: xxxus, ... ``` These two points should help you confirm whether the data path from Mooncake is being active during your bench_multiturn.py runs.

bsakaguchi commented

2026-10-29 21:16:00 +00:00

Author

Owner

@weike7 commented on GitHub (Apr 13, 2026):

@00fish0 Thanks, I just test again, this time it should work:
P node logs:
I0413 03:02:16.743196 1253 real_client.cpp:2111] Time taken for batch_get_into: 2183us, read store: 0us, with memory key count: 256, offload key count: 0
I0413 03:02:16.745728 1253 real_client.cpp:2111] Time taken for batch_get_into: 2190us, read store: 0us, with memory key count: 256, offload key count: 0
I0413 03:02:16.748294 1253 real_client.cpp:2111] Time taken for batch_get_into: 2229us, read store: 0us, with memory key count: 256, offload key count: 0

Mooncake master/meta node log:
I0413 03:03:02.095603 106 rpc_service.cpp:41] Master Metrics: Mem Storage: 54.39 GB / 256.00 GB (21.2%) | SSD Storage: 0 B / 0 B | Keys: 712849 (soft-pinned: 0) | Clients: 2 | Requests (Success/Total): PutStart=1/1, PutEnd=1/1, PutRevoke=0/0, Get=1/1, Exist=1/1, Del=0/0, DelAll=0/0, Ping=1070/1070, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0, EvictDiskReplica=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=2853/0/2853, Item=712848/712848), PutEnd:(Req=2853/0/2853, Item=712848/712848), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=2325/0/2325, Item=589892/589892), ExistKey:(Req=8004/0/8004, Item=2031626/2031626), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=1070/1070), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0

Above log, I see 'Get:(Req=2325/0/2325, Item=589892/589892)' on Master/meta node.
Several questions:

On P node: 'offload key count: 0', it is always '0'. Is it correct?
For the testing case, it sets output length to 0, it is not the real word cases. For the production 'SGLang P/D disaggregation + mooncake store' deployment, how can we measure it benefit to $, such as support more clients, or save cost by memory pool or SSD but get the similar performance, etc?
Any plans to support D node load KVs from store directly? Current KV flow is: P -> Store -> P -> D, the optimized one should be: P -> Store -> D.

@weike7 commented on GitHub (Apr 13, 2026): @00fish0 Thanks, I just test again, this time it should work: P node logs: I0413 03:02:16.743196 1253 real_client.cpp:2111] Time taken for batch_get_into: 2183us, read store: 0us, with memory key count: 256, offload key count: 0 I0413 03:02:16.745728 1253 real_client.cpp:2111] Time taken for batch_get_into: 2190us, read store: 0us, with memory key count: 256, offload key count: 0 I0413 03:02:16.748294 1253 real_client.cpp:2111] Time taken for batch_get_into: 2229us, read store: 0us, with memory key count: 256, offload key count: 0 Mooncake master/meta node log: I0413 03:03:02.095603 106 rpc_service.cpp:41] Master Metrics: Mem Storage: 54.39 GB / 256.00 GB (21.2%) | SSD Storage: 0 B / 0 B | Keys: 712849 (soft-pinned: 0) | Clients: 2 | Requests (Success/Total): PutStart=1/1, PutEnd=1/1, PutRevoke=0/0, Get=1/1, Exist=1/1, Del=0/0, DelAll=0/0, Ping=1070/1070, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0, EvictDiskReplica=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=2853/0/2853, Item=712848/712848), PutEnd:(Req=2853/0/2853, Item=712848/712848), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=2325/0/2325, Item=589892/589892), ExistKey:(Req=8004/0/8004, Item=2031626/2031626), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=1070/1070), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0 Above log, I see 'Get:(Req=2325/0/2325, Item=589892/589892)' on Master/meta node. Several questions: 1. On P node: 'offload key count: 0', it is always '0'. Is it correct? 2. For the testing case, it sets output length to 0, it is not the real word cases. For the production 'SGLang P/D disaggregation + mooncake store' deployment, how can we measure it benefit to $$$, such as support more clients, or save cost by memory pool or SSD but get the similar performance, etc? 3. Any plans to support D node load KVs from store directly? Current KV flow is: P -> Store -> P -> D, the optimized one should be: P -> Store -> D.

bsakaguchi commented

2026-10-29 21:16:00 +00:00

Author

Owner

@ykwd commented on GitHub (Apr 13, 2026):

how can we measure its benefit in $, such as supporting more clients, or saving cost via memory pool or SSD while maintaining similar performance?

One approach is to estimate the ideal KVCache hit rate for your workload (e.g., assuming all reusable cache is successfully reused), and compare it with the actual KVCache hit rate. If there is a significant gap, you can try increasing the KVCache pool size.

Any plans to support D node loading KVs from the store directly? Current KV flow is: P -> Store -> P -> D, the optimized one should be: P -> Store -> D.

As far as I know, there are currently no such plans on the SGLang side.

@ykwd commented on GitHub (Apr 13, 2026): > how can we measure its benefit in $$$, such as supporting more clients, or saving cost via memory pool or SSD while maintaining similar performance? One approach is to estimate the *ideal* KVCache hit rate for your workload (e.g., assuming all reusable cache is successfully reused), and compare it with the *actual* KVCache hit rate. If there is a significant gap, you can try increasing the KVCache pool size. > Any plans to support D node loading KVs from the store directly? Current KV flow is: P -> Store -> P -> D, the optimized one should be: P -> Store -> D. As far as I know, there are currently no such plans on the SGLang side.

bsakaguchi commented

2026-10-29 21:16:00 +00:00

Author

Owner

@00fish0 commented on GitHub (Apr 13, 2026):

On P node: 'offload key count: 0', it is always '0'. Is it correct?

Looking at your Master node metrics: SSD Storage: 0 B / 0 B, it indicates that the SSD offloading feature is either not enabled or has no disk space allocated.

The offload key count will only show non-zero values when the KV cache has been evicted to and then reloaded from the SSD. So, what you are seeing is perfectly normal for an all-in-memory setup.

@00fish0 commented on GitHub (Apr 13, 2026): > On P node: 'offload key count: 0', it is always '0'. Is it correct? Looking at your Master node metrics: SSD Storage: 0 B / 0 B, it indicates that the SSD offloading feature is either not enabled or has no disk space allocated. The offload key count will only show non-zero values when the KV cache has been evicted to and then reloaded from the SSD. So, what you are seeing is perfectly normal for an all-in-memory setup.

bsakaguchi commented

2026-10-29 21:16:00 +00:00

Author

Owner

@ChuanhongLi commented on GitHub (May 26, 2026):

@00fish0 你好，麻烦问下，SGLang PD disaggregation + mooncake store（SSD）分离模式下，P做卸载，D也开启卸载：

MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/home/mooncake-ssd
P 节点：
--enable-hierarchical-cache
--hicache-ratio 2
--page-size xx
--hicache-io-backend xx
--hicache-storage-backend mooncake
--hicache-mem-layout xx
--hicache-write-policy xx
--hicache-storage-prefetch-policy xx
--disaggregation-mode prefill
--disaggregation-bootstrap-port 35000
--disaggregation-ib-device "$IB_DEVICES"
--disaggregation-transfer-backend mooncake

D 节点：
--hicache-ratio 2
--page-size xx
--hicache-io-backend xx
--hicache-storage-backend mooncake
--hicache-mem-layout xx
--hicache-write-policy xx
--hicache-storage-prefetch-policy xx
--disaggregation-decode-enable-offload-kvcache
--disaggregation-mode decode
--disaggregation-ib-device "$IB_DEVICES"
--disaggregation-transfer-backend mooncake \

这种情况下，P做完prefill 把kv cache卸载到SSD中，D节点处理时，也是会卸载到和P节点同一个位置吗？比如这里都是 /home/mooncake-ssd？具体卸载流程是怎么样？可以帮忙解答下嘛？谢谢啦！

@ChuanhongLi commented on GitHub (May 26, 2026): @00fish0 你好，麻烦问下，SGLang PD disaggregation + mooncake store（SSD）分离模式下，P做卸载，D也开启卸载： MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/home/mooncake-ssd P 节点： --enable-hierarchical-cache \ --hicache-ratio 2\ --page-size xx \ --hicache-io-backend xx\ --hicache-storage-backend mooncake \ --hicache-mem-layout xx \ --hicache-write-policy xx\ --hicache-storage-prefetch-policy xx\ --disaggregation-mode prefill \ --disaggregation-bootstrap-port 35000 \ --disaggregation-ib-device "$IB_DEVICES" \ --disaggregation-transfer-backend mooncake D 节点： --hicache-ratio 2 \ --page-size xx \ --hicache-io-backend xx\ --hicache-storage-backend mooncake \ --hicache-mem-layout xx\ --hicache-write-policy xx\ --hicache-storage-prefetch-policy xx\ --disaggregation-decode-enable-offload-kvcache \ --disaggregation-mode decode \ --disaggregation-ib-device "$IB_DEVICES" \ --disaggregation-transfer-backend mooncake \ 这种情况下，P做完prefill 把kv cache卸载到SSD中，D节点处理时，也是会卸载到和P节点同一个位置吗？比如这里都是 /home/mooncake-ssd？具体卸载流程是怎么样？可以帮忙解答下嘛？谢谢啦！

bsakaguchi commented

2026-10-29 21:16:00 +00:00

Author

Owner

@LujhCoconut commented on GitHub (May 26, 2026):

@ChuanhongLi Mooncake SSD offloading现在最主要的是卸载到local SSD。所以不管是P还是D，都是卸载到自己节点的SSD。

@LujhCoconut commented on GitHub (May 26, 2026): @ChuanhongLi Mooncake SSD offloading现在最主要的是卸载到local SSD。所以不管是P还是D，都是卸载到自己节点的SSD。

bsakaguchi commented

2026-10-29 21:16:00 +00:00

Author

Owner

@ChuanhongLi commented on GitHub (May 27, 2026):

@ChuanhongLi Mooncake SSD offloading现在最主要的是卸载到local SSD。所以不管是P还是D，都是卸载到自己节点的SSD。

@LujhCoconut 感谢！还有点疑问，如果P和D分别部署在不同的节点上，在多轮对话场景，得如何复用cache呢？特别是D新生成token ，下一轮的对话时，P节点如何找到它呢？

然后，还有个问题，如果是都卸载到local SSD，是需要在P节点和D节点上分别起个client 去连接master嘛？还是需要如何配置呢？大佬能给个使用SGLang 部署的具体例子不（P和D分别在不同的节点，然后通过Mooncake SSD卸载），对这块还是有点模糊。感谢！

@ChuanhongLi commented on GitHub (May 27, 2026): > [@ChuanhongLi](https://github.com/ChuanhongLi) Mooncake SSD offloading现在最主要的是卸载到local SSD。所以不管是P还是D，都是卸载到自己节点的SSD。 @LujhCoconut 感谢！还有点疑问，如果P和D分别部署在不同的节点上，在多轮对话场景，得如何复用cache呢？特别是D新生成token ，下一轮的对话时，P节点如何找到它呢？然后，还有个问题，如果是都卸载到local SSD，是需要在P节点和D节点上分别起个client 去连接master嘛？还是需要如何配置呢？大佬能给个使用SGLang 部署的具体例子不（P和D分别在不同的节点，然后通过Mooncake SSD卸载），对这块还是有点模糊。感谢！

bsakaguchi commented

2026-10-29 21:16:00 +00:00

Author

Owner

@00fish0 commented on GitHub (May 27, 2026):

@ChuanhongLi

如果P和D分别部署在不同的节点上，在多轮对话场景，得如何复用cache呢？特别是D新生成token ，下一轮的对话时，P节点如何找到它呢？

关于多轮复用机制：

Mooncake 是个 KV 存储，key 是 sglang 对 token 序列做的链式 hash（每 --page-size 个 token 算一段，把上一段的 hash 作为输入，所以前缀相同的 key 一定相同）。这就解决了 "P 如何找到 D 生成的 token" 的问题：

D 开 --disaggregation-decode-enable-offload-kvcache ，新生成的 token 会被异步 offload ，随机选一个client写入 Mooncake Store
下一轮 P 处理同一会话时，对完整 prefix 重算 page hash，去 Mooncake master 查位置；master 返回该 key 所在的 segment（可能是 D 节点，也可能是其他节点；位置可能在 SSD，也可能在内存）
P 通过 Mooncake 的 Transfer Engine（RDMA 或 TCP，由 MOONCAKE_PROTOCOL 决定），跨节点把数据拉回来

如果想同时利用到P和D节点的SSD，需要在 P 和 D 节点上都单独启动 mooncake client（或者让 sglang 进程自己起 client，通过 MOONCAKE_GLOBAL_SEGMENT_SIZE > 0 贡献内存段），都连到同一个 master。两个节点的 client 的 SSD 路径各自配自己节点的SSD 路径。

@00fish0 commented on GitHub (May 27, 2026): @ChuanhongLi > 如果P和D分别部署在不同的节点上，在多轮对话场景，得如何复用cache呢？特别是D新生成token ，下一轮的对话时，P节点如何找到它呢？关于多轮复用机制： Mooncake 是个 KV 存储，key 是 sglang 对 token 序列做的链式 hash（每 `--page-size` 个 token 算一段，把上一段的 hash 作为输入，所以前缀相同的 key 一定相同）。这就解决了 "P 如何找到 D 生成的 token" 的问题： 1. D 开 `--disaggregation-decode-enable-offload-kvcache` ，新生成的 token 会被异步 offload ，随机选一个client写入 Mooncake Store 2. 下一轮 P 处理同一会话时，对完整 prefix 重算 page hash，去 Mooncake master 查位置；master 返回该 key 所在的 segment（可能是 D 节点，也可能是其他节点；位置可能在 SSD，也可能在内存） 3. P 通过 Mooncake 的 Transfer Engine（RDMA 或 TCP，由 `MOONCAKE_PROTOCOL` 决定），跨节点把数据拉回来如果想同时利用到P和D节点的SSD，需要在 P 和 D 节点上都单独启动 mooncake client（或者让 sglang 进程自己起 client，通过 `MOONCAKE_GLOBAL_SEGMENT_SIZE > 0` 贡献内存段），都连到同一个 master。两个节点的 client 的 SSD 路径各自配自己节点的SSD 路径。

bsakaguchi commented

2026-10-29 21:16:00 +00:00

Author

Owner

@ChuanhongLi commented on GitHub (May 27, 2026):

@ChuanhongLi

如果P和D分别部署在不同的节点上，在多轮对话场景，得如何复用cache呢？特别是D新生成token ，下一轮的对话时，P节点如何找到它呢？

关于多轮复用机制：

Mooncake 是个 KV 存储，key 是 sglang 对 token 序列做的链式 hash（每 --page-size 个 token 算一段，把上一段的 hash 作为输入，所以前缀相同的 key 一定相同）。这就解决了 "P 如何找到 D 生成的 token" 的问题：

D 开 --disaggregation-decode-enable-offload-kvcache ，新生成的 token 会被异步 offload ，随机选一个client写入 Mooncake Store

下一轮 P 处理同一会话时，对完整 prefix 重算 page hash，去 Mooncake master 查位置；master 返回该 key 所在的 segment（可能是 D 节点，也可能是其他节点；位置可能在 SSD，也可能在内存）

P 通过 Mooncake 的 Transfer Engine（RDMA 或 TCP，由 MOONCAKE_PROTOCOL 决定），跨节点把数据拉回来

如果想同时利用到P和D节点的SSD，需要在 P 和 D 节点上都单独启动 mooncake client（或者让 sglang 进程自己起 client，通过 MOONCAKE_GLOBAL_SEGMENT_SIZE > 0 贡献内存段），都连到同一个 master。两个节点的 client 的 SSD 路径各自配自己节点的SSD 路径。

@ykwd 感谢解答！
1）D 开 --disaggregation-decode-enable-offload-kvcache ，新生成的 token 会被异步 offload ，随机选一个client写入 Mooncake Store
-- "随机选择一个client写入" 是指如果我D侧没有单独启动client，D 节点会随机选择已有的client写入？假设在1P1D场景，我只在P上启动了client，D卸载时，会写到P这侧的Mooncake Store吗？

2）master 返回该 key 所在的 segment（可能是 D 节点，也可能是其他节点；位置可能在 SSD，也可能在内存）
-- ”也可能在内存“ 这个地方是指Mooncake Store吗？如果是在其他的节点的内存，P也可以直接跨节点把数据拉来吗？

@ChuanhongLi commented on GitHub (May 27, 2026): > [@ChuanhongLi](https://github.com/ChuanhongLi) > > > 如果P和D分别部署在不同的节点上，在多轮对话场景，得如何复用cache呢？特别是D新生成token ，下一轮的对话时，P节点如何找到它呢？ > > 关于多轮复用机制： > > Mooncake 是个 KV 存储，key 是 sglang 对 token 序列做的链式 hash（每 `--page-size` 个 token 算一段，把上一段的 hash 作为输入，所以前缀相同的 key 一定相同）。这就解决了 "P 如何找到 D 生成的 token" 的问题： > > 1. D 开 `--disaggregation-decode-enable-offload-kvcache` ，新生成的 token 会被异步 offload ，随机选一个client写入 Mooncake Store > 2. 下一轮 P 处理同一会话时，对完整 prefix 重算 page hash，去 Mooncake master 查位置；master 返回该 key 所在的 segment（可能是 D 节点，也可能是其他节点；位置可能在 SSD，也可能在内存） > 3. P 通过 Mooncake 的 Transfer Engine（RDMA 或 TCP，由 `MOONCAKE_PROTOCOL` 决定），跨节点把数据拉回来 > > 如果想同时利用到P和D节点的SSD，需要在 P 和 D 节点上都单独启动 mooncake client（或者让 sglang 进程自己起 client，通过 `MOONCAKE_GLOBAL_SEGMENT_SIZE > 0` 贡献内存段），都连到同一个 master。两个节点的 client 的 SSD 路径各自配自己节点的SSD 路径。 @ykwd 感谢解答！ 1）D 开 `--disaggregation-decode-enable-offload-kvcache` ，新生成的 token 会被异步 offload ，随机选一个client写入 Mooncake Store -- "随机选择一个client写入" 是指如果我D侧没有单独启动client，D 节点会随机选择已有的client写入？假设在1P1D场景，我只在P上启动了client，D卸载时，会写到P这侧的Mooncake Store吗？ 2）master 返回该 key 所在的 segment（可能是 D 节点，也可能是其他节点；位置可能在 SSD，也可能在内存） -- ”也可能在内存“ 这个地方是指Mooncake Store吗？如果是在其他的节点的内存，P也可以直接跨节点把数据拉来吗？

bsakaguchi commented

2026-10-29 21:16:01 +00:00

Author

Owner

@00fish0 commented on GitHub (May 27, 2026):

@ChuanhongLi 您好：

-- "随机选择一个client写入" 是指如果我D侧没有单独启动client，D 节点会随机选择已有的client写入？假设在1P1D场景，我只在P上启动了client，D卸载时，会写到P这侧的Mooncake Store吗？

会写到。

”随机选择一个client写入“ 是指会随机选择已有的client写入内存。不管d有没有单独启动client，都是随机选择一个已有的client写入。所以 mooncake 目前默认参数下的行为是：

如果只在1个p上启动client，sglang开启了 --disaggregation-decode-enable-offload-kvcache，就会把所有kvcache写到P这侧的Mooncake Store。
如果1p1d, p和d各启动了一个单独的mooncake client，默认的randomAllocationStrategy，就会以各50%的概率往这两个client对应的内存segment写kvcache。

而SSD功能目前是：对每个节点的mooncake client, 有后台线程把内存的数据不断备份到这个节点的SSD，来实现扩大内存池的效果。所以节点A的mooncake client只能使用节点A的ssd扩大内存池，不能利用到节点B的ssd.

-- ”也可能在内存“ 这个地方是指Mooncake Store吗？如果是在其他的节点的内存，P也可以直接跨节点把数据拉来吗？

对的，内存是指mooncake store注册的内存段。一个完整的过程是：

p的sglang会调用mooncake的batch_get_into接口，先往master发rpc请求，查对应数据具体在哪个segment，再通过TE发起RDMA/TCP传输，把本地节点/远端节点的kvcache数据拉到sglang hicache的L2内存池里。

使用SGLang 部署的具体例子（P和D分别在不同的节点，然后通过Mooncake SSD卸载）

关于SSD+PD分离+SGLang部署的实际例子，非常抱歉，我们目前似乎也没有。有一个单机SGLang+Mooncake，开启SSD的benchmark文档，也许可以参考SSD相关参数的设置：https://kvcache-ai.github.io/Mooncake/performance/ssd-offload-benchmark-results.html

具体的各个参数，可能还是需要根据实际部署环境调优。

cc: @zhangzuo21 @LujhCoconut

@00fish0 commented on GitHub (May 27, 2026): @ChuanhongLi 您好： > -- "随机选择一个client写入" 是指如果我D侧没有单独启动client，D 节点会随机选择已有的client写入？假设在1P1D场景，我只在P上启动了client，D卸载时，会写到P这侧的Mooncake Store吗？会写到。 ”随机选择一个client写入“ 是指会随机选择已有的client写入内存。不管d有没有单独启动client，都是随机选择一个已有的client写入。所以 mooncake 目前默认参数下的行为是： - 如果只在1个p上启动client，sglang开启了 `--disaggregation-decode-enable-offload-kvcache`，就会把所有kvcache写到P这侧的Mooncake Store。 - 如果1p1d, p和d各启动了一个单独的mooncake client，默认的randomAllocationStrategy，就会以各50%的概率往这两个client对应的内存segment写kvcache。而SSD功能目前是：对每个节点的mooncake client, 有后台线程把内存的数据不断备份到这个节点的SSD，来实现扩大内存池的效果。所以节点A的mooncake client只能使用节点A的ssd扩大内存池，不能利用到节点B的ssd. > -- ”也可能在内存“ 这个地方是指Mooncake Store吗？如果是在其他的节点的内存，P也可以直接跨节点把数据拉来吗？对的，内存是指mooncake store注册的内存段。一个完整的过程是： p的sglang会调用mooncake的batch_get_into接口，先往master发rpc请求，查对应数据具体在哪个segment，再通过TE发起RDMA/TCP传输，把本地节点/远端节点的kvcache数据拉到sglang hicache的L2内存池里。 > 使用SGLang 部署的具体例子（P和D分别在不同的节点，然后通过Mooncake SSD卸载）关于SSD+PD分离+SGLang部署的实际例子，非常抱歉，我们目前似乎也没有。有一个单机SGLang+Mooncake，开启SSD的benchmark文档，也许可以参考SSD相关参数的设置：https://kvcache-ai.github.io/Mooncake/performance/ssd-offload-benchmark-results.html 具体的各个参数，可能还是需要根据实际部署环境调优。 cc: @zhangzuo21 @LujhCoconut

bsakaguchi commented

2026-10-29 21:16:01 +00:00

Author

Owner

@ChuanhongLi commented on GitHub (May 28, 2026):

@ChuanhongLi 您好：

-- "随机选择一个client写入" 是指如果我D侧没有单独启动client，D 节点会随机选择已有的client写入？假设在1P1D场景，我只在P上启动了client，D卸载时，会写到P这侧的Mooncake Store吗？

会写到。

”随机选择一个client写入“ 是指会随机选择已有的client写入内存。不管d有没有单独启动client，都是随机选择一个已有的client写入。所以 mooncake 目前默认参数下的行为是：

如果只在1个p上启动client，sglang开启了 --disaggregation-decode-enable-offload-kvcache，就会把所有kvcache写到P这侧的Mooncake Store。

如果1p1d, p和d各启动了一个单独的mooncake client，默认的randomAllocationStrategy，就会以各50%的概率往这两个client对应的内存segment写kvcache。

而SSD功能目前是：对每个节点的mooncake client, 有后台线程把内存的数据不断备份到这个节点的SSD，来实现扩大内存池的效果。所以节点A的mooncake client只能使用节点A的ssd扩大内存池，不能利用到节点B的ssd.

-- ”也可能在内存“ 这个地方是指Mooncake Store吗？如果是在其他的节点的内存，P也可以直接跨节点把数据拉来吗？

对的，内存是指mooncake store注册的内存段。一个完整的过程是：

p的sglang会调用mooncake的batch_get_into接口，先往master发rpc请求，查对应数据具体在哪个segment，再通过TE发起RDMA/TCP传输，把本地节点/远端节点的kvcache数据拉到sglang hicache的L2内存池里。

使用SGLang 部署的具体例子（P和D分别在不同的节点，然后通过Mooncake SSD卸载）

关于SSD+PD分离+SGLang部署的实际例子，非常抱歉，我们目前似乎也没有。有一个单机SGLang+Mooncake，开启SSD的benchmark文档，也许可以参考SSD相关参数的设置：https://kvcache-ai.github.io/Mooncake/performance/ssd-offload-benchmark-results.html

具体的各个参数，可能还是需要根据实际部署环境调优。

cc: @zhangzuo21 @LujhCoconut

@00fish0 感谢回复！我们采用1P1D部署，D 开 --disaggregation-decode-enable-offload-kvcache 后，在多轮对话时，P好像并没有拿到D卸载的kv cache，请问下有什么手段或者日志信息能看到D确实是写成功了吗？或者如何能够跟踪下D写入到具体位置嘛？

@ChuanhongLi commented on GitHub (May 28, 2026): > [@ChuanhongLi](https://github.com/ChuanhongLi) 您好： > > > -- "随机选择一个client写入" 是指如果我D侧没有单独启动client，D 节点会随机选择已有的client写入？假设在1P1D场景，我只在P上启动了client，D卸载时，会写到P这侧的Mooncake Store吗？ > > 会写到。 > > ”随机选择一个client写入“ 是指会随机选择已有的client写入内存。不管d有没有单独启动client，都是随机选择一个已有的client写入。所以 mooncake 目前默认参数下的行为是： > > * 如果只在1个p上启动client，sglang开启了 `--disaggregation-decode-enable-offload-kvcache`，就会把所有kvcache写到P这侧的Mooncake Store。 > * 如果1p1d, p和d各启动了一个单独的mooncake client，默认的randomAllocationStrategy，就会以各50%的概率往这两个client对应的内存segment写kvcache。 > > 而SSD功能目前是：对每个节点的mooncake client, 有后台线程把内存的数据不断备份到这个节点的SSD，来实现扩大内存池的效果。所以节点A的mooncake client只能使用节点A的ssd扩大内存池，不能利用到节点B的ssd. > > > -- ”也可能在内存“ 这个地方是指Mooncake Store吗？如果是在其他的节点的内存，P也可以直接跨节点把数据拉来吗？ > > 对的，内存是指mooncake store注册的内存段。一个完整的过程是： > > p的sglang会调用mooncake的batch_get_into接口，先往master发rpc请求，查对应数据具体在哪个segment，再通过TE发起RDMA/TCP传输，把本地节点/远端节点的kvcache数据拉到sglang hicache的L2内存池里。 > > > 使用SGLang 部署的具体例子（P和D分别在不同的节点，然后通过Mooncake SSD卸载） > > 关于SSD+PD分离+SGLang部署的实际例子，非常抱歉，我们目前似乎也没有。有一个单机SGLang+Mooncake，开启SSD的benchmark文档，也许可以参考SSD相关参数的设置：https://kvcache-ai.github.io/Mooncake/performance/ssd-offload-benchmark-results.html > > 具体的各个参数，可能还是需要根据实际部署环境调优。 > > cc: [@zhangzuo21](https://github.com/zhangzuo21) [@LujhCoconut](https://github.com/LujhCoconut) @00fish0 感谢回复！我们采用1P1D部署，D 开 --disaggregation-decode-enable-offload-kvcache 后，在多轮对话时，P好像并没有拿到D卸载的kv cache，请问下有什么手段或者日志信息能看到D确实是写成功了吗？或者如何能够跟踪下D写入到具体位置嘛？

bsakaguchi commented

2026-10-29 21:16:01 +00:00

Author

Owner

@00fish0 commented on GitHub (May 28, 2026):

@ChuanhongLi

可以先检查一下d是否成功写入了Mooncake：查看master每10s打印一次的metric

I0512 15:03:40.322190 239489 rpc_service.cpp:269] Master Admin Metrics: role=leader, state=serving, summary=role=leader, state=serving, service_ready=true, master={Mem Storage: 0 B / 0 B | SSD Storage: 0 B / 0 B | Keys: 0 (soft-pinned: 0) | Clients: 0 | ...

这里的Mem Storage和SSD Storage 如果有增长就可以说明有成功写入，keys代表现在存储的key数量。

同时也可以看p和d的sglang日志，有没有明显的报错

@00fish0 commented on GitHub (May 28, 2026): @ChuanhongLi 可以先检查一下d是否成功写入了Mooncake：查看master每10s打印一次的metric ``` I0512 15:03:40.322190 239489 rpc_service.cpp:269] Master Admin Metrics: role=leader, state=serving, summary=role=leader, state=serving, service_ready=true, master={Mem Storage: 0 B / 0 B | SSD Storage: 0 B / 0 B | Keys: 0 (soft-pinned: 0) | Clients: 0 | ... ``` 这里的Mem Storage和SSD Storage 如果有增长就可以说明有成功写入，keys代表现在存储的key数量。同时也可以看p和d的sglang日志，有没有明显的报错

bsakaguchi commented

2026-10-29 21:16:01 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 27, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Aug 27, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:16:01 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 4, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Sep 4, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:29:01 +00:00

[PR #530] [MERGED] feat(master): support rpc server address parameter #1628

Rows
Columns

[GH-ISSUE #1860] [HELP] Any standard benches to trigger KVs loaded from mooncake store? #530