[GH-ISSUE #2117] [RFC]: Hierarchical Arch for Intra/Inter Data Center Deployment

bsakaguchi commented

2026-10-29 21:16:46 +00:00

Owner

Originally created by @zhaoyongke on GitHub (May 16, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2117

Originally assigned to: @chestnut-Q on GitHub.

Changes proposed

In AntGroup's practice with Theta KVPool in agentic coding services, where a single LLM service reaches 10,000 card(accordingly 10,000 dummy clients), Mooncake Master becomes an obvious bottleneck. It will fail with these issues:

Allocators management is too heavy;
RPC processing will exhaust the CPU threads;
The recovery time for HA is not acceptable;

Even with HA mode ON, multiple Masters are available, but only one Master instance is active for all clients' requests.

When large mounts of online requests arrived, the former Leader Master instance may crash, the new Leader Master instance is activated, but may crash soon after oplog/snapshot recovery with historical failed and newly arrived requests, which called "avalanche" or "snowslide" effect.

We propose a Hierarchical KVCache Management(HKVM for simple in the remainder part) Architecture for large scale Mooncake Store deployment.

The picture shows a cluster with N sub-cluster and M nodes(we suggest only 1 real client in a node, dummy clients are 8 or 16 times of real clients with different card type) in each sub-cluster, the total nodes are M*N=C. In a sub-cluster, the Sub Master and Clients are the same with original deployment. The only change is a higher level Global Master for global KVCache management.

In HKVM, a Global Master is tailored from original Master. Its functions are:

Registry of Sub Masters;
KVEvent collection from Sub Master;
Act as a Proxy for cross-sub-cluster transferring;
Weak Consistency with Sub Masters(in contrast to Sub Master, which is Strong Consistency within sub-cluster);

We can adjust M and N for performance & reliability trade-off. Specially, when M=1, it will behave like the new P2P Arch; when N=1, it will degenerate into original centralized arch. In our practice, we set M to the scale of P/D group(such as 5 in 2P3D, or 2 in 1P1D), and N to number of P/D groups.

If we add a new level, HKVM naturally supports PrfaaS.

Prefill Service(in the left) and P/D Service(in the right) can run on different clusters and heterogeneous hardwares (H200 + H20 in the paper). In each cluster, we create a global kvcache pool. Prefill Service will generate KVCache of long-input-requests continuously. P/D Service will processing normal requests in parallel. The newly added Inter-DC Master collects KVEvents of both sides, checks the diff and launchs incremental updates via WAN(cross-datacenter) or LAN(intra-datacenter). Thanks to the global kvcache pool, KVCaches transferred can be reused in the future before evicted. We need set the pool sizes in each side according to Prefill Service's and P/D Service's max throughput.
Thanks again to the global kvcache pool, we don't need high-bandwidth network between these clusters, the transfer time is relaxed to KVCache eviction period(from minutes to hours in our real environment).

Our plan for next step:

Global Master implementation, this month;
Optimized real clients implementation, next month;
PrfaaS PoC, 26H2;

Some concerns:

Adding a new level in HKVM may cause an increase in latency, especially in batchIsExist();
Weak consistency may lead to consumer side getting an evicted value in provider side, value integrity check is needed;
Cross-datacenter transferring maybe unreliable, FEC/ECC are needed;

Any discussions, concerns or co-works(e.g. SGLang Distributed KV) are welcomed!

-- from Ant SCT inference team.

Update: The development branch is created at dev/hkvm.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @zhaoyongke on GitHub (May 16, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2117 Originally assigned to: @chestnut-Q on GitHub. ### Changes proposed In AntGroup's practice with Theta KVPool in agentic coding services, where a single LLM service reaches 10,000 card(accordingly 10,000 dummy clients), Mooncake Master becomes an obvious bottleneck. It will fail with these issues: * Allocators management is too heavy; * RPC processing will exhaust the CPU threads; * The recovery time for HA is not acceptable; Even with HA mode ON, multiple Masters are available, but only one Master instance is active for all clients' requests. <img width="2290" height="846" alt="Image" src="https://github.com/user-attachments/assets/4ae94f1d-9f35-4789-8273-ba4cfde651ae" /> When large mounts of online requests arrived, the former Leader Master instance may crash, the new Leader Master instance is activated, but may crash soon after oplog/snapshot recovery with historical failed and newly arrived requests, which called "avalanche" or "snowslide" effect. <img width="2290" height="846" alt="Image" src="https://github.com/user-attachments/assets/bdc71f34-c44b-4530-a10b-314adfa262a0" /> <img width="2290" height="846" alt="Image" src="https://github.com/user-attachments/assets/170e297b-a7c6-4399-83dd-272ad05b8524" /> <img width="2290" height="846" alt="Image" src="https://github.com/user-attachments/assets/15b36fc7-760d-4aab-b841-140dc7fc7537" /> We propose a Hierarchical KVCache Management(HKVM for simple in the remainder part) Architecture for large scale Mooncake Store deployment. <img width="3724" height="1506" alt="Image" src="https://github.com/user-attachments/assets/40ec0330-b815-4f0c-a702-c526a867c9cb" /> The picture shows a cluster with N sub-cluster and M nodes(we suggest only 1 real client in a node, dummy clients are 8 or 16 times of real clients with different card type) in each sub-cluster, the total nodes are M*N=C. In a sub-cluster, the Sub Master and Clients are the same with original deployment. The only change is a higher level Global Master for global KVCache management. In HKVM, a Global Master is tailored from original Master. Its functions are: * Registry of Sub Masters; * KVEvent collection from Sub Master; * Act as a Proxy for cross-sub-cluster transferring; * Weak Consistency with Sub Masters(in contrast to Sub Master, which is Strong Consistency within sub-cluster); We can adjust M and N for performance & reliability trade-off. Specially, when M=1, it will behave like the new [P2P Arch](https://github.com/kvcache-ai/Mooncake/tree/P2P-Mooncake-Store); when N=1, it will degenerate into original centralized arch. In our practice, we set M to the scale of P/D group(such as 5 in 2P3D, or 2 in 1P1D), and N to number of P/D groups. If we add a new level, HKVM naturally supports [PrfaaS](https://arxiv.org/abs/2604.15039). <img width="8210" height="1922" alt="Image" src="https://github.com/user-attachments/assets/c07b0ec5-e8d7-45dc-a97f-f0756a717486" /> Prefill Service(in the left) and P/D Service(in the right) can run on different clusters and heterogeneous hardwares (H200 + H20 in the paper). In each cluster, we create a global kvcache pool. Prefill Service will generate KVCache of long-input-requests continuously. P/D Service will processing normal requests in parallel. The newly added Inter-DC Master collects KVEvents of both sides, checks the diff and launchs incremental updates via WAN(cross-datacenter) or LAN(intra-datacenter). Thanks to the global kvcache pool, KVCaches transferred can be reused in the future before evicted. We need set the pool sizes in each side according to Prefill Service's and P/D Service's max throughput. Thanks again to the global kvcache pool, we don't need high-bandwidth network between these clusters, the transfer time is relaxed to KVCache eviction period(from minutes to hours in our real environment). Our plan for next step: * Global Master implementation, this month; * Optimized real clients implementation, next month; * PrfaaS PoC, 26H2; Some concerns: * Adding a new level in HKVM may cause an increase in latency, especially in batchIsExist(); * Weak consistency may lead to consumer side getting an evicted value in provider side, value integrity check is needed; * Cross-datacenter transferring maybe unreliable, FEC/ECC are needed; Any discussions, concerns or co-works(e.g. [SGLang Distributed KV](https://github.com/sgl-project/sglang/issues/21846)) are welcomed! -- from Ant SCT inference team. Update: The development branch is created at [dev/hkvm](https://github.com/kvcache-ai/Mooncake/tree/dev/hkvm). ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

Store

Roadmap

enhancement

labels

2026-10-29 21:16:46 +00:00

bsakaguchi commented

2026-10-29 21:16:47 +00:00

Author

Owner

@chestnut-Q commented on GitHub (May 26, 2026):

Hi @zhaoyongke, thanks for the HKVM design. This is really helpful for our PrfaaS architecture! I have some questions and discussions:

Why do we need to set up multiple P/D groups within a single cluster? Is there any scheduling difference between P/D nodes that are in the same group vs. across different groups? Could this affect load balancing?

Adding a new level in HKVM may cause an increase in latency, especially in batchIsExist();

I agree with this concern. Especially with the three-level hierarchy introduced by PrfaaS, should we consider capping HKVM at 2 levels, even in the Inter-Datacenter setup?

@chestnut-Q commented on GitHub (May 26, 2026): Hi @zhaoyongke, thanks for the HKVM design. This is really helpful for our PrfaaS architecture! I have some questions and discussions: 1. Why do we need to set up multiple P/D groups within a single cluster? Is there any scheduling difference between P/D nodes that are in the same group vs. across different groups? Could this affect load balancing? > Adding a new level in HKVM may cause an increase in latency, especially in batchIsExist(); 2. I agree with this concern. Especially with the three-level hierarchy introduced by PrfaaS, should we consider capping HKVM at 2 levels, even in the Inter-Datacenter setup?

bsakaguchi commented

2026-10-29 21:16:47 +00:00

Author

Owner

@zhaoyongke commented on GitHub (May 26, 2026):

Why do we need to set up multiple P/D groups within a single cluster? Is there any scheduling difference between P/D nodes that are in the same group vs. across different groups? Could this affect load balancing?

Main reason is the scale, a single master managing ten thousands clients will be a bottleneck, so we split the big cluster into several sub-clusters, and every sub-cluster will have their own master. HKVM only affects mooncake store sharing range, and the scheduler is independent with HKVM.

Adding a new level in HKVM may cause an increase in latency, especially in batchIsExist();

I agree with this concern. Especially with the three-level hierarchy introduced by PrfaaS, should we consider capping HKVM at 2 levels, even in the Inter-Datacenter setup?

Sure, in this case we can get lower latency but the scale in each region may be limited. As PrfaaS introduces inter-region transferring, the latency is more tolerant than intra-region.

@zhaoyongke commented on GitHub (May 26, 2026): > 1. Why do we need to set up multiple P/D groups within a single cluster? Is there any scheduling difference between P/D nodes that are in the same group vs. across different groups? Could this affect load balancing? Main reason is the scale, a single master managing ten thousands clients will be a bottleneck, so we split the big cluster into several sub-clusters, and every sub-cluster will have their own master. HKVM only affects mooncake store sharing range, and the scheduler is independent with HKVM. > > Adding a new level in HKVM may cause an increase in latency, especially in batchIsExist(); > > 2. I agree with this concern. Especially with the three-level hierarchy introduced by PrfaaS, should we consider capping HKVM at 2 levels, even in the Inter-Datacenter setup? Sure, in this case we can get lower latency but the scale in each region may be limited. As PrfaaS introduces inter-region transferring, the latency is more tolerant than intra-region.

bsakaguchi commented

2026-10-29 21:16:47 +00:00

Author

Owner

@chestnut-Q commented on GitHub (May 26, 2026):

Why do we need to set up multiple P/D groups within a single cluster? Is there any scheduling difference between P/D nodes that are in the same group vs. across different groups? Could this affect load balancing?

Main reason is the scale, a single master managing ten thousands clients will be a bottleneck, so we split the big cluster into several sub-clusters, and every sub-cluster will have their own master. HKVM only affects mooncake store sharing range, and the scheduler is independent with HKVM.

Adding a new level in HKVM may cause an increase in latency, especially in batchIsExist();

I agree with this concern. Especially with the three-level hierarchy introduced by PrfaaS, should we consider capping HKVM at 2 levels, even in the Inter-Datacenter setup?

Sure, in this case we can get lower latency but the scale in each region may be limited. As PrfaaS introduces inter-region transferring, the latency is more tolerant than intra-region.

I understand that each sub-cluster still has a considerable number of nodes, say hundreds, but KVCache cannot be shared across sub-clusters via mooncake store. Instead, the scheduler needs to explicitly trigger cross-sub-cluster KVCache migration. Is my understanding correct?

@chestnut-Q commented on GitHub (May 26, 2026): > > 1. Why do we need to set up multiple P/D groups within a single cluster? Is there any scheduling difference between P/D nodes that are in the same group vs. across different groups? Could this affect load balancing? > > Main reason is the scale, a single master managing ten thousands clients will be a bottleneck, so we split the big cluster into several sub-clusters, and every sub-cluster will have their own master. HKVM only affects mooncake store sharing range, and the scheduler is independent with HKVM. > > > > Adding a new level in HKVM may cause an increase in latency, especially in batchIsExist(); > > > > > > > > 2. I agree with this concern. Especially with the three-level hierarchy introduced by PrfaaS, should we consider capping HKVM at 2 levels, even in the Inter-Datacenter setup? > > Sure, in this case we can get lower latency but the scale in each region may be limited. As PrfaaS introduces inter-region transferring, the latency is more tolerant than intra-region. I understand that each sub-cluster still has a considerable number of nodes, say hundreds, but KVCache cannot be shared across sub-clusters via mooncake store. Instead, the scheduler needs to explicitly trigger cross-sub-cluster KVCache migration. Is my understanding correct?

bsakaguchi commented

2026-10-29 21:16:47 +00:00

Author

Owner

@zhaoyongke commented on GitHub (May 27, 2026):

Why do we need to set up multiple P/D groups within a single cluster? Is there any scheduling difference between P/D nodes that are in the same group vs. across different groups? Could this affect load balancing?

Main reason is the scale, a single master managing ten thousands clients will be a bottleneck, so we split the big cluster into several sub-clusters, and every sub-cluster will have their own master. HKVM only affects mooncake store sharing range, and the scheduler is independent with HKVM.

Adding a new level in HKVM may cause an increase in latency, especially in batchIsExist();

I agree with this concern. Especially with the three-level hierarchy introduced by PrfaaS, should we consider capping HKVM at 2 levels, even in the Inter-Datacenter setup?

Sure, in this case we can get lower latency but the scale in each region may be limited. As PrfaaS introduces inter-region transferring, the latency is more tolerant than intra-region.

I understand that each sub-cluster still has a considerable number of nodes, say hundreds, but KVCache cannot be shared across sub-clusters via mooncake store. Instead, the scheduler needs to explicitly trigger cross-sub-cluster KVCache migration. Is my understanding correct?

Actually nodes can share KVCache with other sub-clusters by querying global master, with a slightly added latency, so KVCache is globally available, and scheduler treats all nodes as equal cache-hit rate in L3.

@zhaoyongke commented on GitHub (May 27, 2026): > > > 1. Why do we need to set up multiple P/D groups within a single cluster? Is there any scheduling difference between P/D nodes that are in the same group vs. across different groups? Could this affect load balancing? > > > > > > Main reason is the scale, a single master managing ten thousands clients will be a bottleneck, so we split the big cluster into several sub-clusters, and every sub-cluster will have their own master. HKVM only affects mooncake store sharing range, and the scheduler is independent with HKVM. > > > > Adding a new level in HKVM may cause an increase in latency, especially in batchIsExist(); > > > > > > > > > > > > 2. I agree with this concern. Especially with the three-level hierarchy introduced by PrfaaS, should we consider capping HKVM at 2 levels, even in the Inter-Datacenter setup? > > > > > > Sure, in this case we can get lower latency but the scale in each region may be limited. As PrfaaS introduces inter-region transferring, the latency is more tolerant than intra-region. > > I understand that each sub-cluster still has a considerable number of nodes, say hundreds, but KVCache cannot be shared across sub-clusters via mooncake store. Instead, the scheduler needs to explicitly trigger cross-sub-cluster KVCache migration. Is my understanding correct? Actually nodes can share KVCache with other sub-clusters by querying global master, with a slightly added latency, so KVCache is globally available, and scheduler treats all nodes as equal cache-hit rate in L3.

bsakaguchi commented

2026-10-29 21:16:48 +00:00

Author

Owner

@chestnut-Q commented on GitHub (May 27, 2026):

Actually nodes can share KVCache with other sub-clusters by querying global master, with a slightly added latency, so KVCache is globally available, and scheduler treats all nodes as equal cache-hit rate in L3.

Thanks for the clarification @zhaoyongke. Really looking forward to the Global Master implementation!

One thought for when we later extend to inter-datacenter HKVM: we could attach labels to sub-masters to achieve a soft/hard partitioning effect. Masters sharing the same label would be interconnected with high bandwidth, while masters with different labels could have KVCache transfer bottlenecks. This would give us a more flexible knob for KVCache management policies.

@chestnut-Q commented on GitHub (May 27, 2026): > Actually nodes can share KVCache with other sub-clusters by querying global master, with a slightly added latency, so KVCache is globally available, and scheduler treats all nodes as equal cache-hit rate in L3. Thanks for the clarification @zhaoyongke. Really looking forward to the Global Master implementation! One thought for when we later extend to inter-datacenter HKVM: we could attach labels to sub-masters to achieve a soft/hard partitioning effect. Masters sharing the same label would be interconnected with high bandwidth, while masters with different labels could have KVCache transfer bottlenecks. This would give us a more flexible knob for KVCache management policies.

bsakaguchi commented

2026-10-29 21:16:48 +00:00

Author

Owner

@zhaoyongke commented on GitHub (May 28, 2026):

Actually nodes can share KVCache with other sub-clusters by querying global master, with a slightly added latency, so KVCache is globally available, and scheduler treats all nodes as equal cache-hit rate in L3.

Thanks for the clarification @zhaoyongke. Really looking forward to the Global Master implementation!

One thought for when we later extend to inter-datacenter HKVM: we could attach labels to sub-masters to achieve a soft/hard partitioning effect. Masters sharing the same label would be interconnected with high bandwidth, while masters with different labels could have KVCache transfer bottlenecks. This would give us a more flexible knob for KVCache management policies.

Pretty good suggestion! We'll consider this in the implementation.

@zhaoyongke commented on GitHub (May 28, 2026): > > Actually nodes can share KVCache with other sub-clusters by querying global master, with a slightly added latency, so KVCache is globally available, and scheduler treats all nodes as equal cache-hit rate in L3. > > Thanks for the clarification [@zhaoyongke](https://github.com/zhaoyongke). Really looking forward to the Global Master implementation! > > One thought for when we later extend to inter-datacenter HKVM: we could attach labels to sub-masters to achieve a soft/hard partitioning effect. Masters sharing the same label would be interconnected with high bandwidth, while masters with different labels could have KVCache transfer bottlenecks. This would give us a more flexible knob for KVCache management policies. Pretty good suggestion! We'll consider this in the implementation.

bsakaguchi commented

2026-10-29 21:16:48 +00:00

Author

Owner

@zchuango commented on GitHub (Jul 8, 2026):

Hi @zhaoyongke , we are very interested in the hierarchical master architecture proposed in #2117.

This feature is important for our large-scale deployment scenario, where a single active master may become a bottleneck with many clients. We would like to know if there has been any progress on this work, or if there is any current plan/timeline for it.

Thanks!

@zchuango commented on GitHub (Jul 8, 2026): Hi @zhaoyongke , we are very interested in the hierarchical master architecture proposed in #2117. This feature is important for our large-scale deployment scenario, where a single active master may become a bottleneck with many clients. We would like to know if there has been any progress on this work, or if there is any current plan/timeline for it. Thanks!

bsakaguchi commented

2026-10-29 21:16:48 +00:00

Author

Owner

@zhaoyongke commented on GitHub (Jul 16, 2026):

Hi @zhaoyongke , we are very interested in the hierarchical master architecture proposed in #2117.

This feature is important for our large-scale deployment scenario, where a single active master may become a bottleneck with many clients. We would like to know if there has been any progress on this work, or if there is any current plan/timeline for it.

Thanks!

Hi @zchuango , we're building globally shared kvcache pool in cluster level now, a lot of engineering work is on-going, including inference engine and mooncake store refactoring, and online requests modeling.
The roadmap of HKVM is planned as follow:

Intra datacenter poc will be publicly available in Aug.
Inter datacenter poc will be publicly available in Q4.

Thks for your attention!

@zhaoyongke commented on GitHub (Jul 16, 2026): > Hi [@zhaoyongke](https://github.com/zhaoyongke) , we are very interested in the hierarchical master architecture proposed in [#2117](https://github.com/kvcache-ai/Mooncake/issues/2117). > > This feature is important for our large-scale deployment scenario, where a single active master may become a bottleneck with many clients. We would like to know if there has been any progress on this work, or if there is any current plan/timeline for it. > > Thanks! Hi @zchuango , we're building globally shared kvcache pool in cluster level now, a lot of engineering work is on-going, including inference engine and mooncake store refactoring, and online requests modeling. The roadmap of HKVM is planned as follow: * Intra datacenter poc will be publicly available in Aug. * Inter datacenter poc will be publicly available in Q4. Thks for your attention!

bsakaguchi referenced this issue

2026-10-29 21:29:21 +00:00

[PR #582] [MERGED] mooncake-common: add config class #1668

Rows
Columns

[GH-ISSUE #2117] [RFC]: Hierarchical Arch for Intra/Inter Data Center Deployment #582

Changes proposed

Before submitting a new issue...