kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #903] [RFC] [Store] Improving `global_segment_size` and `local_buffer_size` Parameter Usability #285

New issue

Closed

opened 2026-10-29 21:12:01 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-29 21:12:01 +00:00

Owner

Originally created by @ykwd on GitHub (Sep 29, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/903

Changes proposed

Background

Mooncake Store has two key configuration parameters:

global_segment_size – The amount of local memory a node contributes to the distributed memory pool.
local_buffer_size – The amount of local memory allocated for data transfer within the current instance.

Currently, there are two main issues with these parameters:

No default values:
- global_segment_size is highly dependent on deployment scenarios, and the appropriate value varies greatly depending on the node’s available memory. A default value for global_segment_size cannot simply be a proportion of the server’s total memory, because the number of store instances on the server is unknown. For example, in an 8 TP inference service, there might be 8 store instances, and each instance should contribute only a fraction of the total memory.
- local_buffer_size currently has no default, although a safe default could be reasonable.
User-unfriendly input format:
Both parameters require the user to specify memory in bytes. For example, to set global_segment_size to 10GB, users must manually calculate 10 * 1024^3 = 10737418240 bytes, which is inconvenient.

Proposed Improvements

1. Default Value for `local_buffer_size`

Provide a reasonable default value (e.g., 4GB), since data transfer buffers typically consume a small portion of memory. If only zero-copy methods are used, this could lead to a 4GB memory waste. But maybe this is acceptable in exchange for easy usage.

2. User-Friendly Input

Support human-readable units:
Allow inputs such as 512MB or 10GB instead of raw byte values.
Percentage-based input – Allow specifying a percentage of total memory such as 50%.

Request for Comments

We seek community feedback on this topic. Your feedback will help us improve usability while maintaining flexibility for diverse deployment scenarios.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @ykwd on GitHub (Sep 29, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/903 ### Changes proposed ## Background Mooncake Store has two key configuration parameters: 1. **`global_segment_size`** – The amount of local memory a node contributes to the distributed memory pool. 2. **`local_buffer_size`** – The amount of local memory allocated for data transfer within the current instance. Currently, there are two main issues with these parameters: 1. **No default values**: * `global_segment_size` is highly dependent on deployment scenarios, and the appropriate value varies greatly depending on the node’s available memory. A default value for `global_segment_size` cannot simply be a proportion of the server’s total memory, because the number of store instances on the server is unknown. For example, in an 8 TP inference service, there might be 8 store instances, and each instance should contribute only a fraction of the total memory. * `local_buffer_size` currently has no default, although a safe default could be reasonable. 2. **User-unfriendly input format**: Both parameters require the user to specify memory in bytes. For example, to set `global_segment_size` to 10GB, users must manually calculate `10 * 1024^3 = 10737418240` bytes, which is inconvenient. --- ## Proposed Improvements ### 1. Default Value for `local_buffer_size` Provide a reasonable default value (e.g., 4GB), since data transfer buffers typically consume a small portion of memory. If only zero-copy methods are used, this could lead to a 4GB memory waste. But maybe this is acceptable in exchange for easy usage. ### 2. User-Friendly Input * **Support human-readable units**: Allow inputs such as `512MB` or `10GB` instead of raw byte values. * **Percentage-based input** – Allow specifying a percentage of total memory such as `50%`. --- ## Request for Comments We seek community feedback on this topic. Your feedback will help us improve usability while maintaining flexibility for diverse deployment scenarios. --- ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:12:01 +00:00

bsakaguchi commented

2026-10-29 21:12:02 +00:00

Author

Owner

@stmatengss commented on GitHub (Sep 30, 2025):

For 2, I think https://github.com/sgl-project/sglang/pull/10745 has already solved this issue.

@stmatengss commented on GitHub (Sep 30, 2025): For 2, I think https://github.com/sgl-project/sglang/pull/10745 has already solved this issue.

bsakaguchi commented

2026-10-29 21:12:02 +00:00

Author

Owner

@ykwd commented on GitHub (Sep 30, 2025):

For 2, I think sgl-project/sglang#10745 has already solved this issue.

Yes. This is a major step toward better usability by allowing the "4GB" format parameter for the sglang HiCache Mooncake backend. We also need to apply this to the store’s setup function so that other ways of starting the store can accept the same input. In addition, this RFC also covers other input formats, such as percentage-based values or other forms that users may need.

@ykwd commented on GitHub (Sep 30, 2025): > For 2, I think [sgl-project/sglang#10745](https://github.com/sgl-project/sglang/pull/10745) has already solved this issue. Yes. This is a major step toward better usability by allowing the "4GB" format parameter for the sglang HiCache Mooncake backend. We also need to apply this to the store’s setup function so that other ways of starting the store can accept the same input. In addition, this RFC also covers other input formats, such as percentage-based values or other forms that users may need.

bsakaguchi commented

2026-10-29 21:12:02 +00:00

Author

Owner

@stmatengss commented on GitHub (Sep 30, 2025):

Backward compatibility is important. A uint64 is sufficient for local/global size, and the human-readable conversion should be handled by higher-level software using Mooncake store.

@stmatengss commented on GitHub (Sep 30, 2025): Backward compatibility is important. A uint64 is sufficient for local/global size, and the human-readable conversion should be handled by higher-level software using Mooncake store.

bsakaguchi commented

2026-10-29 21:12:02 +00:00

Author

Owner

@ykwd commented on GitHub (Sep 30, 2025):

Backward compatibility is important. A uint64 is sufficient for local/global size, and the human-readable conversion should be handled by higher-level software using Mooncake store.

Sure. We will maintain backward compatibility. We could add a new setup wrapper that translates the human-readable format into a uint64. The higher-level software could either handle the format itself and call the original setup function, or delegate this to the new setup function.

The advantage of implementing this on the store side is that it can support multiple formats at once, the code can be CI-validated, it can be reused by all upper-layer applications, and it’s easier for us to modify.

@ykwd commented on GitHub (Sep 30, 2025): > Backward compatibility is important. A uint64 is sufficient for local/global size, and the human-readable conversion should be handled by higher-level software using Mooncake store. Sure. We will maintain backward compatibility. We could add a new setup wrapper that translates the human-readable format into a uint64. The higher-level software could either handle the format itself and call the original setup function, or delegate this to the new setup function. The advantage of implementing this on the store side is that it can support multiple formats at once, the code can be CI-validated, it can be reused by all upper-layer applications, and it’s easier for us to modify.

bsakaguchi commented

2026-10-29 21:12:02 +00:00

Author

Owner

@CAICAIIs commented on GitHub (May 23, 2026):

hi, I’d like to take this issue.

I dug through the current code and the remaining gap looks like the store-side ConfigDict setup path in RealClient::setup_internal: the standalone client and MooncakeConfig already handle human-readable sizes, but the dict wrapper still forwards strings into std::stoull, so values like "4GB" get misparsed and then fail the size checks. I’d like to send a narrow PR that keeps the existing uint64 APIs backward compatible, makes the dict setup path unit-aware, and adds regression tests for "4GB", "0", and invalid strings. If percentage-based input is still desired, I’d treat that as a follow-up once the denominator is agreed.

@CAICAIIs commented on GitHub (May 23, 2026): hi, I’d like to take this issue. I dug through the current code and the remaining gap looks like the store-side ConfigDict setup path in `RealClient::setup_internal`: the standalone client and `MooncakeConfig` already handle human-readable sizes, but the dict wrapper still forwards strings into `std::stoull`, so values like "4GB" get misparsed and then fail the size checks. I’d like to send a narrow PR that keeps the existing uint64 APIs backward compatible, makes the dict setup path unit-aware, and adds regression tests for "4GB", "0", and invalid strings. If percentage-based input is still desired, I’d treat that as a follow-up once the denominator is agreed.

bsakaguchi commented

2026-10-29 21:12:03 +00:00

Author

Owner

@ykwd commented on GitHub (May 23, 2026):

hi, I’d like to take this issue.

I dug through the current code and the remaining gap looks like the store-side ConfigDict setup path in RealClient::setup_internal: the standalone client and MooncakeConfig already handle human-readable sizes, but the dict wrapper still forwards strings into std::stoull, so values like "4GB" get misparsed and then fail the size checks. I’d like to send a narrow PR that keeps the existing uint64 APIs backward compatible, makes the dict setup path unit-aware, and adds regression tests for "4GB", "0", and invalid strings. If percentage-based input is still desired, I’d treat that as a follow-up once the denominator is agreed.

Sounds good to me! The issue in the dict wrapper does exist, and thanks a lot for pointing it out.

@ykwd commented on GitHub (May 23, 2026): > hi, I’d like to take this issue. > > I dug through the current code and the remaining gap looks like the store-side ConfigDict setup path in `RealClient::setup_internal`: the standalone client and `MooncakeConfig` already handle human-readable sizes, but the dict wrapper still forwards strings into `std::stoull`, so values like "4GB" get misparsed and then fail the size checks. I’d like to send a narrow PR that keeps the existing uint64 APIs backward compatible, makes the dict setup path unit-aware, and adds regression tests for "4GB", "0", and invalid strings. If percentage-based input is still desired, I’d treat that as a follow-up once the denominator is agreed. Sounds good to me! The issue in the dict wrapper does exist, and thanks a lot for pointing it out.

bsakaguchi referenced this issue

2026-10-29 21:27:33 +00:00

[PR #285] [MERGED] [TransferEngine] only dump metadata on the last selectDevice failure #1451