[PR #823] [MERGED] feat(store): add NUMA node binding support via bind_to_numa_node method #1850

New issue

Closed

opened 2026-10-29 21:30:51 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:30:51 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/823
Author: @xiaguan
Created: 9/11/2025
Status: ✅ Merged
Merged: 9/15/2025
Merged by: @xiaguan

Base: main ← Head: support_numa

📝 Commits (4)

311be28 feat(store): add NUMA node binding support via bind_to_numa_node method
99e4a15 feat(store): Add NUMA node binding with validation and error logging
39d8769 refactor(store): move NUMA binding to module-level function
885152f fmt

📊 Changes

3 files changed (+32 additions, -2 deletions)

View changed files

📝 mooncake-integration/store/store_py.cpp (+30 -1)
📝 mooncake-store/include/pybind_client.h (+1 -1)
📝 mooncake-store/src/pybind_client.cpp (+1 -0)

📄 Description

Test results with vllm + lmcache show that on multi-NUMA systems, setting NUMA affinity in advance significantly improves tail latency and delivers a better performance experience.

main

============ Serving Benchmark Result ============
Successful requests:                     50
Request rate configured (RPS):           4.00
Benchmark duration (s):                  31.87
Total input tokens:                      404739
Total generated tokens:                  6400
Request throughput (req/s):              1.57
Output token throughput (tok/s):         200.82
Total Token throughput (tok/s):          12900.98
---------------Time to First Token----------------
Mean TTFT (ms):                          2672.84
Median TTFT (ms):                        526.80
P99 TTFT (ms):                           9351.38
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          110.42
Median TPOT (ms):                        117.79
P99 TPOT (ms):                           123.70
---------------Inter-token Latency----------------
Mean ITL (ms):                           110.42
Median ITL (ms):                         92.50
P99 ITL (ms):                            359.44
==================================================

this pr

============ Serving Benchmark Result ============
Successful requests:                     50
Request rate configured (RPS):           4.00
Benchmark duration (s):                  29.64
Total input tokens:                      404739
Total generated tokens:                  6280
Request throughput (req/s):              1.69
Output token throughput (tok/s):         211.88
Total Token throughput (tok/s):          13867.48
---------------Time to First Token----------------
Mean TTFT (ms):                          2016.64
Median TTFT (ms):                        370.01
P99 TTFT (ms):                           7266.21
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          100.86
Median TPOT (ms):                        105.79
P99 TPOT (ms):                           111.83
---------------Inter-token Latency----------------
Mean ITL (ms):                           100.95
Median ITL (ms):                         91.73
P99 ITL (ms):                            294.77
==================================================

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/823 **Author:** [@xiaguan](https://github.com/xiaguan) **Created:** 9/11/2025 **Status:** ✅ Merged **Merged:** 9/15/2025 **Merged by:** [@xiaguan](https://github.com/xiaguan) **Base:** `main` ← **Head:** `support_numa` --- ### 📝 Commits (4) - [`311be28`](https://github.com/kvcache-ai/Mooncake/commit/311be289a63c4e6b32b96c872c60a458ae2a3efb) feat(store): add NUMA node binding support via bind_to_numa_node method - [`99e4a15`](https://github.com/kvcache-ai/Mooncake/commit/99e4a15d2d2b661e086157cb312d528fd557b60f) feat(store): Add NUMA node binding with validation and error logging - [`39d8769`](https://github.com/kvcache-ai/Mooncake/commit/39d87697809b74ccd3bf865b2ec11cf518876536) refactor(store): move NUMA binding to module-level function - [`885152f`](https://github.com/kvcache-ai/Mooncake/commit/885152f93f3214e328b0b54ababdd4f3b269a479) fmt ### 📊 Changes **3 files changed** (+32 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-integration/store/store_py.cpp` (+30 -1) 📝 `mooncake-store/include/pybind_client.h` (+1 -1) 📝 `mooncake-store/src/pybind_client.cpp` (+1 -0) </details> ### 📄 Description Test results with vllm + lmcache show that on multi-NUMA systems, setting NUMA affinity in advance significantly improves tail latency and delivers a better performance experience. main ``` ============ Serving Benchmark Result ============ Successful requests: 50 Request rate configured (RPS): 4.00 Benchmark duration (s): 31.87 Total input tokens: 404739 Total generated tokens: 6400 Request throughput (req/s): 1.57 Output token throughput (tok/s): 200.82 Total Token throughput (tok/s): 12900.98 ---------------Time to First Token---------------- Mean TTFT (ms): 2672.84 Median TTFT (ms): 526.80 P99 TTFT (ms): 9351.38 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 110.42 Median TPOT (ms): 117.79 P99 TPOT (ms): 123.70 ---------------Inter-token Latency---------------- Mean ITL (ms): 110.42 Median ITL (ms): 92.50 P99 ITL (ms): 359.44 ================================================== ``` this pr ``` ============ Serving Benchmark Result ============ Successful requests: 50 Request rate configured (RPS): 4.00 Benchmark duration (s): 29.64 Total input tokens: 404739 Total generated tokens: 6280 Request throughput (req/s): 1.69 Output token throughput (tok/s): 211.88 Total Token throughput (tok/s): 13867.48 ---------------Time to First Token---------------- Mean TTFT (ms): 2016.64 Median TTFT (ms): 370.01 P99 TTFT (ms): 7266.21 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 100.86 Median TPOT (ms): 105.79 P99 TPOT (ms): 111.83 ---------------Inter-token Latency---------------- Mean ITL (ms): 100.95 Median ITL (ms): 91.73 P99 ITL (ms): 294.77 ================================================== ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:30:51 +00:00