[PR #28] [feat]: add --kt-numa-nodes for explicit NUMA node mapping #27

New issue

Open

opened 2026-10-29 21:06:36 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:06:36 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/28
Author: @ErvinXie
Created: 3/18/2026
Status: 🔄 Open

Base: main ← Head: feat/kt-numa-nodes

📝 Commits (1)

1b4478a add --kt-numa-nodes CLI parameter for explicit NUMA node mapping

📊 Changes

2 files changed (+13 additions, -0 deletions)

View changed files

📝 python/sglang/srt/layers/moe/kt_ep_wrapper.py (+3 -0)
📝 python/sglang/srt/server_args.py (+10 -0)

📄 Description

Summary

Add --kt-numa-nodes CLI parameter to ServerArgs for specifying NUMA node IDs
Thread numa_nodes through KTConfig dataclass to KTMoEWrapper instantiation
Parse comma-separated string (e.g., "1" or "0,1") into List[int]

Motivation

Companion to kvcache-ai/ktransformers#1891.

Currently subpool_numa_map in kt-kernel defaults to [0, 1, ..., threadpool_count-1]. This makes it impossible to bind a KTransformers instance to a specific NUMA node (e.g., node 1) without external numactl.

Usage

Deploy two independent instances on a dual-NUMA machine, each bound to a different NUMA node:

# Instance 1: bind to NUMA node 0
python -m sglang.launch_server \
  --model /path/to/model \
  --kt-threadpool-count 1 --kt-numa-nodes 0 \
  --kt-cpuinfer 48 \
  --port 30000 \
  ...

# Instance 2: bind to NUMA node 1
python -m sglang.launch_server \
  --model /path/to/model \
  --kt-threadpool-count 1 --kt-numa-nodes 1 \
  --kt-cpuinfer 48 \
  --port 30001 \
  ...

You can also specify multiple NUMA nodes in custom order:

# Reverse order
python -m sglang.launch_server \
  --kt-threadpool-count 2 --kt-numa-nodes 1,0 \
  ...

If --kt-numa-nodes is not specified, the behavior is unchanged.

Tested on

AMD EPYC 9355 dual-socket (2 NUMA nodes, 128 threads)
Verified CPUInfer creates worker pool on correct NUMA node
Verified backward compatibility

Test plan

from sglang.srt.server_args import ServerArgs imports successfully
--kt-numa-nodes is accepted by argparse and parsed correctly
Backward compatible: omitting --kt-numa-nodes behaves as before
End-to-end with companion kt-kernel PR on a multi-NUMA machine

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/28 **Author:** [@ErvinXie](https://github.com/ErvinXie) **Created:** 3/18/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/kt-numa-nodes` --- ### 📝 Commits (1) - [`1b4478a`](https://github.com/kvcache-ai/sglang/commit/1b4478a7cebd8e23eb8c1465e114d8571161191e) add --kt-numa-nodes CLI parameter for explicit NUMA node mapping ### 📊 Changes **2 files changed** (+13 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `python/sglang/srt/layers/moe/kt_ep_wrapper.py` (+3 -0) 📝 `python/sglang/srt/server_args.py` (+10 -0) </details> ### 📄 Description ## Summary - Add `--kt-numa-nodes` CLI parameter to `ServerArgs` for specifying NUMA node IDs - Thread `numa_nodes` through `KTConfig` dataclass to `KTMoEWrapper` instantiation - Parse comma-separated string (e.g., `"1"` or `"0,1"`) into `List[int]` ## Motivation Companion to kvcache-ai/ktransformers#1891. Currently `subpool_numa_map` in kt-kernel defaults to `[0, 1, ..., threadpool_count-1]`. This makes it impossible to bind a KTransformers instance to a specific NUMA node (e.g., node 1) without external `numactl`. ## Usage Deploy two independent instances on a dual-NUMA machine, each bound to a different NUMA node: ```bash # Instance 1: bind to NUMA node 0 python -m sglang.launch_server \ --model /path/to/model \ --kt-threadpool-count 1 --kt-numa-nodes 0 \ --kt-cpuinfer 48 \ --port 30000 \ ... # Instance 2: bind to NUMA node 1 python -m sglang.launch_server \ --model /path/to/model \ --kt-threadpool-count 1 --kt-numa-nodes 1 \ --kt-cpuinfer 48 \ --port 30001 \ ... ``` You can also specify multiple NUMA nodes in custom order: ```bash # Reverse order python -m sglang.launch_server \ --kt-threadpool-count 2 --kt-numa-nodes 1,0 \ ... ``` If `--kt-numa-nodes` is not specified, the behavior is unchanged. ## Tested on - AMD EPYC 9355 dual-socket (2 NUMA nodes, 128 threads) - Verified `CPUInfer` creates worker pool on correct NUMA node - Verified backward compatibility ## Test plan - [x] `from sglang.srt.server_args import ServerArgs` imports successfully - [x] `--kt-numa-nodes` is accepted by argparse and parsed correctly - [x] Backward compatible: omitting `--kt-numa-nodes` behaves as before - [x] End-to-end with companion kt-kernel PR on a multi-NUMA machine 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 21:06:36 +00:00

bsakaguchi referenced this issue

2026-10-29 21:06:36 +00:00

[PR #27] [fix](kt): Fix barrier call to use cpu_group instead of device_group #28

bsakaguchi referenced this issue

2026-10-29 21:06:57 +00:00

[PR #73] [MERGED] fix(dsv4): lazily allocate layerwise prefill slots #72