[GH-ISSUE #796] [Bug] USE_NUMA=1在2socket机器上不生效 / The USE_NUMA=1 setting does not take effect on a 2-socket machine. #577

New issue

Closed

opened 2026-10-29 14:41:25 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:41:25 +00:00

Owner

Originally created by @pkunight on GitHub (Mar 5, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/796

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

我在一台云服务器(经过了虚拟化)内按照下列文档执行了Dual socket的操作(export USE_NUMA=1 and reinstall).
I followed the steps for a dual socket configuration according to the documentation on a cloud server (after virtualization).

https://kvcache-ai.github.io/ktransformers/en/DeepseekR1_V3_tutorial.html#dual-socket-version-64-cores

但是, 模型并没有按文档说明的那样被复制到两个numa node对应的内存上, 而是被平均分配到两个numa node.
However, the model was not copied to the memory corresponding to the two NUMA nodes as described in the documentation. Instead, it was evenly distributed across the two NUMA nodes.

node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89
node 0 size: 414696 MB
node 0 free: 233615 MB
node 1 cpus: 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179
node 1 size: 415218 MB
node 1 free: 204063 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10

在这个场景下, 大量的跨numa传输导致模型生成性能极差, 在65cpu核的情况下只有不到6tokens/s.
In this scenario, the large number of cross-NUMA transfers results in very poor model generation performance, with fewer than 6 tokens per second with 65 CPU cores.

如果采用numactl -N 1 -m 1, 且采用32cpu核, 则可以获得和文档中一样的8.73tokens/s, 证明机器本身没有问题.
If numactl -N 1 -m 1 is used with 32 CPU cores, the performance reaches 8.73 tokens per second, which is the same as described in the documentation, proving that the machine itself is not the issue.

Reproduction

Environment

Originally created by @pkunight on GitHub (Mar 5, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/796 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug 我在一台云服务器(经过了虚拟化)内按照下列文档执行了Dual socket的操作(export USE_NUMA=1 and reinstall). I followed the steps for a dual socket configuration according to the documentation on a cloud server (after virtualization). https://kvcache-ai.github.io/ktransformers/en/DeepseekR1_V3_tutorial.html#dual-socket-version-64-cores 但是, 模型并没有按文档说明的那样被复制到两个numa node对应的内存上, 而是被平均分配到两个numa node. However, the model was not copied to the memory corresponding to the two NUMA nodes as described in the documentation. Instead, it was evenly distributed across the two NUMA nodes. ```available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 node 0 size: 414696 MB node 0 free: 233615 MB node 1 cpus: 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 node 1 size: 415218 MB node 1 free: 204063 MB node distances: node 0 1 0: 10 20 1: 20 10 ``` 在这个场景下, 大量的跨numa传输导致模型生成性能极差, 在65cpu核的情况下只有不到6tokens/s. In this scenario, the large number of cross-NUMA transfers results in very poor model generation performance, with fewer than 6 tokens per second with 65 CPU cores. 如果采用`numactl -N 1 -m 1`, 且采用32cpu核, 则可以获得和文档中一样的8.73tokens/s, 证明机器本身没有问题. If `numactl -N 1 -m 1` is used with 32 CPU cores, the performance reaches 8.73 tokens per second, which is the same as described in the documentation, proving that the machine itself is not the issue. ### Reproduction - ### Environment -

bsakaguchi closed this issue

2026-10-29 14:41:25 +00:00

bsakaguchi commented

2026-10-29 14:41:26 +00:00

Author

Owner

@ErvinXie commented on GitHub (Mar 5, 2025):

请确认是否有安装 libnuma

sudo apt update
sudo apt install libnuma-dev

@ErvinXie commented on GitHub (Mar 5, 2025): 请确认是否有安装 libnuma ```bash sudo apt update sudo apt install libnuma-dev ```

bsakaguchi commented

2026-10-29 14:41:26 +00:00

Author

Owner

@pkunight commented on GitHub (Mar 5, 2025):

请确认是否有安装 libnuma

sudo apt update
sudo apt install libnuma-dev

已确认确实是没安装libnuma-dev导致的, 在文档中并未提到. 感谢!

@pkunight commented on GitHub (Mar 5, 2025): > 请确认是否有安装 libnuma > > sudo apt update > sudo apt install libnuma-dev 已确认确实是没安装libnuma-dev导致的, 在文档中并未提到. 感谢!

bsakaguchi commented

2026-10-29 14:41:26 +00:00

Author

Owner

@ErvinXie commented on GitHub (Mar 5, 2025):

感谢你的反馈，我们会在文档中添加相关的说明。

@ErvinXie commented on GitHub (Mar 5, 2025): 感谢你的反馈，我们会在文档中添加相关的说明。

bsakaguchi referenced this issue

2026-10-29 14:52:22 +00:00

[PR #577] [MERGED] Fix the link address in the doc install.md #1429