kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1136] loading blk加载过慢，且出现numa_sched_setaffinity_v2_int() failed: Invalid argument #791

New issue
Open
opened 2026-10-29 14:44:33 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-29 14:44:33 +00:00
Owner
Copy link

Originally created by @daydayuptang on GitHub (Apr 14, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1136

Originally assigned to: @ErvinXie on GitHub.

我按照如下方式启动：
USE_NUMA=1 python ktransformers/server/main.py
--port 10002
--model_path /work/home/uptang14/DeepSeek-R1
--gguf_path /work/home/uptang14/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
--max_new_tokens 1024
--cache_lens 32768
--chunk_size 256
--max_batch_size 4
--cpu_infer 32
--backend_type balance_serve
但是在加载过程中出现：numa_sched_setaffinity_v2_int() failed: Invalid argument
且加载刀blk.12就不动了，此过程持续2小时：
Image
Image

执行numactl --hardware:
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 262043 MB
node 0 free: 216468 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 262143 MB
node 1 free: 216547 MB
node 2 cpus: 16 17 18 19 20 21 22 23
node 2 size: 262143 MB
node 2 free: 209068 MB
node 3 cpus: 24 25 26 27 28 29 30 31
node 3 size: 249855 MB
node 3 free: 208122 MB
node 4 cpus: 32 33 34 35 36 37 38 39
node 4 size: 262143 MB
node 4 free: 215227 MB
node 5 cpus: 40 41 42 43 44 45 46 47
node 5 size: 262143 MB
node 5 free: 214383 MB
node 6 cpus: 48 49 50 51 52 53 54 55
node 6 size: 262143 MB
node 6 free: 215844 MB
node 7 cpus: 56 57 58 59 60 61 62 63
node 7 size: 262143 MB
node 7 free: 120269 MB
node distances:
node 0 1 2 3 4 5 6 7
0: 10 16 16 16 28 28 22 28
1: 16 10 16 16 28 28 28 22
2: 16 16 10 16 22 28 28 28
3: 16 16 16 10 28 22 28 28
4: 28 28 22 28 10 16 16 16
5: 28 28 28 22 16 10 16 16
6: 22 28 28 28 16 16 10 16
7: 28 22 28 28 16 16 16 10

Originally created by @daydayuptang on GitHub (Apr 14, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1136 Originally assigned to: @ErvinXie on GitHub. 我按照如下方式启动： USE_NUMA=1 python ktransformers/server/main.py \ --port 10002 \ --model_path /work/home/uptang14/DeepSeek-R1 \ --gguf_path /work/home/uptang14/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \ --max_new_tokens 1024 \ --cache_lens 32768 \ --chunk_size 256 \ --max_batch_size 4 \ --cpu_infer 32 \ --backend_type balance_serve 但是在加载过程中出现：numa_sched_setaffinity_v2_int() failed: Invalid argument 且加载刀blk.12就不动了，此过程持续2小时： <img width="247" alt="Image" src="https://github.com/user-attachments/assets/2bbe4096-e3af-4a5f-a0e3-059049a5c3c3" /> <img width="297" alt="Image" src="https://github.com/user-attachments/assets/70304256-5242-4323-8ce8-94f61a25aa60" /> 执行numactl --hardware: available: 8 nodes (0-7) node 0 cpus: 0 1 2 3 4 5 6 7 node 0 size: 262043 MB node 0 free: 216468 MB node 1 cpus: 8 9 10 11 12 13 14 15 node 1 size: 262143 MB node 1 free: 216547 MB node 2 cpus: 16 17 18 19 20 21 22 23 node 2 size: 262143 MB node 2 free: 209068 MB node 3 cpus: 24 25 26 27 28 29 30 31 node 3 size: 249855 MB node 3 free: 208122 MB node 4 cpus: 32 33 34 35 36 37 38 39 node 4 size: 262143 MB node 4 free: 215227 MB node 5 cpus: 40 41 42 43 44 45 46 47 node 5 size: 262143 MB node 5 free: 214383 MB node 6 cpus: 48 49 50 51 52 53 54 55 node 6 size: 262143 MB node 6 free: 215844 MB node 7 cpus: 56 57 58 59 60 61 62 63 node 7 size: 262143 MB node 7 free: 120269 MB node distances: node 0 1 2 3 4 5 6 7 0: 10 16 16 16 28 28 22 28 1: 16 10 16 16 28 28 28 22 2: 16 16 10 16 22 28 28 28 3: 16 16 16 10 28 22 28 28 4: 28 28 22 28 10 16 16 16 5: 28 28 28 22 16 10 16 16 6: 22 28 28 28 16 16 10 16 7: 28 22 28 28 16 16 16 10
bsakaguchi commented 2026-10-29 14:44:34 +00:00
Author
Owner
Copy link

@ErvinXie commented on GitHub (Apr 14, 2025):

你好，我们现在这个版本的 NUMA 会复制权重各一份到每一个 NUMA 上，因此需要每个 NUMA 都有足够的内存。如果是多 NUMA 的机器，可以在 BIOS 当中设置，尽量减少 NUMA 的数量，然后再运行 ktransformers。

<!-- gh-comment-id:2800819870 --> @ErvinXie commented on GitHub (Apr 14, 2025): 你好，我们现在这个版本的 NUMA 会复制权重各一份到每一个 NUMA 上，因此需要每个 NUMA 都有足够的内存。如果是多 NUMA 的机器，可以在 BIOS 当中设置，尽量减少 NUMA 的数量，然后再运行 ktransformers。
bsakaguchi commented 2026-10-29 14:44:34 +00:00
Author
Owner
Copy link

@daydayuptang commented on GitHub (Apr 18, 2025):

感谢您的回复，由于使用的是服务器配置无法在BIOS里进行更改，因此使用env USE_NUMA=1 \ numactl --cpunodebind=0-3,4-7 --interleave=0-3,4-7 \ python ktransformers/server/main.py \ --port 10002 \ --model_path /work/home/uptang14/DeepSeek-R1 \ --gguf_path /work/home/uptang14/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \ --max_new_tokens 1024 \ --cache_lens 32768 \ --chunk_size 256 \ --max_batch_size 4 \ --cpu_infer 32 \ --backend_type balance_serve \ --force_think
指令将8个numa节点逻辑合成四个。但是问题仍然相同，未解决。

值得注意的是，这条指令中修改cup_infer为24，即env USE_NUMA=1 \ numactl --cpunodebind=0-3,4-7 --interleave=0-3,4-7 \ python ktransformers/server/main.py \ --port 10002 \ --model_path /work/home/uptang14/DeepSeek-R1 \ --gguf_path /work/home/uptang14/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \ --max_new_tokens 1024 \ --cache_lens 32768 \ --chunk_size 256 \ --max_batch_size 4 \ --cpu_infer 24 \ --backend_type balance_serve \ --force_think则报错如下：loading token_embd.weight to cpu
loading token_embd.weight to cpu
loading token_embd.weight with CPU
loading blk.0.attn_q_a_norm.weight to cuda
loading blk.0.attn_kv_a_norm.weight to cuda
loading blk.0.attn_kv_b.weight to cuda:0
loading blk.0.attn_norm.weight to cuda
loading blk.0.ffn_norm.weight to cuda
loading blk.1.attn_q_a_norm.weight to cuda
loading blk.1.attn_kv_a_norm.weight to cuda
loading blk.1.attn_kv_b.weight to cuda:0
loading blk.1.attn_norm.weight to cuda
loading blk.1.ffn_norm.weight to cuda
loading blk.2.attn_q_a_norm.weight to cuda
loading blk.2.attn_kv_a_norm.weight to cuda
loading blk.2.attn_kv_b.weight to cuda:0
loading blk.2.attn_norm.weight to cuda
loading blk.2.ffn_norm.weight to cuda
loading blk.3.attn_q_a_norm.weight to cuda
loading blk.3.attn_kv_a_norm.weight to cuda
loading blk.3.attn_kv_b.weight to cuda:0
Process SpawnProcess-1:
Traceback (most recent call last):
File "/opt/conda/envs/kt/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/envs/kt/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 244, in run_engine
engine = Engine(args, token_queue, broadcast_endpoint)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 162, in init
optimize_and_load_gguf(self.model, optimize_config_path, gguf_path, config)
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf
load_weights(module, gguf_loader)
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 127, in load_weights
module.load()
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 127, in load_weights
module.load()
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 127, in load_weights
module.load()
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 1192, in load
self.generate_experts.load(w, warmup=warmup)
File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 185, in load
self.moe = MOE(moe_config)
^^^^^^^^^^^^^^^
KeyboardInterrupt

numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
numa_sched_setaffinity_v2_int() failed: Invalid argument
loading blk.3.attn_norm.weight to cuda
loading blk.3.ffn_norm.weight to cuda
loading blk.4.attn_q_a_norm.weight to cuda
loading blk.4.attn_kv_a_norm.weight to cuda
loading blk.4.attn_kv_b.weight to cuda:0
...

<!-- gh-comment-id:2814819310 --> @daydayuptang commented on GitHub (Apr 18, 2025): 感谢您的回复，由于使用的是服务器配置无法在BIOS里进行更改，因此使用`env USE_NUMA=1 \ numactl --cpunodebind=0-3,4-7 --interleave=0-3,4-7 \ python ktransformers/server/main.py \ --port 10002 \ --model_path /work/home/uptang14/DeepSeek-R1 \ --gguf_path /work/home/uptang14/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \ --max_new_tokens 1024 \ --cache_lens 32768 \ --chunk_size 256 \ --max_batch_size 4 \ --cpu_infer 32 \ --backend_type balance_serve \ --force_think` 指令将8个numa节点逻辑合成四个。**但是问题仍然相同，未解决**。 值得注意的是，这条指令中修改cup_infer为24，即`env USE_NUMA=1 \ numactl --cpunodebind=0-3,4-7 --interleave=0-3,4-7 \ python ktransformers/server/main.py \ --port 10002 \ --model_path /work/home/uptang14/DeepSeek-R1 \ --gguf_path /work/home/uptang14/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \ --max_new_tokens 1024 \ --cache_lens 32768 \ --chunk_size 256 \ --max_batch_size 4 \ --cpu_infer 24 \ --backend_type balance_serve \ --force_think`则报错如下：loading token_embd.weight to cpu loading token_embd.weight to cpu loading token_embd.weight with CPU loading blk.0.attn_q_a_norm.weight to cuda loading blk.0.attn_kv_a_norm.weight to cuda loading blk.0.attn_kv_b.weight to cuda:0 loading blk.0.attn_norm.weight to cuda loading blk.0.ffn_norm.weight to cuda loading blk.1.attn_q_a_norm.weight to cuda loading blk.1.attn_kv_a_norm.weight to cuda loading blk.1.attn_kv_b.weight to cuda:0 loading blk.1.attn_norm.weight to cuda loading blk.1.ffn_norm.weight to cuda loading blk.2.attn_q_a_norm.weight to cuda loading blk.2.attn_kv_a_norm.weight to cuda loading blk.2.attn_kv_b.weight to cuda:0 loading blk.2.attn_norm.weight to cuda loading blk.2.ffn_norm.weight to cuda loading blk.3.attn_q_a_norm.weight to cuda loading blk.3.attn_kv_a_norm.weight to cuda loading blk.3.attn_kv_b.weight to cuda:0 Process SpawnProcess-1: Traceback (most recent call last): File "/opt/conda/envs/kt/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/opt/conda/envs/kt/lib/python3.11/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 244, in run_engine engine = Engine(args, token_queue, broadcast_endpoint) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 162, in __init__ optimize_and_load_gguf(self.model, optimize_config_path, gguf_path, config) File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf load_weights(module, gguf_loader) File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 125, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 127, in load_weights module.load() File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load utils.load_weights(child, self.gguf_loader, self.key+".") File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 125, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 125, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 125, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 127, in load_weights module.load() File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load utils.load_weights(child, self.gguf_loader, self.key+".") File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 125, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/util/utils.py", line 127, in load_weights module.load() File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 1192, in load self.generate_experts.load(w, warmup=warmup) File "/opt/conda/envs/kt/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 185, in load self.moe = MOE(moe_config) ^^^^^^^^^^^^^^^ KeyboardInterrupt numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument numa_sched_setaffinity_v2_int() failed: Invalid argument loading blk.3.attn_norm.weight to cuda loading blk.3.ffn_norm.weight to cuda loading blk.4.attn_q_a_norm.weight to cuda loading blk.4.attn_kv_a_norm.weight to cuda loading blk.4.attn_kv_b.weight to cuda:0 ...
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#791
No description provided.