[GH-ISSUE #1429] [Bug] How to utilize numa in docker approachingai/ktransformers:latest image #955

New issue

Closed

opened 2026-10-29 14:46:54 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:46:54 +00:00

Owner

Originally created by @apalesnail on GitHub (Jul 9, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1429

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

I am utilizing the approachingai/ktransformers:latest docker image and have so far been unable to get this working with numa.

I have a dual socket system & would like to take advantage of it. Has anyone been able to get this working?

Reproduction

USE_BALANCE_SERVE=1 USE_NUMA=1 python /workspace/ktransformers/ktransformers/server/main.py --host 0.0.0.0 --port 10002 --model_path /workspace/models/ollama --model_name KVCache-ai/DeepSeek-R1-0528-Q4_K_M --gguf_path /workspace/models/ollama --fast_safetensors true --max_new_tokens 65536 --cache_lens 65536 --cpu_infer 36 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml --backend_type balance_serve --use_cuda_graph --chunk_size 256 --force_think

Environment

2x Xeon. 1TB Ram (512GB/Socket)

I have also tried running env USE_BALANCE_SERVE=1 USE_NUMA=1 PYTHONPATH="$(which python)" PATH="$(dirname $(which python)):$PATH" bash ./install.sh, and successfully recompiled but this seems to break balance_serve and still does not enable NUMA and performance is significantly degraded vs 1 socket

Originally created by @apalesnail on GitHub (Jul 9, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1429 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug I am utilizing the approachingai/ktransformers:latest docker image and have so far been unable to get this working with numa. I have a dual socket system & would like to take advantage of it. Has anyone been able to get this working? ### Reproduction USE_BALANCE_SERVE=1 USE_NUMA=1 python /workspace/ktransformers/ktransformers/server/main.py --host 0.0.0.0 --port 10002 --model_path /workspace/models/ollama --model_name KVCache-ai/DeepSeek-R1-0528-Q4_K_M --gguf_path /workspace/models/ollama --fast_safetensors true --max_new_tokens 65536 --cache_lens 65536 --cpu_infer 36 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml --backend_type balance_serve --use_cuda_graph --chunk_size 256 --force_think ### Environment 2x Xeon. 1TB Ram (512GB/Socket) I have also tried running env USE_BALANCE_SERVE=1 USE_NUMA=1 PYTHONPATH="$(which python)" PATH="$(dirname $(which python)):$PATH" bash ./install.sh, and successfully recompiled but this seems to break balance_serve and still does not enable NUMA and performance is significantly degraded vs 1 socket

bsakaguchi closed this issue

2026-10-29 14:46:54 +00:00

bsakaguchi commented

2026-10-29 14:46:55 +00:00

Author

Owner

@apalesnail commented on GitHub (Jul 10, 2025):

This ended up being a limitation of Proxmox virtualization. Through some fancy hook scripts host NUMA is preserved in the guest VM.

Getting 5.5-6 T/s on DeepSeek V3 q4km. Not bad for a couple of Xeon V3's launched in 2013 (& 1x3090)!!

@apalesnail commented on GitHub (Jul 10, 2025): This ended up being a limitation of Proxmox virtualization. Through some fancy hook scripts host NUMA is preserved in the guest VM. Getting 5.5-6 T/s on DeepSeek V3 q4km. Not bad for a couple of Xeon V3's launched in 2013 (& 1x3090)!!