[GH-ISSUE #256] When USE_NUMA is set, Multi-GPU rule stop working #156

New issue

Closed

opened 2026-10-29 14:35:04 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:35:04 +00:00

Owner

Originally created by @3wweiweiwu on GitHub (Feb 14, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/256

Environment

OS: Ubuntu 20.04
Hardware: Azure Standard_NV72ads_A10_v5. 2x [A10 GPU 24GB VRAM], 800+GB Memory, 72 CPU Core
Model: deepseek-ai/DeepSeek-R1 671b-Q4_K_M
Python: 3.11
KTransformers Version: V0.2

Expected Behavior

The workload should be offload to 2 GPU

Actual Behavior:

The workload is only in 1 GPU.

Step to Reproduce Problem

git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule init
git submodule update
export USE_NUMA=1
sh ./install.sh
python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path ./deepseekr1-q4-k-m/DeepSeek-R1-Q4_K_M  --cpu_infer 70 --optimize_config_path./ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --total_context 32767 --max_new_tokens 1000

Originally created by @3wweiweiwu on GitHub (Feb 14, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/256 ## Environment **OS**: Ubuntu 20.04 **Hardware**: Azure Standard_NV72ads_A10_v5. 2x [A10 GPU 24GB VRAM], 800+GB Memory, 72 CPU Core **Model**: deepseek-ai/DeepSeek-R1 671b-Q4_K_M **Python**: 3.11 **KTransformers Version**: V0.2 ## Expected Behavior The workload should be offload to 2 GPU ## Actual Behavior: The workload is only in 1 GPU. ![Image](https://github.com/user-attachments/assets/d0de8b1f-4e63-4cb7-aa80-27332be27e03) ## Step to Reproduce Problem ```bash git clone https://github.com/kvcache-ai/ktransformers.git cd ktransformers git submodule init git submodule update export USE_NUMA=1 sh ./install.sh python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path ./deepseekr1-q4-k-m/DeepSeek-R1-Q4_K_M --cpu_infer 70 --optimize_config_path./ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --total_context 32767 --max_new_tokens 1000 ```

bsakaguchi closed this issue

2026-10-29 14:35:04 +00:00