[GH-ISSUE #46] [ENHANCEMENT] improve GPU utilization for multi-GPU #22

New issue

Closed

opened 2026-10-29 14:32:44 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:32:44 +00:00

Owner

Originally created by @ELigoP on GitHub (Aug 20, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/46

When I replace default optimization rule yaml with multi-gpu version and start local_chat with

python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V2-Chat-0628 --gguf_path ~/.cache/lm-studio/models/bartowski/DeepSeek-V2-Chat-0628-GGUF --optimize_rule_path ./build/lib.linux-x86_64-cpython-312/ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu.yaml

it all works, but now GPUs utilization is about 50%:

I understand that this is already cutting edge optimization, and that this implementation attempts to place routing layers on GPU and experts on CPU. But maybe some more layers can still be put on GPUs? If some part of a single expert will be on GPU, it won't hurt, and that expert will be "hit", then there will be some improvement.

Originally created by @ELigoP on GitHub (Aug 20, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/46 When I replace default optimization rule `yaml` with multi-gpu version and start local_chat with ``` python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V2-Chat-0628 --gguf_path ~/.cache/lm-studio/models/bartowski/DeepSeek-V2-Chat-0628-GGUF --optimize_rule_path ./build/lib.linux-x86_64-cpython-312/ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu.yaml ``` it all works, but now GPUs utilization is about 50%: ![image](https://github.com/user-attachments/assets/8db2c648-c1e7-4c30-a446-511e0cfe3377) I understand that this is already cutting edge optimization, and that this implementation attempts to place routing layers on GPU and experts on CPU. But maybe some more layers can still be put on GPUs? If some part of a single expert will be on GPU, it won't hurt, and that expert will be "hit", then there will be some improvement.

bsakaguchi closed this issue

2026-10-29 14:32:45 +00:00

bsakaguchi commented

2026-10-29 14:32:45 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Aug 21, 2024):

Yes, you can put more layers on GPUs.

Puting extra 10 layers to 'cuda:0', for example, 0~9, the yaml about experts will be like this:


- match:
    name: "^model\\.layers\\.(0|[1-9])\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    kwargs:
      prefill_device: "cuda:0"
      prefill_op: "KExpertsTorch"
      generate_device: "cuda:0"
      generate_op:  "KExpertsTorch" # do remember using correct backend, KExpertsCPU only runable on cpu.
      out_device: "cuda:0"
  recursive: False # don't recursively inject submodules of this module

- match:
    name: "^model\\.layers\\.([12][0-9])\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    kwargs:
      prefill_device: "cuda:0"
      prefill_op: "KExpertsTorch"
      generate_device: "cpu"
      generate_op:  "KExpertsCPU"
      out_device: "cuda:0"
  recursive: False # don't recursively inject submodules of this module

- match:
    name: "^model\\.layers\\.([345][0-9])\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    kwargs:
      prefill_device: "cuda:1"
      prefill_op: "KExpertsTorch"
      generate_device: "cpu"
      generate_op:  "KExpertsCPU"
      out_device: "cuda:1"
  recursive: False # don't recursively inject submodules of this module

For more detail please read my tutorial. If you have any problem about tutorial, please feel free to ask!

@Azure-Tang commented on GitHub (Aug 21, 2024): Yes, you can put more layers on GPUs. Puting extra 10 layers to 'cuda:0', for example, 0~9, the yaml about experts will be like this: ```yaml - match: name: "^model\\.layers\\.(0|[1-9])\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts # custom MoE Kernel with expert paralleism kwargs: prefill_device: "cuda:0" prefill_op: "KExpertsTorch" generate_device: "cuda:0" generate_op: "KExpertsTorch" # do remember using correct backend, KExpertsCPU only runable on cpu. out_device: "cuda:0" recursive: False # don't recursively inject submodules of this module - match: name: "^model\\.layers\\.([12][0-9])\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts # custom MoE Kernel with expert paralleism kwargs: prefill_device: "cuda:0" prefill_op: "KExpertsTorch" generate_device: "cpu" generate_op: "KExpertsCPU" out_device: "cuda:0" recursive: False # don't recursively inject submodules of this module - match: name: "^model\\.layers\\.([345][0-9])\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts # custom MoE Kernel with expert paralleism kwargs: prefill_device: "cuda:1" prefill_op: "KExpertsTorch" generate_device: "cpu" generate_op: "KExpertsCPU" out_device: "cuda:1" recursive: False # don't recursively inject submodules of this module ``` For more detail please read my [tutorial](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/injection_tutorial.md). If you have any problem about tutorial, please feel free to ask!

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859

bsakaguchi referenced this issue

2026-10-29 14:45:38 +00:00

[GH-ISSUE #1286] [Bug] inference_context.v_cache[0][i] IndexError: list index out of range #871

bsakaguchi referenced this issue

2026-10-29 14:45:39 +00:00

[GH-ISSUE #1288] [Bug] #876