[GH-ISSUE #1339] 双卡A100部署DS-671-Q4KM加速效果不明显 #901

New issue

Open

opened 2026-10-29 14:45:55 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:45:55 +00:00

Owner

Originally created by @tongda on GitHub (May 27, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1339

ktransformers 版本：0.2.4post1-AVX2
硬件配置：

CPU: 2 * AMD EPYC 7K62
Memory: 1TB
GPU: 2 * A100 80G

模型使用DeepSeek-R1-Q4_K_M。

使用默认配置启动命令：

ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /models/DeepSeek-R1-Q4_K_M/DeepSeek-R1-Q4_K_M/ --optimize_config_path /models/DeepSeek-V3-Chat-multi-gpu.yaml --cpu_infer 190 --max_new_tokens 32768

两张显卡，每张显卡显存使用7GB左右。
测试输出，3.5tok/s

修改配置，把expert部分offload到GPU，配置如下：

# 省略前面的配置，保持默认没变...

- match:
    name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.mlp\\.gate$"
    class: ktransformers.models.modeling_deepseek_v3.MoEGate
  replace:
    class: ktransformers.operators.gate.KMoEGate
    kwargs:
      generate_device: "cuda:0"
      prefill_device: "cuda:0"
- match:
    name: "^model\\.layers\\.([3456][0-9])\\.mlp\\.gate$"
    class: ktransformers.models.modeling_deepseek_v3.MoEGate
  replace:
    class: ktransformers.operators.gate.KMoEGate     # mlp module with custom forward function
    kwargs:
      generate_device: "cuda:1"
      prefill_device: "cuda:1"

# 把0-12层的expert放到GPU
- match:
    name: "^model\\.layers\\.(0|[1-9]|[1][0-2])\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts
    kwargs:
      generate_device: "cuda:0" # run in cuda:0
      generate_op:  "KExpertsMarlin"
  recursive: False

# 30-39层的expert放到GPU
- match:
    name: "^model\\.layers\\.([3][0-9])\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts
    kwargs:
      generate_device: "cuda:1"
      generate_op:  "KExpertsMarlin"
  recursive: False

- match:
    name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    kwargs:
      prefill_device: "cuda:0"
      prefill_op: "KExpertsTorch"
      generate_device: "cpu"
      generate_op:  "KExpertsCPU"
      out_device: "cuda:0"
  recursive: False # don't recursively inject submodules of this module

- match:
    name: "^model\\.layers\\.([3456][0-9])\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    kwargs:
      prefill_device: "cuda:1"
      prefill_op: "KExpertsTorch"
      generate_device: "cpu"
      generate_op:  "KExpertsCPU"
      out_device: "cuda:1"
  recursive: False # don't recursively inject submodules of this module

- match:
    name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.self_attn$"
  replace:
    class: ktransformers.operators.attention.KDeepseekV2Attention # optimized MLA implementation
    kwargs:
      generate_device: "cuda:0"
      prefill_device: "cuda:0"
- match:
    name: "^model\\.layers\\.([3456][0-9])\\.self_attn$"
  replace:
    class: ktransformers.operators.attention.KDeepseekV2Attention # optimized MLA implementation
    kwargs:
      generate_device: "cuda:1"
      prefill_device: "cuda:1"
- match:
    name: "^model$"
  replace:
    class: "ktransformers.operators.models.KDeepseekV2Model"
    kwargs:
      per_layer_prefill_intput_threshold: 0 # 0 is close layer wise prefill
      transfer_map:
        30: "cuda:1"

- match:
    name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\."
  replace:
    class: "default"
    kwargs:
      generate_device: "cuda:0"
      prefill_device: "cuda:0"

- match:
    name: "^lm_head"
    class: torch.nn.Linear
  replace:
    class: ktransformers.operators.linear.KTransformersLinear
    kwargs:
      generate_device: "cuda:1"
      prefill_device: "cuda:1"
      generate_op: "KLinearMarlin"
      prefill_op: "KLinearTorch"

- match:
    name: "(^model\\.layers\\.([3456][0-9])\\.)|(model.norm)"
  replace:
    class: "default"
    kwargs:
      generate_device: "cuda:1"
      prefill_device: "cuda:1

运行命令，关闭了cuda_graph：

ktransformers --model_path /models/deepseek-ai/DeepSeek-R1 --gguf_path /models/DeepSeek-R1-Q4_K_M/DeepSeek-R1-Q4_K_M/ --optimize_config_path /models/DeepSeek-V3-Chat-multi-gpu-marlin.yaml --cpu_infer 190 --max_new_tokens 32768 --no-use_cuda_graph

显存占用是上去了，两张卡都超过60G的占用，但是测试推理速度，只有1.6tok/s，还没有默认配置来的快。

请问如何配置才能最大化双卡A100的运行速度？

Originally created by @tongda on GitHub (May 27, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1339 ktransformers 版本：0.2.4post1-AVX2 硬件配置： * CPU: 2 * AMD EPYC 7K62 * Memory: 1TB * GPU: 2 * A100 80G 模型使用DeepSeek-R1-Q4_K_M。 1. 使用默认配置启动命令： ``` ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /models/DeepSeek-R1-Q4_K_M/DeepSeek-R1-Q4_K_M/ --optimize_config_path /models/DeepSeek-V3-Chat-multi-gpu.yaml --cpu_infer 190 --max_new_tokens 32768 ``` 两张显卡，每张显卡显存使用7GB左右。 **测试输出，3.5tok/s** 2. 修改配置，把expert部分offload到GPU，配置如下： ``` # 省略前面的配置，保持默认没变... - match: name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.mlp\\.gate$" class: ktransformers.models.modeling_deepseek_v3.MoEGate replace: class: ktransformers.operators.gate.KMoEGate kwargs: generate_device: "cuda:0" prefill_device: "cuda:0" - match: name: "^model\\.layers\\.([3456][0-9])\\.mlp\\.gate$" class: ktransformers.models.modeling_deepseek_v3.MoEGate replace: class: ktransformers.operators.gate.KMoEGate # mlp module with custom forward function kwargs: generate_device: "cuda:1" prefill_device: "cuda:1" # 把0-12层的expert放到GPU - match: name: "^model\\.layers\\.(0|[1-9]|[1][0-2])\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts kwargs: generate_device: "cuda:0" # run in cuda:0 generate_op: "KExpertsMarlin" recursive: False # 30-39层的expert放到GPU - match: name: "^model\\.layers\\.([3][0-9])\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts kwargs: generate_device: "cuda:1" generate_op: "KExpertsMarlin" recursive: False - match: name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts # custom MoE Kernel with expert paralleism kwargs: prefill_device: "cuda:0" prefill_op: "KExpertsTorch" generate_device: "cpu" generate_op: "KExpertsCPU" out_device: "cuda:0" recursive: False # don't recursively inject submodules of this module - match: name: "^model\\.layers\\.([3456][0-9])\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts # custom MoE Kernel with expert paralleism kwargs: prefill_device: "cuda:1" prefill_op: "KExpertsTorch" generate_device: "cpu" generate_op: "KExpertsCPU" out_device: "cuda:1" recursive: False # don't recursively inject submodules of this module - match: name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.self_attn$" replace: class: ktransformers.operators.attention.KDeepseekV2Attention # optimized MLA implementation kwargs: generate_device: "cuda:0" prefill_device: "cuda:0" - match: name: "^model\\.layers\\.([3456][0-9])\\.self_attn$" replace: class: ktransformers.operators.attention.KDeepseekV2Attention # optimized MLA implementation kwargs: generate_device: "cuda:1" prefill_device: "cuda:1" - match: name: "^model$" replace: class: "ktransformers.operators.models.KDeepseekV2Model" kwargs: per_layer_prefill_intput_threshold: 0 # 0 is close layer wise prefill transfer_map: 30: "cuda:1" - match: name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\." replace: class: "default" kwargs: generate_device: "cuda:0" prefill_device: "cuda:0" - match: name: "^lm_head" class: torch.nn.Linear replace: class: ktransformers.operators.linear.KTransformersLinear kwargs: generate_device: "cuda:1" prefill_device: "cuda:1" generate_op: "KLinearMarlin" prefill_op: "KLinearTorch" - match: name: "(^model\\.layers\\.([3456][0-9])\\.)|(model.norm)" replace: class: "default" kwargs: generate_device: "cuda:1" prefill_device: "cuda:1 ``` 运行命令，关闭了cuda_graph： ``` ktransformers --model_path /models/deepseek-ai/DeepSeek-R1 --gguf_path /models/DeepSeek-R1-Q4_K_M/DeepSeek-R1-Q4_K_M/ --optimize_config_path /models/DeepSeek-V3-Chat-multi-gpu-marlin.yaml --cpu_infer 190 --max_new_tokens 32768 --no-use_cuda_graph ``` 显存占用是上去了，两张卡都超过60G的占用，但是测试推理速度，**只有1.6tok/s**，还没有默认配置来的快。请问如何配置才能最大化双卡A100的运行速度？

bsakaguchi commented

2026-10-29 14:45:56 +00:00

Author

Owner

@KathySkyer commented on GitHub (May 29, 2025):

楼主解决了吗？

@KathySkyer commented on GitHub (May 29, 2025): 楼主解决了吗？

bsakaguchi commented

2026-10-29 14:45:56 +00:00

Author

Owner

@Naist4869 commented on GitHub (May 29, 2025):