[GH-ISSUE #1181] [Bug] serve模式启动服务失败 #812

New issue

Closed

opened 2026-10-29 14:44:50 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:44:50 +00:00

Owner

Originally created by @zhaoyukoon on GitHub (Apr 22, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1181

检查清单

1. 我已经搜索过相关问题，但未能获得预期的帮助
2. 该问题在最新版本中尚未修复
3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

我仿照说明中可以启动以docker方式ktransformers serve模式。我希望减少显存使用，因此尝试减少并发、cache_size、max_new_tokens，总是会报错。

复现步骤

我使用docker compose方式启动deepseek-r1服务:

services:
  deepseek-r1:
    image: approachingai/ktransformers:v0.2.4post1-AVX512
    ports:
      - "8005:8005"
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: ["gpu"]
              driver: nvidia
              device_ids: ["GPU-0ad0418d-f718-8200-188f-78e76d3a10da"]
              options:  # 关键配置
                memory: 16g  # 限制显存为16GB
    volumes:
      - /mnt/backup/models:/models
    working_dir: /workspace/ktransformers
    entrypoint: /opt/nvidia/nvidia_entrypoint.sh
    command: >
      python -m ktransformers.server.main
      --model_path /models/deepseek-ai/DeepSeek-R1
      --gguf_path /models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M
      --cpu_infer 24
      --chunk_size 256
      --max_new_tokens 8192
      --cache_lens 40960
      --max_batch_size 4
      --port 8005
      --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
      --force_think
      --backend_type balance_serve
    healthcheck:
      test: ["CMD-SHELL", 'wget -O- deepseek-r1:8005/v1/assistants/status > /dev/null']
      interval: 10s
      retries: 5
      start_period: 30s
      timeout: 10s
    restart: unless-stopped

显卡是4090，可以正常启动，占用17GB显存。现在我希望降低显存占用到16GB以下，以使用其他显卡。
我尝试修改

      --max_new_tokens 4096
      --cache_lens 20480
      --max_batch_size 2

运行总是会得到如下错误：

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/ktransformers/ktransformers/server/backend/interfaces/balance_serve.py", line 246, in run_engine
    engine.model_runner.warmup()
  File "/workspace/ktransformers/ktransformers/server/balance_serve/inference/model_runner.py", line 107, in warmup
    self.model.flash_infer_attn_plan(self.input[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf,
  File "/workspace/ktransformers/ktransformers/models/custom_modeling_deepseek_v3.py", line 145, in flash_infer_attn_plan
    self.wrapper.plan(minibatch.q_indptr, minibatch.kv_indptr, minibatch.kv_indices,
  File "/opt/conda/lib/python3.11/site-packages/flashinfer/mla.py", line 246, in plan
    self._qo_indptr_buf[:len(qo_indptr)].copy_(qo_indptr, non_blocking=True)
RuntimeError: The size of tensor a (4) must match the size of tensor b (5) at non-singleton dimension 0

环境信息

ubuntu 24.04, AMD9135x2, 48GBx24, NVIDIA 4090 24GB

Originally created by @zhaoyukoon on GitHub (Apr 22, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1181 ### 检查清单 - [x] 1. 我已经搜索过相关问题，但未能获得预期的帮助 - [x] 2. 该问题在最新版本中尚未修复 - [x] 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性 - [ ] 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭 - [ ] 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭 ### 问题描述我仿照说明中可以启动以docker方式ktransformers serve模式。我希望减少显存使用，因此尝试减少并发、cache_size、max_new_tokens，总是会报错。 ### 复现步骤我使用docker compose方式启动deepseek-r1服务: ``` services: deepseek-r1: image: approachingai/ktransformers:v0.2.4post1-AVX512 ports: - "8005:8005" deploy: resources: reservations: devices: - capabilities: ["gpu"] driver: nvidia device_ids: ["GPU-0ad0418d-f718-8200-188f-78e76d3a10da"] options: # 关键配置 memory: 16g # 限制显存为16GB volumes: - /mnt/backup/models:/models working_dir: /workspace/ktransformers entrypoint: /opt/nvidia/nvidia_entrypoint.sh command: > python -m ktransformers.server.main --model_path /models/deepseek-ai/DeepSeek-R1 --gguf_path /models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --cpu_infer 24 --chunk_size 256 --max_new_tokens 8192 --cache_lens 40960 --max_batch_size 4 --port 8005 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml --force_think --backend_type balance_serve healthcheck: test: ["CMD-SHELL", 'wget -O- deepseek-r1:8005/v1/assistants/status > /dev/null'] interval: 10s retries: 5 start_period: 30s timeout: 10s restart: unless-stopped ``` 显卡是4090，可以正常启动，占用17GB显存。现在我希望降低显存占用到16GB以下，以使用其他显卡。我尝试修改 ``` --max_new_tokens 4096 --cache_lens 20480 --max_batch_size 2 ``` 运行总是会得到如下错误： ```bash Traceback (most recent call last): File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/workspace/ktransformers/ktransformers/server/backend/interfaces/balance_serve.py", line 246, in run_engine engine.model_runner.warmup() File "/workspace/ktransformers/ktransformers/server/balance_serve/inference/model_runner.py", line 107, in warmup self.model.flash_infer_attn_plan(self.input[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, File "/workspace/ktransformers/ktransformers/models/custom_modeling_deepseek_v3.py", line 145, in flash_infer_attn_plan self.wrapper.plan(minibatch.q_indptr, minibatch.kv_indptr, minibatch.kv_indices, File "/opt/conda/lib/python3.11/site-packages/flashinfer/mla.py", line 246, in plan self._qo_indptr_buf[:len(qo_indptr)].copy_(qo_indptr, non_blocking=True) RuntimeError: The size of tensor a (4) must match the size of tensor b (5) at non-singleton dimension 0 ``` ### 环境信息 ubuntu 24.04, AMD9135x2, 48GBx24, NVIDIA 4090 24GB

bsakaguchi closed this issue

2026-10-29 14:44:50 +00:00

bsakaguchi commented

2026-10-29 14:44:51 +00:00

Author

Owner

@zhaoyukoon commented on GitHub (Apr 22, 2025):

--chunk_size 128
--cache_lens 28000
--max_batch_size 3

可以正常启动服务，且显存使用15.7GB。

不过我看启动过程中会有峰值20GB的情况，不确定在16GB显卡能否正常启动。

@zhaoyukoon commented on GitHub (Apr 22, 2025): ``` --chunk_size 128 --cache_lens 28000 --max_batch_size 3 ``` 可以正常启动服务，且显存使用15.7GB。不过我看启动过程中会有峰值20GB的情况，不确定在16GB显卡能否正常启动。

bsakaguchi commented

2026-10-29 14:44:51 +00:00

Author

Owner

@qiyuxinlin commented on GitHub (Apr 22, 2025):

@zhaoyukoon 您好，ktransformers 峰值显存会出现在 lm_head 解量化过程中，这部分权重需要 10GB 以上显存，您加载的镜像（历史版本）会先申请 kvcache 再加载模型，可能会导致 OOM，主线代码现在会先加载模型后再申请 kvcache，但是我也不确定 16G 能不能成功运行起来

@qiyuxinlin commented on GitHub (Apr 22, 2025): @zhaoyukoon 您好，ktransformers 峰值显存会出现在 lm_head 解量化过程中，这部分权重需要 10GB 以上显存，您加载的镜像（历史版本）会先申请 kvcache 再加载模型，可能会导致 OOM，主线代码现在会先加载模型后再申请 kvcache，但是我也不确定 16G 能不能成功运行起来

bsakaguchi commented

2026-10-29 14:44:51 +00:00

Author

Owner

@zhaoyukoon commented on GitHub (Apr 23, 2025):

我今天做了测试，我先启动一个qwen2.5-7b模型占用了8GB显存，剩余16GB显存。

我再通过docker 启动 ktransformers，可以正常启动。

多谢！

@zhaoyukoon commented on GitHub (Apr 23, 2025): 我今天做了测试，我先启动一个qwen2.5-7b模型占用了8GB显存，剩余16GB显存。我再通过docker 启动 ktransformers，可以正常启动。多谢！