kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1020] [Bug] 0.2.4无法运行 #723

New issue
Open
opened 2026-10-29 14:43:28 +00:00 by bsakaguchi · 9 comments
bsakaguchi commented 2026-10-29 14:43:28 +00:00
Owner
Copy link

Originally created by @yeungtuzi on GitHub (Apr 2, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1020

检查清单

  • 1. 我已经搜索过相关问题，但未能获得预期的帮助
  • 2. 该问题在最新版本中尚未修复
  • 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
  • 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
  • 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

正常编译，运行时控制台不报错，后台无显存占用，dmesg显示segfault

[347557.489256] python3[102229]: segfault at 150 ip 00007372beca3601 sp 00007ffc181202c0 error 4 in libsched.so[7372bec7b000+6a000] likely on CPU 11 (core 11, socket 0)

每次运行都随机报某一个核心上segfault

复现步骤

export HF_ENDPOINT="https://hf-mirror.com"
ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml

Image Image

环境信息

2x Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
64GB DDR4-2933 * 16
ubuntu 22.04.5 lts
Linux local 6.8.0-52-generic

Name: ktransformers
Version: 0.2.4+cu128torch26fancy

Originally created by @yeungtuzi on GitHub (Apr 2, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1020 ### 检查清单 - [x] 1. 我已经搜索过相关问题，但未能获得预期的帮助 - [x] 2. 该问题在最新版本中尚未修复 - [x] 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性 - [x] 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭 - [x] 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭 ### 问题描述 正常编译，运行时控制台不报错，后台无显存占用，dmesg显示segfault [347557.489256] python3[102229]: segfault at 150 ip 00007372beca3601 sp 00007ffc181202c0 error 4 in libsched.so[7372bec7b000+6a000] likely on CPU 11 (core 11, socket 0) 每次运行都随机报某一个核心上segfault ### 复现步骤 export HF_ENDPOINT="https://hf-mirror.com" ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml <img width="761" alt="Image" src="https://github.com/user-attachments/assets/b0a898bc-ad5c-401c-b511-9c47e4201181" /> <img width="1746" alt="Image" src="https://github.com/user-attachments/assets/a412833d-26b9-411c-9434-2f243f8f9b9f" /> ### 环境信息 2x Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz 64GB DDR4-2933 * 16 ubuntu 22.04.5 lts Linux local 6.8.0-52-generic Name: ktransformers Version: 0.2.4+cu128torch26fancy
bsakaguchi commented 2026-10-29 14:43:29 +00:00
Author
Owner
Copy link

@qiyuxinlin commented on GitHub (Apr 2, 2025):

我们定位到了这个问题，请您下载或指定 deepseek-v3 的config 目录(一般保存在 ~/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/ 下)，我们紧急修复下

<!-- gh-comment-id:2772421702 --> @qiyuxinlin commented on GitHub (Apr 2, 2025): 我们定位到了这个问题，请您下载或指定 deepseek-v3 的config 目录(一般保存在 ~/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/ 下)，我们紧急修复下
bsakaguchi commented 2026-10-29 14:43:29 +00:00
Author
Owner
Copy link

@yeungtuzi commented on GitHub (Apr 2, 2025):

启动命令已经修改为

export HF_ENDPOINT="https://hf-mirror.com"
ktransformers --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml

正常启动了

<!-- gh-comment-id:2772459481 --> @yeungtuzi commented on GitHub (Apr 2, 2025): 启动命令已经修改为 export HF_ENDPOINT="https://hf-mirror.com" ktransformers --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml 正常启动了
bsakaguchi commented 2026-10-29 14:43:29 +00:00
Author
Owner
Copy link

@qiyuxinlin commented on GitHub (Apr 2, 2025):

今天修不出来了，明天接着看看

<!-- gh-comment-id:2773030597 --> @qiyuxinlin commented on GitHub (Apr 2, 2025): 今天修不出来了，明天接着看看
bsakaguchi commented 2026-10-29 14:43:29 +00:00
Author
Owner
Copy link

@MotorBottle commented on GitHub (Apr 2, 2025):

启动命令已经修改为

export HF_ENDPOINT="https://hf-mirror.com" ktransformers --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml

正常启动了

使用绝对路径确实能解决这个问题，但load完各层之后又遇到新报错

Process SpawnProcess-1:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 246, in run_engine
    engine.model_runner.warmup()
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 123, in warmup
    self.outputs_buf[i] = self.model(self.input[i], self.features_buf[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, self.page_idx_buf[i], self.page_offset_buf[i])
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 112, in forward
    hidden_states, residual = decode_layer.input_layernorm(hidden_states, num_tokens_tensors, residual)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: DeepseekV3RMSNorm.forward() takes 2 positional arguments but 4 were given
<!-- gh-comment-id:2773143653 --> @MotorBottle commented on GitHub (Apr 2, 2025): > 启动命令已经修改为 > > export HF_ENDPOINT="https://hf-mirror.com" ktransformers --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml > > 正常启动了 使用绝对路径确实能解决这个问题，但load完各层之后又遇到新报错 ``` Process SpawnProcess-1: Traceback (most recent call last): File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 246, in run_engine engine.model_runner.warmup() File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 123, in warmup self.outputs_buf[i] = self.model(self.input[i], self.features_buf[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, self.page_idx_buf[i], self.page_offset_buf[i]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 112, in forward hidden_states, residual = decode_layer.input_layernorm(hidden_states, num_tokens_tensors, residual) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: DeepseekV3RMSNorm.forward() takes 2 positional arguments but 4 were given ```
bsakaguchi commented 2026-10-29 14:43:30 +00:00
Author
Owner
Copy link

@qiyuxinlin commented on GitHub (Apr 3, 2025):

启动命令已经修改为
export HF_ENDPOINT="https://hf-mirror.com" ktransformers --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
正常启动了

使用绝对路径确实能解决这个问题，但load完各层之后又遇到新报错

Process SpawnProcess-1:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 246, in run_engine
    engine.model_runner.warmup()
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 123, in warmup
    self.outputs_buf[i] = self.model(self.input[i], self.features_buf[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, self.page_idx_buf[i], self.page_offset_buf[i])
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 112, in forward
    hidden_states, residual = decode_layer.input_layernorm(hidden_states, num_tokens_tensors, residual)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: DeepseekV3RMSNorm.forward() takes 2 positional arguments but 4 were given

可否贴出 ineject 信息，这似乎是 DeepseekV3RMSNorm 函数没有被 inject 导致的

<!-- gh-comment-id:2774148682 --> @qiyuxinlin commented on GitHub (Apr 3, 2025): > > 启动命令已经修改为 > > export HF_ENDPOINT="https://hf-mirror.com" ktransformers --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml > > 正常启动了 > > 使用绝对路径确实能解决这个问题，但load完各层之后又遇到新报错 > > ``` > Process SpawnProcess-1: > Traceback (most recent call last): > File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap > self.run() > File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run > self._target(*self._args, **self._kwargs) > File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 246, in run_engine > engine.model_runner.warmup() > File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 123, in warmup > self.outputs_buf[i] = self.model(self.input[i], self.features_buf[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, self.page_idx_buf[i], self.page_offset_buf[i]) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl > return self._call_impl(*args, **kwargs) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl > return forward_call(*args, **kwargs) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/conda/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 112, in forward > hidden_states, residual = decode_layer.input_layernorm(hidden_states, num_tokens_tensors, residual) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl > return self._call_impl(*args, **kwargs) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl > return forward_call(*args, **kwargs) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > TypeError: DeepseekV3RMSNorm.forward() takes 2 positional arguments but 4 were given > ``` 可否贴出 ineject 信息，这似乎是 DeepseekV3RMSNorm 函数没有被 inject 导致的
bsakaguchi commented 2026-10-29 14:43:30 +00:00
Author
Owner
Copy link

@MotorBottle commented on GitHub (Apr 5, 2025):

启动命令已经修改为
export HF_ENDPOINT="https://hf-mirror.com" ktransformers --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
正常启动了

使用绝对路径确实能解决这个问题，但load完各层之后又遇到新报错

Process SpawnProcess-1:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 246, in run_engine
    engine.model_runner.warmup()
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 123, in warmup
    self.outputs_buf[i] = self.model(self.input[i], self.features_buf[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, self.page_idx_buf[i], self.page_offset_buf[i])
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 112, in forward
    hidden_states, residual = decode_layer.input_layernorm(hidden_states, num_tokens_tensors, residual)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: DeepseekV3RMSNorm.forward() takes 2 positional arguments but 4 were given

可否贴出 ineject 信息，这似乎是 DeepseekV3RMSNorm 函数没有被 inject 导致的

这个问题已解决，optimize config需要改用DeepSeek-V3-Chat-serve.yaml，DeepSeek-V3-Chat.yaml会报以上错

<!-- gh-comment-id:2780672978 --> @MotorBottle commented on GitHub (Apr 5, 2025): > > > 启动命令已经修改为 > > > export HF_ENDPOINT="https://hf-mirror.com" ktransformers --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml > > > 正常启动了 > > > > > > 使用绝对路径确实能解决这个问题，但load完各层之后又遇到新报错 > > ``` > > Process SpawnProcess-1: > > Traceback (most recent call last): > > File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap > > self.run() > > File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run > > self._target(*self._args, **self._kwargs) > > File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 246, in run_engine > > engine.model_runner.warmup() > > File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 123, in warmup > > self.outputs_buf[i] = self.model(self.input[i], self.features_buf[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, self.page_idx_buf[i], self.page_offset_buf[i]) > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl > > return self._call_impl(*args, **kwargs) > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl > > return forward_call(*args, **kwargs) > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > File "/opt/conda/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 112, in forward > > hidden_states, residual = decode_layer.input_layernorm(hidden_states, num_tokens_tensors, residual) > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl > > return self._call_impl(*args, **kwargs) > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl > > return forward_call(*args, **kwargs) > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > TypeError: DeepseekV3RMSNorm.forward() takes 2 positional arguments but 4 were given > > ``` > > 可否贴出 ineject 信息，这似乎是 DeepseekV3RMSNorm 函数没有被 inject 导致的 这个问题已解决，optimize config需要改用DeepSeek-V3-Chat-serve.yaml，DeepSeek-V3-Chat.yaml会报以上错
bsakaguchi commented 2026-10-29 14:43:30 +00:00
Author
Owner
Copy link

@dijiupianhai9 commented on GitHub (Apr 24, 2025):

启动命令已经修改为
export HF_ENDPOINT="https://hf-mirror.com" ktransformers --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
正常启动了

使用绝对路径确实能解决这个问题，但load完各层之后又遇到新报错

Process SpawnProcess-1:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 246, in run_engine
    engine.model_runner.warmup()
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 123, in warmup
    self.outputs_buf[i] = self.model(self.input[i], self.features_buf[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, self.page_idx_buf[i], self.page_offset_buf[i])
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 112, in forward
    hidden_states, residual = decode_layer.input_layernorm(hidden_states, num_tokens_tensors, residual)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: DeepseekV3RMSNorm.forward() takes 2 positional arguments but 4 were given

可否贴出 ineject 信息，这似乎是 DeepseekV3RMSNorm 函数没有被 inject 导致的

这个问题已解决，optimize config需要改用DeepSeek-V3-Chat-serve.yaml，DeepSeek-V3-Chat.yaml会报以上错

请教一下，使用DeepSeek-V3-Chat-serve.yaml是多卡推理吗？

<!-- gh-comment-id:2826574389 --> @dijiupianhai9 commented on GitHub (Apr 24, 2025): > > > > 启动命令已经修改为 > > > > export HF_ENDPOINT="https://hf-mirror.com" ktransformers --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml > > > > 正常启动了 > > > > > > > > > 使用绝对路径确实能解决这个问题，但load完各层之后又遇到新报错 > > > ``` > > > Process SpawnProcess-1: > > > Traceback (most recent call last): > > > File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap > > > self.run() > > > File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run > > > self._target(*self._args, **self._kwargs) > > > File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 246, in run_engine > > > engine.model_runner.warmup() > > > File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 123, in warmup > > > self.outputs_buf[i] = self.model(self.input[i], self.features_buf[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, self.page_idx_buf[i], self.page_offset_buf[i]) > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl > > > return self._call_impl(*args, **kwargs) > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl > > > return forward_call(*args, **kwargs) > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > File "/opt/conda/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 112, in forward > > > hidden_states, residual = decode_layer.input_layernorm(hidden_states, num_tokens_tensors, residual) > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl > > > return self._call_impl(*args, **kwargs) > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl > > > return forward_call(*args, **kwargs) > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > TypeError: DeepseekV3RMSNorm.forward() takes 2 positional arguments but 4 were given > > > ``` > > > > > > 可否贴出 ineject 信息，这似乎是 DeepseekV3RMSNorm 函数没有被 inject 导致的 > > 这个问题已解决，optimize config需要改用DeepSeek-V3-Chat-serve.yaml，DeepSeek-V3-Chat.yaml会报以上错 请教一下，使用DeepSeek-V3-Chat-serve.yaml是多卡推理吗？
bsakaguchi commented 2026-10-29 14:43:30 +00:00
Author
Owner
Copy link

@MotorBottle commented on GitHub (Apr 24, 2025):

启动命令已经修改为
export HF_ENDPOINT="https://hf-mirror.com" ktransformers --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
正常启动了

使用绝对路径确实能解决这个问题，但load完各层之后又遇到新报错

Process SpawnProcess-1:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 246, in run_engine
    engine.model_runner.warmup()
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 123, in warmup
    self.outputs_buf[i] = self.model(self.input[i], self.features_buf[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, self.page_idx_buf[i], self.page_offset_buf[i])
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 112, in forward
    hidden_states, residual = decode_layer.input_layernorm(hidden_states, num_tokens_tensors, residual)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: DeepseekV3RMSNorm.forward() takes 2 positional arguments but 4 were given

可否贴出 ineject 信息，这似乎是 DeepseekV3RMSNorm 函数没有被 inject 导致的

这个问题已解决，optimize config需要改用DeepSeek-V3-Chat-serve.yaml，DeepSeek-V3-Chat.yaml会报以上错

请教一下，使用DeepSeek-V3-Chat-serve.yaml是多卡推理吗？

单卡

<!-- gh-comment-id:2826578000 --> @MotorBottle commented on GitHub (Apr 24, 2025): > > > > > 启动命令已经修改为 > > > > > export HF_ENDPOINT="https://hf-mirror.com" ktransformers --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1/snapshots/56d4cbbb4d29f4355bab4b9a39ccb717a14ad5ad --gguf_path /root/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --port 10002 --max_new_tokens=32768 --cache_lens=131072 --model_name "DeepSeek-R1" --chunk_size 256 --max_batch_size 4 --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml > > > > > 正常启动了 > > > > > > > > > > > > 使用绝对路径确实能解决这个问题，但load完各层之后又遇到新报错 > > > > ``` > > > > Process SpawnProcess-1: > > > > Traceback (most recent call last): > > > > File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap > > > > self.run() > > > > File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run > > > > self._target(*self._args, **self._kwargs) > > > > File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 246, in run_engine > > > > engine.model_runner.warmup() > > > > File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 123, in warmup > > > > self.outputs_buf[i] = self.model(self.input[i], self.features_buf[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, self.page_idx_buf[i], self.page_offset_buf[i]) > > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl > > > > return self._call_impl(*args, **kwargs) > > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl > > > > return forward_call(*args, **kwargs) > > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > > File "/opt/conda/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 112, in forward > > > > hidden_states, residual = decode_layer.input_layernorm(hidden_states, num_tokens_tensors, residual) > > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl > > > > return self._call_impl(*args, **kwargs) > > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > > File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl > > > > return forward_call(*args, **kwargs) > > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > > TypeError: DeepseekV3RMSNorm.forward() takes 2 positional arguments but 4 were given > > > > ``` > > > > > > > > > 可否贴出 ineject 信息，这似乎是 DeepseekV3RMSNorm 函数没有被 inject 导致的 > > > > > > 这个问题已解决，optimize config需要改用DeepSeek-V3-Chat-serve.yaml，DeepSeek-V3-Chat.yaml会报以上错 > > 请教一下，使用DeepSeek-V3-Chat-serve.yaml是多卡推理吗？ 单卡
bsakaguchi commented 2026-10-29 14:43:30 +00:00
Author
Owner
Copy link

@Adadxz commented on GitHub (Jul 7, 2025):

今天修不出来了，明天接着看看

请问这个问题解决了吗？在0.3.1版本似乎还是可以碰到这样的问题
OS: Ubuntu 22.04.1

ktransformers：0.3.1+cu128torch27fancy

CPU：GENUINE INTEL(R) XEON(R)
Core(s) per socket: 72
Socket(s): 2

GPU：A6000

memory: 1.47T

Run cmd:
image

Build ktransformser cmd:

For those who have two cpu and 1T RAM（Dual NUMA）:

USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh

Run result:

Image
<!-- gh-comment-id:3043889472 --> @Adadxz commented on GitHub (Jul 7, 2025): > 今天修不出来了，明天接着看看 请问这个问题解决了吗？在0.3.1版本似乎还是可以碰到这样的问题 OS: Ubuntu 22.04.1 ktransformers：0.3.1+cu128torch27fancy CPU：GENUINE INTEL(R) XEON(R) Core(s) per socket: 72 Socket(s): 2 GPU：A6000 memory: 1.47T Run cmd: image Build ktransformser cmd: # For those who have two cpu and 1T RAM（Dual NUMA）: USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh Run result: <img width="1281" height="394" alt="Image" src="https://github.com/user-attachments/assets/3fc9e853-52b2-4d8e-ac47-e898242c96b1" />
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#723
No description provided.