[GH-ISSUE #1338] [Bug] #898

New issue

Open

opened 2026-10-29 14:45:53 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:45:53 +00:00

Owner

Originally created by @withyou971 on GitHub (May 26, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1338

检查清单

1. 我已经搜索过相关问题，但未能获得预期的帮助
2. 该问题在最新版本中尚未修复
3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

python ktransformers/local_chat.py --architectures Qwen3MoeForCausalLM --model_path /mnt/model/Qwen3-30B-A3B --gguf_path /mnt/model/Qwen3-30B-A3B --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml

loading model.layers.47.input_layernorm.weight to cuda
loading model.layers.47.post_attention_layernorm.weight to cuda
loading model.norm.weight to cuda
Getting inference context from sched_client.
sched_rpc started with PID: 186
Got inference context, sending it to subscribers.
Rebuilding kvcache
Process SpawnProcess-1:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 277, in run_engine
engine = Engine(args, token_queue, broadcast_endpoint, kvcache_event)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 192, in init
inference_context = self.sched_client.rebuild_inferece_context(inference_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/sched_rpc.py", line 196, in rebuild_inferece_context
inference_context.k_cache = [fn(*args) for fn,args in data['k_cache']]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/sched_rpc.py", line 196, in
inference_context.k_cache = [fn(*args) for fn,args in data['k_cache']]
^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/multiprocessing/reductions.py", line 181, in rebuild_cuda_tensor
storage = storage_cls._new_shared_cuda(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/storage.py", line 1434, in _new_shared_cuda
return torch.UntypedStorage._new_shared_cuda(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: invalid resource handle
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

复现步骤

环境信息

windows wsl ubtunu 2204 3090*1
使用镜像approachingai/ktransformers:v0.3.1-AVX2

Originally created by @withyou971 on GitHub (May 26, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1338 ### 检查清单 - [x] 1. 我已经搜索过相关问题，但未能获得预期的帮助 - [x] 2. 该问题在最新版本中尚未修复 - [x] 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性 - [x] 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭 - [x] 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭 ### 问题描述 python ktransformers/local_chat.py --architectures Qwen3MoeForCausalLM --model_path /mnt/model/Qwen3-30B-A3B --gguf_path /mnt/model/Qwen3-30B-A3B --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml loading model.layers.47.input_layernorm.weight to cuda loading model.layers.47.post_attention_layernorm.weight to cuda loading model.norm.weight to cuda Getting inference context from sched_client. sched_rpc started with PID: 186 Got inference context, sending it to subscribers. Rebuilding kvcache Process SpawnProcess-1: Traceback (most recent call last): File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 277, in run_engine engine = Engine(args, token_queue, broadcast_endpoint, kvcache_event) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 192, in __init__ inference_context = self.sched_client.rebuild_inferece_context(inference_context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/sched_rpc.py", line 196, in rebuild_inferece_context inference_context.k_cache = [fn(*args) for fn,args in data['k_cache']] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/sched_rpc.py", line 196, in <listcomp> inference_context.k_cache = [fn(*args) for fn,args in data['k_cache']] ^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/multiprocessing/reductions.py", line 181, in rebuild_cuda_tensor storage = storage_cls._new_shared_cuda( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/storage.py", line 1434, in _new_shared_cuda return torch.UntypedStorage._new_shared_cuda(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: CUDA error: invalid resource handle CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ### 复现步骤 python ktransformers/local_chat.py --architectures Qwen3MoeForCausalLM --model_path /mnt/model/Qwen3-30B-A3B --gguf_path /mnt/model/Qwen3-30B-A3B --backend_type balance_serve --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml ### 环境信息 windows wsl ubtunu 2204 3090*1 使用镜像approachingai/ktransformers:v0.3.1-AVX2

bsakaguchi commented

2026-10-29 14:45:54 +00:00

Author

Owner

@withyou971 commented on GitHub (May 26, 2025):

不使用镜像会卡在loading blk.44.attn_k_norm.weight to cuda
loading blk.44.attn_norm.weight to cuda
loading blk.44.ffn_norm.weight to cuda
loading blk.45.attn_q_norm.weight to cuda
loading blk.45.attn_k_norm.weight to cuda
loading blk.45.attn_norm.weight to cuda
loading blk.45.ffn_norm.weight to cuda
loading blk.46.attn_q_norm.weight to cuda
loading blk.46.attn_k_norm.weight to cuda
loading blk.46.attn_norm.weight to cuda
loading blk.46.ffn_norm.weight to cuda
loading blk.47.attn_q_norm.weight to cuda
loading blk.47.attn_k_norm.weight to cuda
loading blk.47.attn_norm.weight to cuda
loading blk.47.ffn_norm.weight to cuda
loading output_norm.weight to cuda
Getting inference context from sched_client.
sched_rpc started with PID: 65096
Got inference context, sending it to subscribers.
Rebuilding kvcache
48
kv_cache loaded successfully.

@withyou971 commented on GitHub (May 26, 2025): 不使用镜像会卡在loading blk.44.attn_k_norm.weight to cuda loading blk.44.attn_norm.weight to cuda loading blk.44.ffn_norm.weight to cuda loading blk.45.attn_q_norm.weight to cuda loading blk.45.attn_k_norm.weight to cuda loading blk.45.attn_norm.weight to cuda loading blk.45.ffn_norm.weight to cuda loading blk.46.attn_q_norm.weight to cuda loading blk.46.attn_k_norm.weight to cuda loading blk.46.attn_norm.weight to cuda loading blk.46.ffn_norm.weight to cuda loading blk.47.attn_q_norm.weight to cuda loading blk.47.attn_k_norm.weight to cuda loading blk.47.attn_norm.weight to cuda loading blk.47.ffn_norm.weight to cuda loading output_norm.weight to cuda Getting inference context from sched_client. sched_rpc started with PID: 65096 Got inference context, sending it to subscribers. Rebuilding kvcache 48 kv_cache loaded successfully.

bsakaguchi referenced this issue

2026-10-29 14:53:03 +00:00

[PR #898] [MERGED] Release 0.2.3post2 #1510